Advice on finding counterexamples

Perhaps the best way of going about these things is to look towards the simplest possible counterexamples first. For example, if you have ever taken point-set topology, you'll know that the indiscrete topology and the discrete topology are the first two spaces that should pop into your mind when looking for examples simply because they are so easily "grokked" by our minds (followed by metric spaces, then Hausdorff spaces, and so forth).

So what about when it comes to problems like this in abstract algebra? Well, we've been well-acquainted with the additive group $\mathbb{Z}$ since grade school! Why not start there?

My (naive) advice is to try to make a property fail in simple, easy-to-work-with structures first, and only dive into the abstraction as a last resort. Finally, as a general rule of thumb, it's easier to prove something once you've developed a certain degree of intuition and are convinced the result should go one way or another. The great Riemann conveyed this best: "If only I had the theorems! Then I should find the proofs easily enough."

If you don’t find a counterexample using the approaches already suggested, try to prove the theorem, and see where you get stuck: this may well give you a better handle on the properties that a counterexample will have to have, and that’s invaluable information when you’re searching for a counterexample or trying to construct one from scratch. This doesn’t really arise in the case under consideration, because counterexamples are very easy to come by: it’s almost harder not to find one. In more complicated settings, however, the technique is very helpful.

In this particular case, Lagrange's Theorem is the first thing that comes to my mind. If we just pick two subgroups $H, K \leq G$, there's no reason that $|H \cup K|$ should divide $|G|$, while it must, in order to be a subgroup.

The easiest kind of $|G|$ is some power of $2$ (this gives me the option of finding two subgroups of order $2$ whose union must have size $3$, instantly disqualifying the union from being a subgroup, as $3 \not\mid 2^n$).

So, pick your favorite two subgroups of the Klein $4$-group!

To try and go into a bit more detail about the though process/tips specific to group theory. Lagrange's Theorem is one of the most fundamental results concerning a relationship between a group and its subgroups. So, it seems natural to me that we should be able to find some counter-example utilizing something so fundamental. Almost all of my group theory background is in the finite case, where divisibility plays an enormous role.

So, I honed in on one of the most basic theorems we have, regarding a group and its subgroups. Then I thought about what conditions on $H, K$ would make finding $H, K$ as easy as possible, so that it couldn't possibly be the case that $|H \cup K|$ divides $|G|$. Using subgroups of size $2$ is nice, as $H \neq K$ immediately implies that $|H \cap K| = 1$, so that $|H \cup K| = 4 - 1 = 3$.

Having a handful of small groups, or small 'kinds' of groups also does wonders, like the Klein $4$-group here.

First, your own question to give me some clarity:

I am asking if someone could post a counterexample to the following problem and explain to me their thought process. This would help me so much.

Problem: Give an example of two subgroups $H$ and $K$ of a group $G$ whose union $H\cup K$ is not a subgroup of $G$.

Here is my thought process:

I need to find a group, $G$, big enough to have two different subgroups.

I need to make sure that I don't have $H \subset K$ lest $H \cup K = K$, a subgroup of $G$.

Similarly, I must avoid $K \subset H$.

Okay: the symmetries of a triangle give rise to the group $S_3$, which has six elements. This example comes to mind because it is a bit weird among the small groups; for example, it is the smallest nonabelian group. Also, it is has a bunch of subgroups that I sort of remember.

Once I have this example in mind, I know the concerns above about picking subgroups with one contained in the other can be avoided. My strategy now would be either to pick two of the fold groups (the ones represented by switching two vertices of a triangle) and checking to see if that messes up closure or set size, or to pick one fold group and one spinning group (the three element rotation group isomorphic to $\mathbb{Z}/3\mathbb{Z}$) and then see if that messes up closure or set size.

When I say set size, I do mean, in particular, that a group's order is divisible by the order of any of its subgroups (i.e., Lagrange's Theorem).

Okay: Let's give it a shot using the latter idea. We take the group with three elements (the rotations) and another group with two elements (a fold); let us see if we can mess up group size. Each of these groups contains only the identity in their intersection, so their union would have $3+2-1 = 4$ elements (three in the first group; two in the second group; but we have double-counted the identity, so we subtract one).

But now our union of subgroups has $4$ elements, which does not divide the order of $S_3$, i.e., $6$. So the union being a subgroup would contradict Lagrange's Theorem, and we are done.

In retrospect, I might check to see what would have happened with closure using the subgroups we ended up picking. Alternatively, I might take a shot at the other idea: Picking two of the fold groups and seeing if their union messes up group size (no, the union has $2 + 2 - 1 = 3$ elements, which divides $|S_3| = 6$) or closure. (A quick check reveals that, indeed, closure is a problem for both the subgroup we picked and any two of the fold groups; so counterexamples abound!)

It seems like people have already come up with answers regarding your particular problem, so I think I'll just throw a few words around regarding how to find counterexamples. I have two basic strategies myself that I use when thinking about theorems and problems.

1) Consult a list of known counter-examples. This is a really good strategy in subjects like analysis and topology. I don't know how well it works in algebra, but I expect there are several known pathological counter-examples. Although I don't have a whole lot of experience, I expect groups like the Klein 4-group, and perhaps scalar copies of the integers are probably really good starting points in group theory. Even if you don't have a list of known counter-examples to work with, you do have certain knowledge of classes of groups, and so you can try working with some of them. Take a pass at scalar copies of the integers. If that doesn't work, think about why it didn't work. Maybe the integers modulo a prime will work.

2) Try to construct the specific counter-example you have in mind. This method is generally a little bit harder, but is sometimes a good method. It's what I would do to solve this particular problem, and it's basically what pjs36 did in his answer. To use this more direct method, the point is to try to quantify some facet of the thing in question, and use it to create the object you want. In his example, he is basically saying that if $|G|$ has order 4, and you have two normal subgroups, each has order 2, but they are different groups, both with the identity. So they each contain the identity and one other element. But then there are three elements, so this thing cannot possibly be a subgroup since 3 does not divide 4. The trick to this latter technique is having a (sufficiently) simple theorem or even a rule of thumb you can use as a guide line, which you will eventually end up violating. The Lagrange theorem is that theorem in this case, as it is a very simple and useful counting theorem which gives lots of meaningful information.

And of course as with all problems and all skills in general, practice is the most important thing. If you are just looking for practice coming up with counter-examples, there are two small books you could check out, Counter-examples in Analysis and Counter-examples in Topology. You could read these books and try to come up with the counter-example before reading the proof. If you can't see it, read what the counter-example is, and then try to come up with the proof yourself. If you still can't see the trick, then read the proof, and try to take careful notes on what is at work conceptually, so you can see exactly what has to be exploited in order to come up with the counter-example.

I know that this is not exactly related to your struggles in algebra, but the topology text really helped me when I was in a similar situation.