Is $a \circ b = \sqrt{a^2+b^2}$ ever a group?
Solution 1:
Let $R$ be a ring* such that for each $x\in R$ there is a unique element $x'$ such that $x'^2= x$, call that element $\sqrt x$.
Then the operation $a\circ b=\sqrt{a^2+b^2}$ can be defined, is associative and commutative, and has the identity $0$ (the additive identity of $R$). Now suppose for an element $a$ there is a $b$ such that $a\circ b=0$, that is, $a$ has an inverse. It follows from the definition of $\sqrt\cdot$ and the ring axioms that $\sqrt 0=0$, so we must have $a^2+b^2=0$ and $b^2=-a^2$. Thus $a$ has an inverse for $\circ$ iff $-a^2$ is the square of some element, but this is guaranteed by our assumptions about $R$.
Thus, as long as $\sqrt\cdot$ is uniquely defined and $R$ is closed under it, $R$ is a group under $\circ$. Note that uniqueness really is important, since if $\sqrt\cdot$ isn't unique, there isn't even an identity. If $x\neq y$ both have the same square, then either $x\circ0\neq x$ or $y\circ0\neq y$. This eliminates the possibility of just taking something like $\mathbb C$ and defining $\sqrt\cdot$ piecewise.
* Technically, we don't need distributivity. $R$ needs to be a group under addition and satisfy $0^2=0$.
Solution 2:
To show an example of how Jack's construction works, here's a specific example:
It's well-known that the non-negative integers $\mathbb{Z}^{\geq 0}$ form a group under the operation $\oplus$ of nim-addition (or equivalently, under bitwise exclusive-or), with identity $0$ and each number being its own inverse. This infinite group (which is the projective limit of the product of $n$ copies of $\mathbb{Z}/2\mathbb{Z}$ as $n\to\infty$ in the obvious way) has finite subgroups corresponding to the restriction to $0\leq n\lt 2^i$ for each $i$.
What's less-known is that there's also a multiplication operation, nim-multiplication, which is compatible with nim-addition and forms a ring (in fact, a field!) over $\mathbb{Z}^{\geq 0}$. One way of defining the multiplication is to set $\alpha\otimes\beta = \mathop{mex}(\alpha'\otimes\beta\oplus\alpha\otimes\beta'\oplus\alpha'\otimes\beta' : \alpha'\lt\alpha, \beta'\lt\beta)$, where $\mathop{mex}()$ refers to the minimal excluded value, the smallest number not in the given set (under this notation, nim-addition itself can be defined by $\alpha\oplus\beta = \mathop{mex}(\alpha'\oplus\beta, \alpha\oplus\beta')$); another (admittedly more straightforward) way is as the projective limit of the Galois fields of order $2^{2^n}$; and much as in the addition case, the restrictions (in this case, the individual fields of size $2^{2^i}$) are all finite subfields. Here (swiped from Wikipedia) is the multiplication table for $n\lt 16 = 2^{2^2}$:
$$\begin{matrix} 0&0&0&0&0&0&0&0&0&0&0&0&0&0&0&0\\ 0&1&2&3&4&5&6&7&8&9&10&11&12&13&14&15\\ 0&2&3&1&8&10&11&9&12&14&15&13&4&6&7&5\\ 0&3&1&2&12&15&13&14&4&7&5&6&8&11&9&10\\ 0&4&8&12&6&2&14&10&11&15&3&7&13&9&5&1\\ 0&5&10&15&2&7&8&13&3&6&9&12&1&4&11&14\\ 0&6&11&13&14&8&5&3&7&1&12&10&9&15&2&4\\ 0&7&9&14&10&3&13&4&15&8&6&1&5&2&12&11\\ 0&8&12&4&11&3&6&15&13&5&1&9&6&14&10&2\\ 0&9&14&7&15&6&1&8&5&12&11&2&10&3&4&13\\ 0&10&15&5&3&9&12&6&1&11&14&4&2&8&13&7\\ 0&11&13&6&7&12&10&1&9&2&4&15&14&5&3&8\\ 0&12&4&8&13&1&9&5&6&10&2&14&11&7&15&3\\ 0&13&6&11&9&4&15&2&14&3&8&5&7&10&1&12\\ 0&14&7&9&5&11&2&12&10&4&13&3&15&1&8&6\\ 0&15&5&10&1&14&4&11&2&13&7&8&3&12&6&9\\ \end{matrix} $$ You can clearly see the subfield of order $4=2^{2^1}$ embedded in the top left corner of this multiplication table; you can also see that the diagonal is a permutation of $[0\ldots 15]$, or in other words that for each $x\in F_{16}$ there's a unique $y$ such that $y\otimes y=x$. (This isn't a proof for the arbitrary case, of course, but that can also be done; I believe Conway's On Numbers And Games covers the uniqueness of square roots). Determining what group of order 16 the operation $a\circ b = \sqrt[\otimes]{a^{\otimes2}\oplus b^{\otimes2}}$ generates is a nice exercise (hint: what's the inverse of $a$ under $\circ$?); in fact, $\circ$ here turns out to be even simpler than might be expected, thanks to a straightforward-but-unexpected identity! (another hint: expand $(a\oplus b)\otimes(a\oplus b)$.)
Solution 3:
To address the second paragraph of the question: Yes, $\circ$ is a group operation on any of the following subsets of $\mathbb{C}$:
- $\{0\}$
- Your set $\mathbb{H}$, together with the nonnegative real numbers
- More generally, $\sqrt{G}$ where $\sqrt{\phantom{G}}$ is a branch of the square root and $G$ is a subgroup of $\mathbb{C}$.