How smooth can non-nice associative operations on the reals be?
Solution 1:
This is a partial answer assuming $\ast$ has a formal power series, that is,
$$X\ast Y=\sum_{p,q\geq 0}\alpha_{p,q}X^p Y^q$$
In order to be associative, we must have $\alpha_{0,0}=0$ (why?), so assume $\alpha_{0,0}=0$.
It is first shown in Lemma 3.2 of this article that $\alpha_{0,1},\alpha_{1,0}\in\left\{0,1\right\}$.
It is then shown in Proposition 3.3 that if $\alpha_{1,0}=0$ and $\alpha_{0,1}=1$, then $X\ast Y=Y$.
Theorem 3.6 then shows that if $\alpha_{1,1}=0$, then $\ast=0$.
The case $\alpha_{1,0}=\alpha_{0,1}=0$ is shown to make $\ast$ commutative in Proposition 3.10. (side remark: Theorem 4.3 gives the following beautiful representation theorem: $\ast$ is associative iff it is of the form $a\ast b=f^{-1}(f(a)f(b))$ for some invertible formal power series $f$. )
The case $\alpha_{1,0}=\alpha_{0,1}=1$ is described on page 38 of this book, and is also shown to make $\ast$ commutative.
Side remark on rational associative functions: Theorem C2 of this article gives the following representation theorem for associative rational functions: $X\ast Y=f(G(f^{-1}(X),f^{-1}(Y)))$, for some homografic function $f$ and $G$ one of $+$, $\cdot$, and $(X,Y)\mapsto\frac{X+Y}{1-XY}$.