Maximizing an angle based on certain constraints
Solution 1:
It is given that $a, b \gt 0$ so both $A$ and $B$ are on the same side of x-axis. The first point to note is that $\triangle ABC$ is obtuse and $\angle ACB$ is acute. Now we use the relationship $AB = 2 R \sin C$ where $R$ is the circumradius of $\triangle ABC$. As $AB$ is fixed, we maximize $\angle C$ when we minimize $R$ given $\sin$ function is strictly increasing for $ \left(0, \frac{\pi}{2} \right)$.
Also note that $O$ must be on the perpendicular bisector of $AB$ which is parallel to x-axis. So, $R = OC$ is minimum when $OC$ is perpendicular to x-axis.
Solution 2:
Let $\omega$ be the circumcirle of $\triangle ABC$ where $C$ is a point on the $x$-axis such that $\angle ACB$ is maximum. Assume $\omega$ intersects the $x$-axis twice, at $C$ and $D$.
Let $F$ be any point on the arc ${CD}$ (not containing $A, B$) and define $E$ as the intersection of $AF$ and $x$-axis. Observe, $$\angle ACB=\angle AFB<\angle AEB$$ which contradicts the fact that $\angle ACB$ is maximum.
Therefore, the assumption that $\omega$ intersects the $x$-axis twice is incorrect, which implies $\omega$ is tangent to the $x$-axis at $C$.
Solution 3:
Let $\bigcirc K$ through $A$ and $B$ be tangent to the $x$-axis at $D$. For $C$ on the $x$-axis (and on the same side of the $y$-axis as $D$), let $A'$ and $B'$ be the "other" points where $\overleftrightarrow{AC}$ and $\overleftrightarrow{BC}$ meet this circle.
A corollary to the Inscribed Angle Theorem states that we can write $$\angle C = \frac12 \left(\;\angle AKB - \angle A'KB'\;\right)$$ Since $\angle AKB$ is fixed, maximizing $\angle C$ amounts to minimizing $\angle A'KB'$. This happens when (and only when) $A'$ and $B'$ coincide; hence, when $C$ and $D$ coincide. $\square$
Solution 4:
We use the fact that if $A$ and $B$ are on a given circle, then if you have $C$ on the circle and $C_0$ (strictly) inside the circle (and $C$, $C_0$ are on the same side of $\overline{AB}$) $$ \angle AC_0B > \angle ACB $$
You can see this by extending $AC_0$ to the circle at $C'_0$, in which case $\angle AC_0B > \angle AC'_0B$, but because of the inscribed angle theorem, $\angle AC'_0B = \angle ACB$.
Now, write the desired angle $\angle ACB$ as $f(x)$ in terms of $x$; we want to maximize $f(x)$.
The circumcircle of $\triangle ABC$ always intersects the $x$-axis at $C = (x_0, 0)$. Now, say for contradiction that $f(x_0)$ is maximal, and that the circumcircle of $\triangle ABC$ also intersects the $x$-axis at $C' = (x_0', 0) \neq C$. Then the midpoint $M$ of $\overline{CC'}$ is inside the circle, so $\angle AMB > \angle ACB$; and $M = \left(\frac{x_0 + x'_0}{2}, 0 \right)$ is on the $x$-axis, so $$f\left( \frac{x_0+x_0'}{2} \right) > f(x_0), $$ contradicting the fact that $f(x_0)$ is maximal.
Thus the circumcircle of $\triangle ABC$ must intersect the $x$-axis at exactly one point.