Of all polygons inscribed in a given circle which one has the maximum sum of squares of side lengths?

My son presented me with an interesting problem:

Of all possible polygons inscribed in a circle of radius $R$, find the one that has the sum $S$ of squared side lengths maximized: $S=a_1^2+a_2^2+\dots+a_n^2$, with $a_i$ representing the length of the $i$-th side. The number of sides is not fixed, you should consider all triangles, quadrilaterals, pentagons...

It's not that complicated, at least in the beginning. It's easy to show that the optimal polygon (with $n>3$) cannot have obtuse ($>90^\circ$) angles. For example, if such an angle $A_{i-1}A_{i}A_{i+1}$ exists, by cosine theorem:

$$|A_{i-1}A_{i}|^2+|A_{i}A_{i+1}|^2<|A_{i-1}A_{i+1}|^2$$

So if you drop vertex $A_i$, you get a polygon with a bigger $S$. This quickly eliminates all polygons with $n>4$.

All candidate polygons with $n=4$ must be rectangles and if their sides are $a$ and $b$, the sum $S$ is $2a^2+2b^2=8R^2$. So with respect to $S$ all rectangles inscribed in the circle are equivalent. In fact, a right triangle with sides $a$, $b$ and $2R$ has the same $S$ as any inscribed rectangle.

But maybe there is an inscribed triangle with $S>8R^2$. I was able to show that for an inscribed triangle with sides $a,b,c$ and $b\ne c$, an isosceles triangle with all acute angles and base $a$ has better value of $S$. So the optimal triangle must be isosceles. Looking from all three sides, the only possible solution is the equilateral triangle and the sum $S$ in that case is $9R^2$.

However, to prove that fact I had to use trigonometry which is not so complicated (and I can present it here if you want so), but it leaves impression that there has to be some simpler explanation why the equilateral triangle is the best choice. My trigonometry proof takes a few lines of text, I want something more elegant.

Just an idea: if you draw lines through the center of the circle perpendicular to the sides of a triangle and denote the pedal lengths with $h_a,h_b,h_c$, it's easy to show that in order to maximize $a^2+b^2+c^2$ you have to minimize $h_a^2+h_b^2+h_c^2$. And then - what?

EDIT: I want to present the part of the proof that I don't like. Take an arbitrary triangle $ABC$ with sides $a,b,c$ inscribed in a circle. Consider side $a$ fixed and play with angle $\gamma$ to get different values of $b,c$. I want to prove that isosceles triangle $BCA_1$ has bigger $S$ than any other triangle with one side equal to $a$.

$$b=2R\sin\frac{\pi-\alpha+\gamma}{2}=2R\cos\left(\frac\alpha2-\frac\gamma2\right)$$

$$c=2R\sin\frac{\pi-\alpha-\gamma}{2}=2R\cos\left(\frac\alpha2+\frac\gamma2\right)$$

$$b^2=4R^2\cos^2\left(\frac\alpha2-\frac\gamma2\right)=2R^2(1+\cos(\alpha-\gamma))$$

$$c^2=4R^2\cos^2\left(\frac\alpha2+\frac\gamma2\right)=2R^2(1+\cos(\alpha+\gamma))$$

$$b^2+c^2=4R^2+2R^2(\cos(\alpha-\gamma)+\cos(\alpha+\gamma))=4R^2(1+\cos\alpha\cos\gamma)$$

And this sum achieves maximum obviously for $\gamma=0$, or for $A\equiv A_1$. So for any given side $a$, $b$ and $c$ must be of equal. But you can look at the optimal triangle from sides $b$ and $c$ as well. The only triangle which has no better option is equilateral triangle.

EDIT 2: This “moving vertex” procedure can be repeated infinite number of times and the result is an equilateral triangle! Check excellent proof by Noah Schweber here.

Yes, the maximal sum is the one of the equilateral triangle, that is $9R^2$.

Since Prove that in any triangle $ABC$, $\cos^2A+\cos^2B+\cos^2C\geq\frac{3}{4}$ then $$ \sin^2 A+\sin^2 B+\sin^2 C=3-\cos^2 A-\cos^2 B-\cos^2 C\leq \frac{9}{4} $$ where $A$, $B$ and $C$ are non negative numbers such that $A+B+C=\pi$. Hence, for any inscribed triangle, the sum of the squares of the sides is $$(2R\sin A)^2+(2R\sin B)^2+(2R\sin C)^2\leq 9R^2.$$

This problem can be stated as

$$ \max_{n}\sum_{k=1}^n\left(2r\sin\left(\frac{\theta_k}{2}\right)\right)^2 $$

s. t.

$$ \sum_{k=1}^n\theta_k = 2\pi $$

but

$$ \sum_{k=1}^n\left(2r\sin\left(\frac{\theta_k}{2}\right)\right)^2\ge n\left(2^{2n}r^{2n}\prod_{k=1}^n\sin^2\left(\frac{\theta_k}{2}\right)\right)^{\frac 1n} $$

assuming $\theta_1=\cdots=\theta_n$ we have

$$ \sum_{k=1}^n\left(2r\sin\left(\frac{\theta_k}{2}\right)\right)^2\ge n\left(2^{2n}r^{2n}\sin^{2n}\left(\frac{\pi}{n}\right)\right)^{\frac 1n} = n2^2r^2\sin^{2}\left(\frac{\pi}{n}\right) $$

Now calling

$$ f(n) = n\sin^{2}\left(\frac{\pi}{n}\right) $$

we have clearly a maximum about $n = 3$ as can be depicted in the attached plot

Let $\theta_k$ be the successive angles subtended by the sides, but the last one. The sum of squares is given by

$$4\sum_{k=1}^n\sin^2\frac{\theta_k}2+4\sin^2\left(\pi-\frac12\sum_{k=1}^n\theta_k\right)$$ which has the same extrema as $$\sum_{k=1}^n\cos\theta_k-\cos\left(\sum_{k=1}^n\theta_k\right).$$

Canceling the gradient,

$$\sin\theta_k=\sin\left(\sum_{k=1}^n\theta_k\right).$$

This shows that all angles $\theta_k$ must be equal, and then

$$n\cos\theta-\cos n\theta$$ is minimized with $n\theta=2\pi$.

Finally,

$$n\cos\frac{2\pi} n-\cos\pi$$ is the smallest with $n=3$.

The objective function is continuous on the domain of interest (all triples of points on the circle), which is also compact. Therefore by the extreme value theorem it has a global maximum. That reduces the problem to the part you are interested in, namely proving that if the optimal triangle has sides $a,b,c$ then $b = c$. Firstly it has to be acute as you observed. Thus maximizing $b^2+c^2$ $= a^2+2bc·\cos(\angle A)$ is equivalent to maximizing $bc$, since fixing $B,C$ fixes $a$ and $\angle A$. Letting $x = \angle BAO$ and $y = \angle OAC$ we have $bc = 4R^2·\cos(x)\cos(y)$, and finally note $2 \cos(x)\cos(y)$ $= \cos(x+y) + \cos(x-y)$ $\le \cos(x+y) + 1$ $= \cos(\angle A) + 1$ with equality exactly when $x=y$.

$$ \begin{align} AB^2 + BC^2 + CA^2 &= (\vec{OB} - \vec{OA})^2 + (\vec{OC} - \vec{OB})^2 + (\vec{OA} - \vec{OC})^2\\ &= 2(\vec{OA})^2 + 2(\vec{OB})^2 + 2(\vec{OC})^2 - 2 \times \vec{OA} \cdot \vec{OB} - 2 \times \vec{OB} \cdot \vec{OC} - 2 \times \vec{OC} \cdot \vec{OA} \\ &= 3(\vec{OA})^2 + 3(\vec{OB})^2 + 3(\vec{OC})^2 - (\vec{OA} + \vec{OB} + \vec{OC})^2 \\ &\leq 9R^2. \end{align} $$