$\cos A+\cos B+\cos C=2\cos\frac{A+B}{2}\cos\frac{A-B}{2}+1-2\sin^2\frac{C}{2}$

$=2\sin\frac{C}{2}\cos\frac{A-B}{2}+1-2\sin^2\frac{C}{2}\quad$ as $\quad A+B+C=\pi$ $\implies\cos\frac{A+B}{2}=\sin\frac{C}{2}$

$\leq 2\sin\frac{C}{2}+1-2\sin^2\frac{C}{2}\quad$ as $\quad\cos\frac{A-B}{2}\leq 1$

$=-\frac{1}{2}(2\sin\frac{C}{2}-1)^2+1+\frac{1}{2}$

The maximum value will come if $2\sin\frac{C}{2}=1\quad$ and if $\quad\cos\frac{A-B}{2}=1$

or if $\quad C=\frac{\pi}{3}\quad$ and $\quad A=B$. Then $A+B=\frac{2\pi}{3}\implies A=B=\frac{\pi}{3}=C$

The maximum value of $\cos A+\cos B+\cos C\quad$ thus $\quad\frac{3}{2}$

(i)Now, $4(\cos A+\cos B+\cos C)=2(\cos A+\cos B+\cos C)+2(\cos A+\cos B+\cos C)$

$\leq 2(\cos A+\cos B+\cos C)+2.\frac{3}{2}$

$=3+(\cos A+\cos B)+(\cos B+\cos C)+(\cos C+\cos A)$

Now, $\cos A+\cos B=2\cos\frac{A+B}{2}\cos\frac{A-B}{2}=2\sin\frac{C}{2}\cos\frac{A-B}{2}$

So, the problem reduces to $\sum 2\sin\frac{C}{2}\cos\frac{A-B}{2}≤ \sum \cos\frac{A-B}{2}$

(ii) Now, $4(\cos A+\cos B+\cos C)=(\cos A+\cos B+\cos C)+3(\cos A+\cos B+\cos C)$

$\leq (\cos A+\cos B+\cos C)+3.\frac{3}{2}$

$=3+\frac{1}{2}\sum (\cos A+\cos B + 1)$

$=3+\frac{1}{2}\sum (2\sin\frac{C}{2}\cos\frac{A-B}{2} + 1)$

So, the problem reduces to $\sum( \sin\frac{C}{2}\cos\frac{A-B}{2} + \frac{1}{2}) ≤ \sum \cos\frac{A-B}{2} $

I'm trying to prove this.


Excuse me if what I'm going to tell is too imprecise but an approach that might work is the following: define $x_1=\cos\frac{A}{2},x_2=\sin\frac{A}{2}$ and similarly $x_3,x_4$ for $B$ and $x_5,x_6$ for $C$. Using the formulas for the sine and cosine of the sum of two angles the inequality we want to prove is (if I didn't mess anything up) $$ 4(x_1^2-x_2^2+x_3^2-x_4^2+x_5^2-x_6^2)\leq 3+x_1x_3+x_2x_4+x_1x_5+x_2x_6+x_3x_5+x_4x_6. $$ Consider then the function $f:\mathbb{R}^6\rightarrow\mathbb{R}$ defined by $$ f(x_1,\ldots,x_6)=4(x_1^2-x_2^2+x_3^2-x_4^2+x_5^2-x_6^2)-(x_1x_3+x_2x_4+x_1x_5+x_2x_6+x_3x_5+x_4x_6). $$ We want to maximise it with the constraint given by $A+B+C=\pi$, which can be expressed in terms of the $x_i$, although not uniquely I'm afraid. Then we could make use of Lagrange multipliers.

Also $f$ is obviously a homogeneous degree 2 polynomial, so it defines a (projective) quadric. I wonder if this might be of help.

EDIT: The constraints $\cos(A+B+C)=-1$ and $\sin(A+B+C)=0$ in terms of the $x_i$ are $$ (x_1^2-x_2^2)(x_2^2-x_3^2)(x_5^2-x_6^2)-4(x_1^2-x_2^2)x_3x_4x_5x_6-4x_1x_2(x_3^2-x_4^2)x_5x_6-4x_1x_2x_3x_4(x_5^2-x_6^2)=-1 $$ and $$ 2x_1x_2(x_3^2-x_4^2)(x_5^2-x_6^2)+2(x_1^2-x_2^2)x_3x_4(x_5^2-x_6^2)+2(x_1^2-x_2^2)(x_3^2-x_4^2)x_5x_6-8x_1x_2x_3x_4x_5x_6=0. $$ Note that this only tells us that $A+B+C$ is an odd multiple of $\pi$ and not exactly $\pi$.


WLOG let $C$ be the largest angle, let $x=\frac{A-B}{2}, y=\frac{A+B}{2}$ then $-\frac{\pi}{4}<x<\frac{\pi}{4}, 0<y<\frac{\pi}{2}$. $$ A = x+y,~~ B=y-x,~~ C = \pi-2y \\ \cos\left(\frac{B-C}{2}\right) = \sin\left(\frac{3y-x}{2}\right), ~~ \cos\left(\frac{C-A}{2}\right) = \sin\left(\frac{3y+x}{2}\right) \\ \sin\left(\frac{3y-x}{2}\right)+\sin\left(\frac{3y+x}{2}\right) = 2\cos(x/2)\sin(3y/2) \\ \cos(x/2)\ge \cos x>\frac{1}{\sqrt{2}}, ~\sin(3y/2)>0 \Rightarrow 2\cos(x/2)\sin(3y/2)\ge2\cos x\sin(3y/2) $$

Then we can write $$ \begin{align} 3 \!&+\cos\left(\frac{A-B}{2}\right)+\cos\left(\frac{B-C}{2}\right)+\cos\left(\frac{C-A}{2}\right) -4(\cos A + \cos B + \cos C) \\ & = 3 + \cos x + 2\cos(x/2)\sin(3y/2)-4(\cos(x+y)+\cos(y-x)-\cos2y) \\ & = -1+\cos x+2\cos(x/2)\sin(3y/2)-8\cos x\cos y+8\cos^2y \\ & \ge 8\cos^2y-1+\cos x(1+2\sin(3y/2)-8\cos y) \\ & = 8\cos^2y-1+\cos x \cdot f(y) \end{align} $$ where we define $f(y)=1+2\sin(3y/2)-8\cos y$. Writing $v=y/2$ we can work through $$ \begin{align} 8\cos^2y-1+f(y) & = 7-8\sin^22v+1+2\sin3v-8\cos 2v\\ & = 8-16\sin^2v(1-2\sin^2v)+6\sin v-8\sin^3v-8(1-2\sin^2v) \\ & = 32\sin^4v-8\sin^3v-16\sin^2v+6\sin v \\ & = 2\sin v(1-2\sin v)^2(3+4\sin v) \end{align} $$ This last expression is clearly $\ge 0$ when $\sin v\ge 0$, but to answer the original question we now need to consider cases.

If $f(y)\le 0$, which corresponds to $0<y\le R=1.192797\cdots$ then $$ 8\cos^2y-1+\cos x\cdot f(y)\ge 8\cos^2y-1+f(y) \ge 0 $$ as we just described.

On the other hand if $R<y<\pi/2, ~f(y)>0$ then since $|x|<\pi/4$ $$ \begin{align} 8\cos^2y-1+\cos x\cdot f(y) & \ge 8\cos^2y-1+\frac{1}{\sqrt{2}}f(y)\\ & = 8\cos^2y-1+f(y)-f(y)\left(1-\frac{1}{\sqrt{2}}\right) \\ & = 2\sin v(1-2\sin v)^2(3+4\sin v)-f(y)\left(1-\frac{1}{\sqrt{2}}\right) \end{align} $$ I don't have an algebraic way to proceed from here, but it is easy to check numerically that the last expression is positive for $y$ in the desired range, which establishes the original inequality in $A,B,C$.