Prove or disprove that the function $f(x)=x^{x^{x^{x}}}$ is convex on $(0,1)$

Proof:

First, we have that

(1) Midpoint convex implies rational convex

(2) And Rational convex plus continuous implies convex.

See Midpoint-Convex and Continuous Implies Convex for details on this.

So it suffices to prove the function is mid-point convex. Let $x\in (0,1)$ and let $y\in (x,1)$ then we want to show that

$$f(\frac{x+y}{2}) \le \frac{f(x)+f(y)}{2}$$

That is, we want to show that

$$\left(\frac{x+y}{2}\right)^{\left(\frac{x+y}{2}\right)^{\left(\frac{x+y}{2}\right)^{\left(\frac{x+y}{2}\right)}}} \le \frac{x^{x^{x^{x}}}+y^{y^{y^{y}}}}{2}$$

but this follows from the argument in your original post.

Mathematica 12.3 does it in moment by

NMinimize[{D[x^x^x^x, {x, 2}], x > 0 && x < 1}, x]

$\{0.839082,\{x\to 0.669764\}\}$

Since the minimum value of the second derivative on $(0,1)$ is positive, the function under consideration is convex on $(0,1)$.

Addition. The @RiverLi user states doubts concerning the NMinimize result. Here are additional arguments. First, as

D[x^x^x^x, {x, 2}] // Simplify

$ x^{x^x+x^{x^x}-2} \left(x^{2 x} \log (x) \left(x \log ^2(x)+x \log (x)+1\right)^2+x^{x+1} \log ^2(x)+x^{x+2} \log ^2(x) (\log (x)+1)^2+3 x^{x+1} \log (x) (\log (x)+1)+2 x^x+x^x \log (x) (x+x \log (x)-1)+x^{x^x} \left(x^x \log (x) \left(x \log ^2(x)+x \log (x)+1\right)+1\right)^2-1\right)$ and

D[x^x^x^x, {x, 3}] // Simplify

$x^{x^x+x^{x^x}} \left(x^{3 x} \log (x) \left(\frac{1}{x}+\log ^2(x)+\log (x)\right)^3+x^{2 x-3} \left(x \log ^2(x)+x \log (x)+1\right)^2+x^{x-2} \left(\frac{1}{x}+\log ^2(x)+\log (x)\right) \left(x^{x+1} \log (x) (\log (x)+1)+x^x-1\right)+x^{2 x^x} \left(x^x \log (x) \left(\frac{1}{x}+\log ^2(x)+\log (x)\right)+\frac{1}{x}\right)^3+3 x^{x^x-3} \left(x^{x+1} \log ^3(x)+x^{x+1} \log ^2(x)+x^x \log (x)+1\right) \left(x^{2 x+2} \log ^5(x)+2 x^x+\left(x^x+4 x-1\right) x^x \log (x)+\left(2 x^x+1\right) x^{x+2} \log ^4(x)+\left(x^{x+1}+2 x^x+2 x\right) x^{x+1} \log ^3(x)+\left(2 x^x+x+5\right) x^{x+1} \log ^2(x)-1\right)+2 x^{x-3} \left(x^2 \log ^3(x)+2 x^2 \log ^2(x)+2 x+x (x+3) \log (x)-1\right)+3 x^{2 x-3} \log (x) \left(x \log ^2(x)+x \log (x)+1\right) \left(x^2 \log ^3(x)+2 x^2 \log ^2(x)+2 x+x (x+3) \log (x)-1\right)+\frac{\left(x^{x+1} \log (x) (\log (x)+1)+x^x-1\right)^2}{x^3}+\frac{x^{x+1} \log (x)+2 x^{x+1} (\log (x)+1)+x^{x+2} \log (x) (\log (x)+1)^2-x^x+1}{x^3}+x^{x-3} \log (x) \left(x^3 \log ^4(x)+3 x^3 \log ^3(x)+3 x^2+3 x^2 (x+2) \log ^2(x)+x \left(x^2+9 x-4\right) \log (x)+2\right)\right)$ show, the second derivative is continuously differentiable on $(0,1]$. Therefore, we can draw a conclusion that the second derivative is continuously differentiable on $[0.01,1]$.

Second,

Limit[D[x^x^x^x, {x, 2}], x -> 0, Direction -> "FromAbove"]

$\infty$

and

D[x^x^x^x, {x, 2}] /. x -> 0.01

$77.923$

and

D[x^x^x^x, {x, 2}] /. x -> 1

$2$

Third, the command of Maple (here Maple is stronger than Mathematica)

DirectSearch:-SolveEquations(diff(x^(x^(x^x)), x $ 3) = 0, {0 <= x, x <= 1}, AllSolutions);

$$\left[\begin{array}{cccc} 2.58795803978585\times10^{-24} & \left[\begin{array}{c} - 1.60871316268185183\times10^{-12} \end{array}\right] & \left[x= 0.669764056702161\right] & 23 \end{array}\right] $$ shows there is only one critical point of the second derivative on $[0.01,1]$. Combining the above with the value of the second derivative at $x=0.669764056702161$, i.e. with $0.83908$, and with the result of

NMaximize[{D[x^x^x^x, {x,3}] // Simplify, x >= 0 && x <= 0.01}, x]

$\{-4779.93,\{x\to 0.01\}\},$

we conclude that the second derivative takes its global minimum on $(0,1]$ at $x=0.669764056702161$ .