show for$a\in\left(0,1\right)$ $x^{-a}$ is not self concordant s.t $0\leq x$

I want to show for $a\in\left(0,1\right)$ $x^{-a}$ is not self concordant s.t. $x > 0$. By definition, a function is called self-concordant function if $\left|\left(x^{-\alpha}\right)^{\prime\prime\prime}\right|\leq2\cdot\left(\left(x^{-\alpha}\right)^{\prime\prime}\right)^{\frac{3}{2}}$.

So what I need to prove is that $$\left|-ax^{-a-3}\left(-a-1\right)\left(-a-2\right)\right|\leq2\cdot\left(-ax^{-a-2}\left(-a-1\right)\right)^{\frac{3}{2}}$$ doesn't hold.

I got to $-x^{-a-3}\left(-a-1\right)\leq2\cdot\left(ax^{-a-2}\right)\sqrt{\left(-ax^{-a-2}\left(-a-1\right)\right)}$, but I don't know how to continue from here.

Your approach is correct. In fact, the inequality you have obtained can be expressed as $$ { a(a+1)(a+2)x^{-a-3}\le 2a^\frac{3}{2}(a+1)^\frac{3}{2} x^{-\frac{3}{2}a-3} \implies \\ a+2\le 2a^\frac{1}{2}(a+1)^\frac{1}{2} x^{\frac{3}{2}a} } $$ which does not hold for all $x>0$ since for sufficiently small $x$, $LHS>RHS$ and for sufficiently large $x$, $LHS<RHS$. However, the function is self-concordant over $ \left[\frac{(a+2)^\frac{2}{3a}}{(4a(a+1))^\frac{1}{3a}},\infty\right) $.

We have $f(x) = x^{-\alpha}, \alpha \in (0,1), x>0$. Therefore

$$\begin{align} & f'(x) = -\alpha x^{-\alpha-1}\\ & f''(x) = \alpha(\alpha + 1) x^{-\alpha-2} \\ & f'''(x) = -\alpha(\alpha + 1)(\alpha + 2)x^{-\alpha-3} \end{align}$$

Base on the definition of self concordant according to wiki (basically the "convex optimization" book by Boyd and Vandenberghe, chapter 9), we must have

$$|f'''(x)|\le 2|f''(x)|^{\frac{3}{2}}$$

We form the quotient $Q(x)=\frac{|f'''(x)|}{ 2|f''(x)|^{\frac{3}{2}}}$ as follow

$$Q(x) = \frac{\frac{\alpha(\alpha + 1)(\alpha + 2)}{x^{\alpha+3}}}{2{\left(\frac{\alpha(\alpha + 1)}{x^{\alpha+2}}\right)^{\frac{3}{2}}}} = \frac{(\alpha + 2)x^{\frac{1}{2}\alpha}}{2\sqrt{\alpha(\alpha + 1)}} $$

By taking derivative of $Q(x)$, as $Q'(x) = \frac{(\frac{1}{2}\alpha)(\alpha + 2)x^{\frac{1}{2}\alpha-1}}{2\sqrt{\alpha(\alpha + 1)}} = 0$, we see that derivative of $Q$ at origin is unbounded below, in other words $x=0$ is asymptote of $Q(x)$ at origin and $Q(x) \ge 1$ for a sufficiently large $x$. $f(x)\; \color{red}{\text{is not self-concordant!}}$

However if you put $h(x) = -\ln(-f(x)) -\ln(x)$, then $h(x)$ is self-concordant.