Solution 1:

You can write $$\int_{0}^{\infty} \frac{\log^2(1-e^{-x})x^5}{e^{x}-1}dx=\int_{0}^{\infty} \frac{\left[\log^2(e^{x}-1)-\log^2(e^{x})\right]x^5}{e^{x}-1}dx$$ $$=\int_{0}^{\infty} \frac{\left[\log^2(e^{x}-1)-x^2\right]x^5}{e^{x}-1}dx$$ $$=\int_{0}^{\infty} \frac{\log^2(e^{x}-1)x^5}{e^{x}-1}dx-\int_{0}^{\infty}\frac{x^7}{e^{x}-1}dx:=I_1-I_2.$$ Now, for $I_1,$ put $y=e^{x}-1$ then $x=\log(y+1)$ and $dx= \frac{dy}{y+1}$ and therefore $$I_1=\int_{0}^{\infty} \frac{\log^2(y)}{y} \times \frac{\log^5(y+1)}{(y+1)} dy$$ and $$I_2=\int_{0}^{\infty}\frac{\log(y+1)^7}{y(y+1)}dy.$$ Then $$I_1-I_2=\int_{0}^{\infty} \frac{\log^5(y+1)\left[\log^2(y)-\log^2(y+1)\right]}{y(y+1)}dy.$$ Then remark that $\frac{1}{y(y+1)}=\frac{1}{y}-\frac{1}{y+1}.$