How to prove these Lie algebra relations for representations of $\mathfrak{sl}_2(\mathbb C)$
Solution 1:
First Part
(I assume that really $X\equiv d(X)$ for some representation $d$; this makes no odds.)
You're apparently familiar with the fact that $[[H,Y],Y]=0$ gives (inductively) $$[H,Y^n]=nY^{n-1}[H,Y]$$
Another inductive argument (which can in fact be generalized) would show $$[X,Y^n]=n(H+(n-1))Y^{n-1}$$
However there is a more satisfying approach involving showing
$$\boxed{e^{A}Be^{-A}=B+[A,B]+\frac{1}{2!}[A,[A,B]]+\frac{1}{3!}[A,[A,[A,B]]]+\cdots}$$
(Wikipedia points out that this is essentially equivalent to $e^{\textrm{ad}(A)} B$ in the case of the fundamental representation.)
Proof: Write $f_B(\lambda)=e^{\lambda A}Be^{-\lambda A}$. Then $$f_B'(\lambda)=e^{\lambda A}(AB-BA)e^{-\lambda A} = f_{[A,B]}(\lambda)$$ Inductively, we find $$f_B^{(n)}(\lambda) = f_{[A,\cdots [A,B]\cdots]}(\lambda)$$ Setting $\lambda=1$ and using the Taylor expansion of $f_B(1)$ gives the result. QED.
This generalizes the identity you already used. It is useful because, setting $A=Y,B=X$, we find $[A,B]=-H$ and so $[A,[A,B]]=[Y,-H]=-2Y$, and finally $$\boxed{[A,[A,[A,B]]]=[Y,-2Y]=0}$$ so the series terminates. You should be able to use this to compute the result you want.
Second Part
To see that the root is negated, I think you're simply supposed to use the definition of the action of group elements on the Lie algebra to observe that $$\Theta H_\alpha \Theta^{-1} = -H_\alpha \Theta \Theta^{-1} = -H_\alpha$$ You can use the $X,Y$ as well if you like.