Question on Baby Rudin theorem 10.7
Solution 1:
First, let me answer your first question.
$F_m'(0)$$e_m$ is going to be the $m$-th column of the $n\times n$ Jacobian matrix of $F_m$ at $x=0$. I think using the $e_i$ to define the function makes it hard to visualize it. Let's build up how $F_m$ looks in vector form so we can build its Jacobian.
$$F_m(x)=\begin{bmatrix}I_{m-1} & 0\\ 0 & 0\end{bmatrix}x+\begin{bmatrix}0_{m-1\times1}\\ \alpha_m(x)\\ \vdots \\ \alpha_n(x)\end{bmatrix}=\begin{bmatrix}x_1 \\ \vdots \\ x_{m-1} \\ \alpha_m(x) \\ \vdots \\ \alpha_n(x) \end{bmatrix}$$
Note that, in the first matrix above, I ommitted the dimensions of the $0$ block matrices because otherwise it looked hideous; you can deduce these dimensions unambiguously. In the second matrix I included that the $0$ block is $m-1 \times 1$.
Now the Jacobian matrix of $F_m$ at $x$ is: $$\begin{bmatrix}I_{m-1} & 0\\ 0 & 0\end{bmatrix}+\begin{bmatrix}0_{m-1\times n}\\ \nabla\alpha_m(x)^\text{T}\\ \vdots \\ \nabla\alpha_n(x)^\text{T}\end{bmatrix}$$
Its $m$-th column is
$$F'_m(x)e_m=\begin{bmatrix}0_{n\times 1}\end{bmatrix}+\begin{bmatrix}0_{m-1\times1}\\ D_m\alpha_m(x)\\ \vdots \\ D_m\alpha_n(x)\end{bmatrix}=\sum_{i=m}^{n} D_m\alpha_i(x)e_i$$
Then $F'_m(0)e_m=\sum_{i=m}^{n} D_m\alpha_i(0)e_i$.
Now the second question. In the beginning statements we had $F_m(0)=0$. Looking at the vector expression for $F_m$ above, this means $\alpha_i(0)=0$ for all $i=m,\ldots,n$, including $i=k$. Now look at the definition for $G_m$ and you'll see $G_m(0)=0$, so $G_m^{-1}(0)=0$. Finally, $$F_{m+1}(0)=B_m F_m(G_m^{-1}(0))=B_m F_m(0)= B_m\ 0=0.$$