Prove: If $N\lhd G$ and $N$ and $\frac{G}{N'}$ are nilpotent, then $G$ is nilpotent.

When $G$ is finite, I see the following argument: let $P$ be a Sylow $p$-subgroup of $G$ for some prime $p.$ Then $PN^{\prime} \lhd G$ by hypothesis. Then $G = N^{\prime}PN_{G}(P)$ by the Frattini argument. Hence $G = N^{\prime}N_{G}(P)$ and $N = N^{\prime}N_{N}(P)$ by Dedekind's Lemma. Now $N^{\prime} \leq \Phi(N)$ since $N$ is nilpotent (or just observe that every maximal subgroup of $N$ is normal of prime index, so contains $N^{\prime}).$ Hence we have $N = N_{N}(P).$ Thus $G = N^{\prime}N_{G}(P) \subseteq N_{G}(P)$ and $P \lhd G.$ Since $P$ was an arbitrary Sylow subgroup of $G$, $G$ is nilpotent.


Let's first prove that, if group $G$ acts nilpotently on abelian groups $M$ and $N$, then its induced action on $M \otimes N$ is also nilpotent.

By acting nilpotently, I mean that there are subgroup series $M = M_0 > M_1 > \cdots > M_m = 1$ and $N = N_0 > N_1 > \cdots > N_n = 1$ such that $G$ normalizes all subgroups in these series and centralizes all quotient groups $M_{i-1}/M_i$ and $N_{i-1}/N_i$.

To prove the claim, for $0 \le t \le m+n$, let $L_t$ be the subgroup of $M \otimes N$ generated by its subgroups $M_i \otimes N_j$ with $i+j=t$. Now (switching to additive notation in the abelian groups), for $g \in G$, $a \in M_i$, $b \in N_j$, we have $(a \otimes b)^g - (a \otimes b) = a^g \otimes (b^g-b) + (a^g-a) \otimes b \in L_{i+j+1}$, and so $G$ centralizes each quotient $L_t/L_{t+1}$, and we are done.

To use this to prove the required result, first note that $G/N'$ nilpotent implies that $G$ acts nilpotently on $N/N' = N^{(1)}/N^{(2)}$. For all $k > 1$, commutation induces a natural map $N/N' \otimes N^{(k-1)}/N^{(k)} \to N^{(k)}/N^{(k+1)}$, which is preserved by the action of $G$. So, using the result above, we can show by induction on $k$ that $G$ acts nilpotently on each factor $N^{(k-1)}/N^{(k)}$, hence $G$ is nilpotent.


Note that $N$ being nilpotent implies $N'\le\Phi(N)$.

Proof: Maximal subgroups are normal, and the quotient is cyclic.

Also, since $N$ is normal, $\Phi(N)\le\Phi(G)$.

Proof: If $M\le G$ is maximal and does not contain $\Phi(N)$, then $\Phi(N)M=G$, so $N=\Phi(N)(N\cap M)=(N\cap M)$, contradiction.

Thus, since being nilpotent is closed under quotient, $G/N'$ nilpotent implies $G/\Phi(G)$ nilpotent. This is enough to show $G$ is nilpotent.

Proof: Since $G/\Phi(G)$ is nilpotent, every maximal subgroup of $G/\Phi(G)$ is normal. Since $\Phi(G)$ is contained in every maximal subgroup of $G$, this implies every maximal subgroup of $G$ is normal, so $G$ is nilpotent.