A mistake in Grothendieck's Tôhoku paper? Theorem 5.2.1
As Grothendieck writes, the $G$-invariant global sections functor $\Gamma^G_X$ can be factored as the global sections functor $\Gamma_X$ followed by the ordinary $G$-invariants functor $\Gamma^G$, or as the the relative $G$-invariants functor $f_*^G$ followed by the global sections functor $\Gamma_Y$. (Also recall that the relative $G$-invariants functor $f_*^G$ can be factored as the direct image functor $f_*$ followed by the $G$-invariants functor $\Gamma^G$.)
The right derived functors of $\Gamma_X$ are the usual sheaf cohomology functors $H^* (X, -)$. You can check that for any $G$-sheaf $A$, $H^* (X, A)$ has an induced $G$-action, so it makes sense to take its $G$-invariants. The right derived functors of $\Gamma^G$ are the usual group cohomology functors $H^* (G, -)$. Thus, the Grothendieck spectral sequence applied to the factorisation $\Gamma^G_X \cong \Gamma^G \Gamma_X$ gives us a spectral sequence relating $H^* (G, H^* (X, A))$ to $H^* (X; G, A)$.
On the other hand, we also have the factorisation $\Gamma^G_X \cong \Gamma_Y f^G_*$. The right derived functors of $f^G_*$ are what you might call $\mathscr{H}^* (G, -)$. The right derived functors of $\Gamma_Y$ are $H^* (Y, -)$, of course. Thus, the Grothendieck spectral sequence relates $H^* (Y, \mathscr{H}^* (G, A))$ to $H^* (X; G, A)$.
So I think there are two typographical errors: $\mathscr{H}^q (X, A)$ in the expression for $\mathrm{I}^{p, q}_2 (A)$ should instead be $\mathscr{H}^q (G, A)$, and $\mathscr{H}^p$ in the expression for $\mathrm{II}^{p, q}_2 (A)$ should instead be $H^p$. This is consistent with the later remarks about edge homomorphisms and 5-term exact sequences.