Definition of group action

The first definition assumes that the map $\phi: G \to \mbox{Perm}S$ is a homomorphism of groups, i.e. that the map preserves multiplication. Now, already, elements of $\mbox{Perm}(S)$ are just bijections $p: S \to S.$ So for any $g \in G,$ $\phi(g): S \to S$ will be such a bijection. Perhaps it is good to emphasize that $\phi$ takes an element of $g$ to a function on $S.$ Sometimes we write this function as $\phi_g.$ This makes the notation nicer; it is nicer to write $\phi_g(s)$ for the image of an element $s \in S$ under the function $\phi(g) \in \mbox{Perm}(S)$ than it is to write $\phi(g)(s).$

In any case, the corresponding map $\psi: G \times S \to S$ is just $\psi(g,s) = \phi_g(s),$ i.e. the image of $s$ under $\phi(g).$ This map in fact yields a group action since $\phi_{g h} = \phi(g h) = \phi(g) \circ \phi(h)= \phi_g \circ \phi_h$ so $\psi(g h,s) = \phi_{g h}(s) = \phi_g(\phi_h(s)) = \psi(g,\psi(h,s)),$ which is what is required for a map to be a group action (cf. @Dylan Moreland's comment).

Conversely, if we were to start off with such a group action $\psi: G \times S \to S$ satisfying $\psi(g h, s) = \psi(g, \psi(h,s)), $ then we may define $\phi: G \to \mbox{Perm}(S)$ by how the image of an element of g under $\phi$ acts on elements of $s.$ That is, we define $\phi(g)$ to be that element $f:S\to S$ of $\mbox{Perm}{S}$ such that $f(s) = \psi(g,s).$ Then you may check that this yields a homomorphism of $G$ into $\mbox{Perm}{S}.$

Your author's definition is that of a permutation representation which is equivalent to a group action.

If $\varphi:G \to \mathrm{Perm}(S)$, then given $g \in G$, $\varphi(g) \in \mathrm{Perm}(S)$ (a permutation of $S$). Suppose $s\in S$, then one can define $g \cdot s = (\varphi(g))(s)$. It's not hard to show that this is a group action.

Conversely, if one has a group action, it is not hard to show that $s \mapsto g\cdot s$ is a permutation, call it $\varphi(g)$. Then one has a map $\varphi:G\to \mathrm{Perm}(S)$ which is a group homomorphism.

So you can look at a homomorphism from your group to a group of permutations on $S$ or you can consider your group acting on $S$. They are sort of two sides of the same coin.

This sort of thing appears in many contexts usually under the names representation and module.