Why category of Groups is not a subcategory/(full subcategory) of the category of Sets and notational clarifications help for Fuller and Anderson text
Solution 1:
Here's a good way to figure out what $ \circ $ means. In this context, If it appears as $(G, \circ)$ or $(H, \circ )$ then it's the group multiplication. Otherwise it's probably composition in a category.
To write it out: If you have $(G, \circ)$ then here $\circ$ is a set-map $G \times G \rightarrow G$ with certain conditions. $mor_G((G,∘),(H,∘))$ is the set of group homomorphisms between the two groups. There's bad notation being used here since $G$ refers to the category of groups (which is better denoted by something like $\mathsf{Grp}$) and to a certain group.
You don't need to look at homomorphic maps $G \rightarrow G$ or $H \rightarrow H$ here.
(*) Here's the crux of the argument: Any homomorphism of groups $f: (G,\circ) \rightarrow (H, \circ)$ is a function/set-map $f: G \rightarrow H$ (with additional properties) where $G$ and $H$ are the sets underlying the respective groups. So, $mor_G((G,∘),(H,∘))\subset Map(G,H)$.
(**) $Map((G,∘),(H,∘))$ is a different beast entirely, since it is the set of (set-)functions from $(G,\circ)$ to $(H, \circ)$. With very mild abuse of notation, note that as a set, $(G,\circ) = (G, X)$ where $X$ is the appropriate subset of $(G \times G) \times G$ which represents $\circ$. Depending on your definition of an ordered pair, $(G,X)$ looks something like $ \{G, \{G, X \} \}$.
Ultimately an element of $mor_G((G,∘),(H,∘))$ is a set-map $G \rightarrow H$, and the elements of $Map((G,∘),(H,∘))$ are not of this type.