Can one distinguish finite groups by their maps from abelian groups?

No. Consider two semidirect products $(\mathbf{Z}/7\mathbf{Z})^2\rtimes(\mathbf{Z}/3\mathbf{Z})$; where the canonical generator of $\mathbf{Z}/3\mathbf{Z}$ acts in the first case by the matrix $\begin{pmatrix}2 & 0\\0 & 2\end{pmatrix}$, and in the second case by $\begin{pmatrix}2 & 0\\0 & 4\end{pmatrix}$. They're clearly not isomorphic, but they have the same "combinatorics" of abelian subgroups: the unique $7$-Sylow subgroup (elementary abelian of order $7^2$) and its subgroups, and $7^2$ subgroups of order 3, and all these have pairwise trivial intersection.

Not a full answer but hopefully a good starting point. Let $G_1,G_2$ be two groups and $\varphi\colon G_1 \to G_2$ a homomorphism. We'll say $\varphi$ is good if in the semi-direct product $G_1 \rtimes_\varphi G_2$ elements $x,y$ commute iff either $x,y\in G_1$ or $x,y\in G_2$. In that case, any homomorphism $A\to G_1\rtimes_\varphi G_2$ from an abelian group lands in either $G_1$ or $G_2$. In other words, if $\varphi$ is good, then $(G_1\rtimes_\varphi G_2)_A\cong (G_1/A)\times (G_2/A)$. In particular, if you can find two good twisting homomorphisms $\varphi_1,\varphi_2$, then the category invariant you consider will not be able to differentiate between $G_1\rtimes_{\varphi_1} G_2$ and $G_1 \rtimes_{\varphi_2}G_2$. If $\varphi_1,\varphi_2$ can be chosen to produce non-isomorpihc semi-direct products, then you're done. This latter property is easy to achieve (generally, semi-direct products on the same factors are highly sensitive to the twisting homomorphism, even for very small groups). I suspect making sure the twisting homomorphisms are also good is possible, though I don't have a construction right now.