Can we define factorials in sufficiently nice categories?

Yes if the ambient category $\mathcal{C}$ has pullbacks (and, of course, is cartesian closed). I will denote $B^A = \underline{\mathrm{Hom}}(A,B)$ and construct a subobject $\underline{\mathrm{Isom}}(A,B)$ as follows: Consider the morphisms $$\underline{\mathrm{Hom}}(A,B) \times \underline{\mathrm{Hom}}(B,A) \to \underline{\mathrm{Hom}}(A,A) \times \underline{\mathrm{Hom}}(B,B) \longleftarrow \star$$ given by $(f,g) \mapsto (gf,fg)$ and $(id_A,id_B) \leftarrow \star$ (if the notation is not clear, ask). Let $\underline{\mathrm{Isom}}(A,B)$ be their pullback. The composition $\underline{\mathrm{Isom}}(A,B) \to \underline{\mathrm{Hom}}(A,B) \times \underline{\mathrm{Hom}}(B,A) \to \underline{\mathrm{Hom}}(A,B)$ is a monomorphism, which exhibits the subobject of isomorphisms from $A$ to $B$.

For $A=B$ I wouldn't use $A!$ as a notation, but rather $\underline{\mathrm{Aut}}(A)$.

A typical example is $\mathcal{C}=\mathsf{Sh}(X)$, the category of sheaves on a space $X$. If $A,B$ are sheaves on $X$, the isomorphism sheaf $\underline{\mathrm{Isom}}(A,B)$ is the subsheaf of the usual homomorphism sheaf $\underline{\mathrm{Hom}}(A,B)$ whose sections on $U \to X$ are the isomorphisms $A|_U \to B|_U$.