Is there something wrong with this question concerning Groups
I checked the matrices in your subgroup $H$, and I found that two of them had order 6, two of them had order 3, one had order 2, and one had order 1. This tells me that $H$ is not isomorphic to $D_6$ because $D_6$ does not have any elements of order 6.
You claim that this subgroup is not Abelian. While this is true for matrices in general, certain special matrices do in fact commute. All the matrices in this subgroup commute, so the subgroup is Abelian.
I would advise performing the multiplications again and remember to mod the entries by 3. It is easy to make an error.