Why cant a group member modify a file in a group he is part of?

You setup the groups correctly, but the user's list of groups is set when they first login, and won't normally ever be re-evaluated. If you make a change to a user's list of groups, you either need to log out and log back in, or you need to run the newgrp command.