Is there any way of telling if a Pokémon in the GTS is hacked before you trade?

Solution 1:

No.

Besides a few telltale obvious signs that something is probably hacked, like being level 100, shiny, and in a hard-to-get (or impossible) pokéball, even if you have the pokemon in your party, there's no 100% way. A pokémon's data structure is not very big or complicated, and if you know how it works, it's not difficult to make it do what you want. A well-hacked pokémon is indistinguishable from an identical legitimate one.

If you want to make sure you are only trading for legitimate shinies, you pretty much have to trade in regulated communities, like forums.