When do I start finding gems?

Solution 1:

I found both my first socketed item and my first gem in Act II after I

killed Magdha.

However, both @Sterno and @OrigamiRobot found gems before the activity above - so far, it seems Act II is the only real concrete limitation so far.

However, both OrigamiRobot and @Resorath confirm a similar experience (a gem dropping off the activity above). It seems that a gem will always drop from said mob. It does not seem a socketed item drop is guaranteed, though.

It's a few more quests until you find the Jewelry Artisan, Covetous Shen, but it is shortly thereafter in the same Act. You can find gems before him, and before the activity above.

Solution 2:

You start getting gems in Act 2, around the time you meet up with the Jewelry Artisan. I got my first gems shortly before I met him.

It is likely that this is intentional, as a way of introducing you to the concept in a cohesive fashion (giving you both the gems, and the means of improving and reusing them at roughly the same time).