Is it viable to buy AP items with Shen?

Somewhat viable to a certain extent, but not to the extent of getting Rabadons, for instance. He gets so much more from stacking HP and resistances, since he is, in essence, a damage-soaking tank. His kit (a shield, a taunt and a global shield-teleport) make it a must to build tanky.

Pro-tip though: (This is not for Shen jungle, but for Shen Top.) Build a rylai's crystal scepter if you get ahead, which you should, since you are Shen after all. That bonus AP and Health are great for Shen, and the 35% slow on Vorpal Blade is just too much. That slow + taunt and you'll shut-down the enemy carries every time.

I suppose you could get things like zhonya's, abyssal or even DFG but those would be tremendously not cost-effective on Shen.

I think it's viable, but it ends at Rylai's and masteries 9/21/0 going for the MPen.