What's the meaning of this sentence? "Don't grind your pearls"

To Grind your pearls is a casual reference to grinding one's teeth. The man is telling his friend to pay no attention to the ladies as it would make him frustrated and likely to grind his teeth, unproductively. Despite being common enough I've been unable to nail down a reference.