What is the word for a group holding back one of its members trying to rise above the group?

This is often called crab mentality, due to the tendency of crabs in a bucket to pull each other down whenever one tries to escape. The Wikipedia article on crab mentality says it is "best described by the phrase 'If I can't have it, neither can you.'"

It probably isn't the word you heard, but the phenomenon is often refered to as the tall poppy syndrome or cutting down the tall poppies. It is probably an allusion to a legend of Lucius Tarquinius Superbus:

His son, Sextus, pretending to be ill-treated by his father, and covered with the bloody marks of stripes, fled to Gabii. The infatuated inhabitants entrusted him with the command of their troops, and when he had obtained the unlimited confidence of the citizens, he sent a messenger to his father to inquire how he should deliver the city into his hands. The king, who was walking in his garden when the messenger arrived, made no reply, but kept striking off the heads of the tallest poppies with his stick. Sextus took the hint, and put to death, or banished on false charges, all the leading men of Gabii, after which he had no difficulty in compelling the city to submit.
en.wikipedia.org emphasis mine