Word that means "private word used by a group"

Solution 1:

I would call the word itself a pet term. This is an interesting topic, and pet terms are probably common within families and other small, tight-knit groups.

An example of this usage, in a headline from a media analysis website:

Limbaugh Explains His Pet Term "New Castrati": Men Who Are "Bullied By Women And The Power Structure And Liberalism"

Note that this is not the same as a term of endearment.

EDIT: This is not an established linguistic term. I consulted a few sociolinguistics textbooks and didn’t find any discussion of this kind of ephemeral in-group language. One place to look would be studies of college slang, like Connie Eble’s Slang and Sociability.

Pet term is also different from pet word, which seems to have a generally accepted meaning: a word that is frequently used or otherwise favored by an individual person or writer. Pet words are already part of the general lexicon, though. Brad Leithauser has a 2013 New Yorker piece about pet words. He gives some examples (sweet for Shakespeare, lad for A.E. Housman) and compares them to stray cats taken in by their users:

Each of these words presents the critic with a little puzzle of devotion: What was it about this particular package of syllables? Why was this stray cat escorted into the author’s studio and offered a saucer of cream and a plump pillow by the fireplace? It’s not as though the studio were soundproof; during working hours, the author no doubt could hear other strays, seemingly no less deserving, meowing clamorously for admission.

Link: Pet Words

So, pet term (rather than word) can convey that it's favored by the in-group, but not established as a true word yet.

Solution 2:

I prefer "Shibboleth". It has a great history and REALLY closely matches the context-specific nature of your request.

Per Wikipedia: a word or custom whose variations in pronunciation or style can be used to differentiate members of ingroups from those of outgroups. Within the mindset of the ingroup, a connotation or value judgment of correct/incorrect or superior/inferior can be ascribed to the two variants.

http://en.wikipedia.org/wiki/Shibboleth

Solution 3:

I guess argot:

special words or expressions used by a profession or group that are difficult for others to understand.