What is an unambiguous single word for a five-pointed star without enclosure?

To reiterate what you described, in strict usage the following apply:

This is a pentagram – a type of five-pointed star

This is a pentacle – a pentagram circumscribed by a circle:

This is a five-pointed star that is not a pentagram:

Yes, as you described, strict usage is not always the case, and when you say pentagram it evokes pentacle in the minds of many people. However, I am convinced that there is no single word that is both common and unambiguously understood to mean the first case. I am assuming that since you consider five-pointed star "a bit of a mouthful" that you mean to repeat this term at least a few times in whatever you're writing. If I were in your place, I would use the term pentagram (since after all, it is correct in formal usage), and use a picture to show the reader exactly what you mean. If you are writing in a medium where you cannot use pictures, I would spend a sentence to describe it as a five-pointed star drawn with five straight strokes.

According to Wikipedia, it is a Pentagram.

Wikipedia points out that Pentacles are magical objects and

despite the sound of the word, often had no connotation of "five" in the old magical texts, but were, rather, magical talismans inscribed with any symbol or character.

Doczi in 'The Power of Limits' p 6, viewable here, distinguishes between a pentagon, a pentagram and a pentagonal star (this last (highlighted in bold) refers to your (1) above). Unfortunately it's 2 words, not 1.