Can a valid Unicode string contain FFFF? Is Java/CharacterIterator broken?

Solution 1:

EDIT (2013-12-17): Peter O. brings up an excellent point below, which renders this answer wrong. Old answer below, for historical accuracy.

Answering your questions:

Is the prescribed traversal idiom "broken" because it makes the wrong assumption about \uFFFF?

No. U+FFFF is a so-called non-character. From Section 16.7 of the Unicode Standard:

Noncharacters are code points that are permanently reserved in the Unicode Standard for internal use. They are forbidden for use in open interchange of Unicode text data.

...

The Unicode Standard sets aside 66 noncharacter code points. The last two code points of each plane are noncharacters: U+FFFE and U+FFFF on the BMP, U+1FFFE and U+1FFFF on Plane 1, and so on, up to U+10FFFE and U+10FFFF on Plane 16, for a total of 34 code points. In addition, there is a contiguous range of another 32 noncharacter code points in the BMP: U+FDD0..U+FDEF.

Is the StringCharacterIterator implementation "broken" because it doesn't e.g. throw an IllegalArgumentException if in fact \uFFFF is forbidden in valid Unicode strings?

Not quite. Applications are allowed to use those code points internally in any way they want. Quoting the standard again:

Applications are free to use any of these noncharacter code points internally but should never attempt to exchange them. If a noncharacter is received in open interchange, an application is not required to interpret it in any way. It is good practice, however, to recognize it as a noncharacter and to take appropriate action, such as replacing it with U+FFFD REPLACEMENT CHARACTER, to indicate the problem in the text. It is not recommended to simply delete noncharacter code points from such text, because of the potential security issues caused by deleting uninterpreted characters.

So while you should never encounter such a string from the user, another application or a file, you may well put it into a Java String if you know what you're doing (this basically means that you cannot use the CharacterIterator on that string, though.

Is it actually true that valid Unicode strings should not contain \uFFFF?

As quoted above, any string used for interchange must not contain them. Within your application you're free to use them in whatever way they want.

Of course, a Java char, being just a 16-bit unsigned integer doesn't really care about the value it holds as well.

If that's true, then is Java "broken" for violating the Unicode specification by (for the most parts) allowing String to contain \uFFFF anyway?

No. In fact, the section on noncharacters even suggests the use of U+FFFF as sentinel value:

In effect, noncharacters can be thought of as application-internal private-use code points. Unlike the private-use characters discussed in Section 16.5, Private-Use Characters, which are assigned characters and which are intended for use in open interchange, subject to interpretation by private agreement, noncharacters are permanently reserved (unassigned) and have no interpretation whatsoever outside of their possible application-internal private uses.

U+FFFF and U+10FFFF. These two noncharacter code points have the attribute of being associated with the largest code unit values for particular Unicode encoding forms. In UTF-16, U+FFFF is associated with the largest 16-bit code unit value, FFFF₁₆. U+10FFFF is associated with the largest legal UTF-32 32-bit code unit value, 10FFFF₁₆. This attribute renders these two noncharacter code points useful for internal purposes as sentinels. For example, they might be used to indicate the end of a list, to represent a value in an index guaranteed to be higher than any valid character value, and so on.

CharacterIterator follows this in that it returns U+FFFF when no more characters are available. Of course, this means that if you have another use for that code point in your application you may consider using a different non-character for that purpose since U+FFFF is already taken – at least if you're using CharacterIterator.

Solution 2:

Some of these answers have changed in the meantime.

The Unicode Consortium recently issued Corrigendum 9 that clarifies the role of noncharacters, including U+FFFF, in Unicode strings. It states that while noncharacters are intended for internal use, they can occur legally in Unicode strings.

That means the statement "The value is \uFFFF, the 'not a character' value which should not occur in any valid Unicode string." is now incorrect, since U+FFFF can occur in valid Unicode strings.

Accordingly:

Is the StringCharacterIterator implementation "broken" because it doesn't throw an exception if \uFFFF is forbidden in valid Unicode strings? Since U+FFFF is valid, this doesn't apply here. But an implementation has wide flexibility in signaling an error when it encounters text that's illegal for other reasons, such as unpaired surrogate code points, which still remain illegal (see conformance clause C10 in chapter 3 of the Unicode Standard).
Is it true that valid Unicode strings should not contain \uFFFF? U+FFFF is not illegal in a valid Unicode string.

However U+FFFF is reserved as a noncharacter and so will generally not occur in meaningful text. The corrigendum deleted the text that noncharacters "should never be interchanged", which the corrigendum says happens "anytime a Unicode string crosses an API boundary", including the StringCharacterIterator API at issue here.
If that's true, then is Java "broken" for violating the Unicode specification by allowing String to contain \uFFFF anyway? The specification for java.lang.String says "A String represents a string in the UTF-16 format." U+FFFF is legal in a Unicode string, so Java doesn't violate Unicode for allowing U+FFFF in a string containing it.

In general, a higher-level protocol can impose its own rules on top of the Unicode Standard, on the question of which characters are allowed in documents accepted by the protocol. This is the case, for example, in the XML specification. In general, U+FFFF (and other Unicode scalar values) can validly appear in a text string unless a higher-level protocol (such as XML) specifies otherwise. Indeed, there is a current effort (as of November 15, 2021) to restrict the use of Unicode bidirectional override characters in certain programming languages such as Rust, to reduce security attacks due to visual confusion.