Why are emoji characters like 👩‍👩‍👧‍👦 treated so strangely in Swift strings?

Solution 1:

This has to do with how the String type works in Swift, and how the contains(_:) method works.

The '👩‍👩‍👧‍👦 ' is what's known as an emoji sequence, which is rendered as one visible character in a string. The sequence is made up of Character objects, and at the same time it is made up of UnicodeScalar objects.

If you check the character count of the string, you'll see that it is made up of four characters, while if you check the unicode scalar count, it will show you a different result:

print("👩‍👩‍👧‍👦".characters.count)     // 4
print("👩‍👩‍👧‍👦".unicodeScalars.count) // 7

Now, if you parse through the characters and print them, you'll see what seems like normal characters, but in fact the three first characters contain both an emoji as well as a zero-width joiner in their UnicodeScalarView:

for char in "👩‍👩‍👧‍👦".characters {
    print(char)

    let scalars = String(char).unicodeScalars.map({ String($0.value, radix: 16) })
    print(scalars)
}

// 👩‍
// ["1f469", "200d"]
// 👩‍
// ["1f469", "200d"]
// 👧‍
// ["1f467", "200d"]
// 👦
// ["1f466"]

As you can see, only the last character does not contain a zero-width joiner, so when using the contains(_:) method, it works as you'd expect. Since you aren't comparing against emoji containing zero-width joiners, the method won't find a match for any but the last character.

To expand on this, if you create a String which is composed of an emoji character ending with a zero-width joiner, and pass it to the contains(_:) method, it will also evaluate to false. This has to do with contains(_:) being the exact same as range(of:) != nil, which tries to find an exact match to the given argument. Since characters ending with a zero-width joiner form an incomplete sequence, the method tries to find a match for the argument while combining characters ending with a zero-width joiners into a complete sequence. This means that the method won't ever find a match if:

the argument ends with a zero-width joiner, and
the string to parse doesn't contain an incomplete sequence (i.e. ending with a zero-width joiner and not followed by a compatible character).

To demonstrate:

let s = "\u{1f469}\u{200d}\u{1f469}\u{200d}\u{1f467}\u{200d}\u{1f466}" // 👩‍👩‍👧‍👦

s.range(of: "\u{1f469}\u{200d}") != nil                            // false
s.range(of: "\u{1f469}\u{200d}\u{1f469}") != nil                   // false

However, since the comparison only looks ahead, you can find several other complete sequences within the string by working backwards:

s.range(of: "\u{1f466}") != nil                                    // true
s.range(of: "\u{1f467}\u{200d}\u{1f466}") != nil                   // true
s.range(of: "\u{1f469}\u{200d}\u{1f467}\u{200d}\u{1f466}") != nil  // true

// Same as the above:
s.contains("\u{1f469}\u{200d}\u{1f467}\u{200d}\u{1f466}")          // true

The easiest solution would be to provide a specific compare option to the range(of:options:range:locale:) method. The option String.CompareOptions.literal performs the comparison on an exact character-by-character equivalence. As a side note, what's meant by character here is not the Swift Character, but the UTF-16 representation of both the instance and comparison string – however, since String doesn't allow malformed UTF-16, this is essentially equivalent to comparing the Unicode scalar representation.

Here I've overloaded the Foundation method, so if you need the original one, rename this one or something:

extension String {
    func contains(_ string: String) -> Bool {
        return self.range(of: string, options: String.CompareOptions.literal) != nil
    }
}

Now the method works as it "should" with each character, even with incomplete sequences:

s.contains("👩")          // true
s.contains("👩\u{200d}")  // true
s.contains("\u{200d}")    // true

Solution 2:

The first problem is you're bridging to Foundation with contains (Swift's String is not a Collection), so this is NSString behavior, which I don't believe handles composed Emoji as powerfully as Swift. That said, Swift I believe is implementing Unicode 8 right now, which also needed revision around this situation in Unicode 10 (so this may all change when they implement Unicode 10; I haven't dug into whether it will or not).

To simplify thing, let's get rid of Foundation, and use Swift, which provides views that are more explicit. We'll start with characters:

"👩‍👩‍👧‍👦".characters.forEach { print($0) }
👩‍
👩‍
👧‍
👦

OK. That's what we expected. But it's a lie. Let's see what those characters really are.

"👩‍👩‍👧‍👦".characters.forEach { print(String($0).unicodeScalars.map{$0}) }
["\u{0001F469}", "\u{200D}"]
["\u{0001F469}", "\u{200D}"]
["\u{0001F467}", "\u{200D}"]
["\u{0001F466}"]

Ah… So it's ["👩ZWJ", "👩ZWJ", "👧ZWJ", "👦"]. That makes everything a bit more clear. 👩 is not a member of this list (it's "👩ZWJ"), but 👦 is a member.

The problem is that Character is a "grapheme cluster," which composes things together (like attaching the ZWJ). What you're really searching for is a unicode scalar. And that works exactly as you're expecting:

"👩‍👩‍👧‍👦".unicodeScalars.contains("👩") // true
"👩‍👩‍👧‍👦".unicodeScalars.contains("\u{200D}") // true
"👩‍👩‍👧‍👦".unicodeScalars.contains("👧") // true
"👩‍👩‍👧‍👦".unicodeScalars.contains("👦") // true

And of course we can also look for the actual character that is in there:

"👩‍👩‍👧‍👦".characters.contains("👩\u{200D}") // true

(This heavily duplicates Ben Leggiero's points. I posted this before noticing he'd answered. Leaving in case it is clearer to anyone.)

Solution 3:

It seems that Swift considers a ZWJ to be an extended grapheme cluster with the character immediately preceding it. We can see this when mapping the array of characters to their unicodeScalars:

Array(manual.characters).map { $0.description.unicodeScalars }

This prints the following from LLDB:

▿ 4 elements
  ▿ 0 : StringUnicodeScalarView("👩‍")
    - 0 : "\u{0001F469}"
    - 1 : "\u{200D}"
  ▿ 1 : StringUnicodeScalarView("👩‍")
    - 0 : "\u{0001F469}"
    - 1 : "\u{200D}"
  ▿ 2 : StringUnicodeScalarView("👧‍")
    - 0 : "\u{0001F467}"
    - 1 : "\u{200D}"
  ▿ 3 : StringUnicodeScalarView("👦")
    - 0 : "\u{0001F466}"

Additionally, .contains groups extended grapheme clusters into a single character. For instance, taking the hangul characters ᄒ, ᅡ, and ᆫ (which combine to make the Korean word for "one": 한):

"\u{1112}\u{1161}\u{11AB}".contains("\u{1112}") // false

This could not find ᄒ because the three codepoints are grouped into one cluster which acts as one character. Similarly, \u{1F469}\u{200D} (WOMAN ZWJ) is one cluster, which acts as one character.

Solution 4:

The other answers discuss what Swift does, but don't go into much detail about why.

Do you expect “Å” to equal “Å”? I expect you would.

One of these is a letter with a combiner, the other is a single composed character. You can add many different combiners to a base character, and a human would still consider it to be a single character. To deal with this sort of discrepancy the concept of a grapheme was created to represent what a human would consider a character regardless of the codepoints used.

Now text messaging services have been combining characters into graphical emoji for years :) → 🙂. So various emoji were added to Unicode.
These services also started combining emoji together into composite emoji.
There of course is no reasonable way to encode all possible combinations into individual codepoints, so The Unicode Consortium decided to expand on the concept of graphemes to encompass these composite characters.

What this boils down to is "👩‍👩‍👧‍👦" should be considered as a single "grapheme cluster" if you trying to work with it at the grapheme level, as Swift does by default.

If you want to check if it contains "👦" as a part of that, then you should go down to a lower level.

I don't know Swift syntax so here is some Perl 6 which has similar level of support for Unicode.
(Perl 6 supports Unicode version 9 so there may be discrepancies)

say "\c[family: woman woman girl boy]" eq "👩‍👩‍👧‍👦"; # True

# .contains is a Str method only, in Perl 6
say "👩‍👩‍👧‍👦".contains("👩‍👩‍👧‍👦")    # True
say "👩‍👩‍👧‍👦".contains("👦");        # False
say "👩‍👩‍👧‍👦".contains("\x[200D]");  # False

# comb with no arguments splits a Str into graphemes
my @graphemes = "👩‍👩‍👧‍👦".comb;
say @graphemes.elems;                # 1

Let's go down a level

# look at it as a list of NFC codepoints
my @components := "👩‍👩‍👧‍👦".NFC;
say @components.elems;                     # 7

say @components.grep("👦".ord).Bool;       # True
say @components.grep("\x[200D]".ord).Bool; # True
say @components.grep(0x200D).Bool;         # True

Going down to this level can make some things harder though.

my @match = "👩‍👩‍👧‍👦".ords;
my $l = @match.elems;
say @components.rotor( $l => 1-$l ).grep(@match).Bool; # True

I assume that .contains in Swift makes that easier, but that doesn't mean there aren't other things which become more difficult.

Working at this level makes it much easier to accidentally split a string in the middle of a composite character for example.

What you are inadvertently asking is why does this higher level representation not work like a lower level representation would. The answer is of course, it's not supposed to.

If you are asking yourself “why does this have to be so complicated”, the answer is of course “humans”.

Why are emoji characters like 👩‍👩‍👧‍👦 treated so strangely in Swift strings?

Solution 1:

Solution 2:

Solution 3:

Solution 4:

Related

Recent Posts