lower() vs. casefold() in string matching and converting to lowercase

TL;DR

• Purely ASCII Text -> lower()
• Unicode text/user input -> casefold()

Casefolding is a more aggressive version of lower() that is set up to make many of the more unique unicode characters more comparable. It is another form of normalizing text that may initially appear to be very different, but it takes characters of many different languages into account.

I suggest you take a closer look into what case folding actually is, so here's a good start: W3 Case Folding Wiki

To answer your other two questions, if you are working strictly in the English language, lower() and casefold() should be yielding exactly the same results. However, if you are trying to normalize text from other languages that use more than our simple 26-letter alphabet (using only ASCII), I would use casefold() to compare your strings, as it will yield more consistent results.

Another source: Elastic.co Case Folding

Edit: I just recently found another very good related answer to a slightly different question here on SO (doing a case-insensitive string comparison)

Another Edit: @Voo's comments have been bouncing around in the back of my mind for a few months, so here are some further thoughts:

As Voo mentioned, there aren't any languages that never use text outside the standard ASCII values. That's pretty much why Unicode exists. With that in mind, it makes more sense to me to use casefold() on anything that is user-entered that can contain non-ascii values. This might end up excluding some text that might come from a database that strictly deals with ASCII, but, in general, probably most user input would be dealt with using casefold() because it has the logic to properly de-uppercase all of the characters.

On the other hand, values that are known to be generated into the ASCII character space like hex UUIDs or something like that should be normalized with lower() because it is a much simpler transformation. Simply put, lower() will require less memory or less time because there are no lookups, and it's only dealing with 26 characters it has to transform. Additionally, if you know that the source of your information is coming from a CHAR or VARCHAR (SQL Server fields) database field, you can similarly just use lower because Unicode characters can't be entered into those fields.

So really, this question comes down to knowing the source of your data, and when in doubt about your user-entered information, just casefold().

lower() vs casefold() and when to use, Details are given below.