unicode().decode('utf-8', 'ignore') raising UnicodeEncodeError
When I first started messing around with python strings and unicode, It took me awhile to understand the jargon of decode and encode too, so here's my post from here that may help:
Think of decoding as what you do to go from a regular bytestring to unicode and encoding as what you do to get back from unicode. In other words:
You de-code a str
to produce a unicode
string (in Python 2)
and en-code a unicode
string to produce a str
(in Python 2)
So:
unicode_char = u'\xb0'
encodedchar = unicode_char.encode('utf-8')
encodedchar
will contain your unicode character, displayed in the selected encoding (in this case, utf-8
).
The same principle applies to Python 3. You de-code a bytes
object to produce a str
object. And you en-code a str
object to produce a bytes
object.
From http://wiki.python.org/moin/UnicodeEncodeError
Paradoxically, a UnicodeEncodeError may happen when decoding. The cause of it seems to be the coding-specific decode() functions that normally expect a parameter of type str. It appears that on seeing a unicode parameter, the decode() functions "down-convert" it into str, then decode the result assuming it to be of their own coding. It also appears that the "down-conversion" is performed using the ASCII encoder. Hence an encoding failure inside a decoder.
You're trying to decode a unicode
. The implicit encoding to make the decode work is what's failing.