What is the most accurate encoding detector? [closed]

Solution 1:

I've checked juniversalchardet and ICU4J on some CSV files, and the results are inconsistent: juniversalchardet had better results:

• UTF-8: Both detected.
• Windows-1255: juniversalchardet detected when it had enough hebrew letters, ICU4J still thought it was ISO-8859-1. With even more hebrew letters, ICU4J detected it as ISO-8859-8 which is the other hebrew encoding(and so the text was OK).
• SHIFT_JIS(Japanese): juniversalchardet detected and ICU4J thought it was ISO-8859-2.
• ISO-8859-1: detected by ICU4J, not supported by juniversalchardet.

So one should consider which encodings he will most likely have to deal with. In the end I chose ICU4J.

Notice that ICU4J is still maintained.

Also notice that you may want to use ICU4J, and in case that it returns null because it didn't succeed, try to use juniversalchardet. Or the opposite.

AutoDetectReader of Apache Tika does exactly this - first tries to use HtmlEncodingDetector, then UniversalEncodingDetector(which is based on juniversalchardet), and then tries Icu4jEncodingDetector(based on ICU4J).

Solution 2:

I found an answer online:

http://fredeaker.blogspot.com/2007/01/character-encoding-detection.html

It says something vealuable here:

The strength of a character encoding detector lies in whether or not its focus is on statistical analysis or HTML META and XML prolog discovery. If you are processing HTML files that have META, use cpdetector. Otherwise, your best bet is either monq.stuff.EncodingDetector or com.sun.syndication.io.XmlReader.

So that's why I am using cpdetector now. I will update the post with the result of it.

Solution 3:

I've personally used jchardet in our project (juniversalchardet wasn't available back then) just to check if a stream was UTF-8 or not.

It was easier to integrate with our application than the other and yielded great results.