What are the differences between Linux and Windows .txt files (Unicode encoding)

"Unicode" on Windows is UTF-16LE, and each character is 2 or 4 bytes. Linux uses UTF-8, and each character is between 1 and 4 bytes.

"The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)"

Line breaks

Windows uses CRLF (\r\n, 0D 0A) line endings while Unix just uses LF (\n, 0A).

Character Encoding

Most modern (i.e., since 2004 or so) Unix-like systems make UTF-8 the default character encoding.

Windows, however, lacks native support for UTF-8. It internally works in UTF-16, and assumes that char-based strings are in a legacy code page. Fortunately, Notepad is capable of reading UTF-8 files; unfortunately, "ANSI" encoding is still the default.

Problematic Special Characters

U+001A SUBSTITUTE

Windows (rarely) uses Ctrl+Z as an end-of-file character. For example, if you type a file at the command prompt, it will be truncated at the first 1A byte.

On Unix, Ctrl+Z is nothing special.

U+FEFF ZERO WITH NO-BREAK SPACE (Byte-Order Mark)

On Windows, UTF-8 files often start with a "byte order mark" EF BB BF to distinguish them from ANSI files.

On Linux, the BOM is discouraged because it breaks things like shebang lines in shell scripts. Plus, it'd be pointless to have a UTF-8 signature when UTF-8 is the default encoding anyway.

One difference I've hear is the use of \r\n (Windows) vs. \n for line breaks (Linux).

Yes. Most UNIX text editors will handle this automatically, Windows programmers editors may handle this, general text editors (base Notepad) will not.

Windows seems to also need the EOF (Ctrl-Z) as END OF FILE in some contexts, whereas you'll probably never see it on UNIX.

Remember that MacOS X is now UNIX underneath, so it uses UNIX line endings. Though before OS X (MacOS 9 and below) it had its own ending (\r)

EDIT: in other format CR and LF:

• \n is ASCII 0x0A, Line Feed (LF)
• \r is ASCII 0x0D, Carriage return (CR)

What Unicode encoding is used is not OS based.

Even Windows notepad.exe has options listed- (i'll put in brackets what notepad means by that) ANSI(not unicode), Unicode(notepad means Unicode LE), Unicode Big Endian(BE), UTF-8

ANSI isn't unicode it involves a very limited number of characters so lets put that aside.

But see even notepad can do LE, or BE, or UTF-8

And notepad aside, UTF-8 can be with or without a BOM.

And I use Windows with Cygwin though Windows ports may well do \r\n even when you specify \n Have seen sed do that.

There is no one rule of what Unicode encoding a particular OS uses. It wouldn't be a very flexible OS if there was.

To really see the differences know the Software, what Encoding a piece of software uses or offers.

Get Cygwin and xxd, and/or a hex editor and look at what is really inside the file. Use the 'file' command to help identify a file. Then you actually see what UTF 16bit LE is. What UTF 16bit BE is. What UTF-8 is (and UTF-8 can be with or without a BOM).

Sometimes you can tell notepad to save as unicode(by which notepad means unicode 16 bit little endian), and it won't. But choose a unicode font like arial unicode, and copy in some unicode characters from charmap and it will.. And a good way to see what notepad or whatever software is doing, is by looking at the hex of a file

C:\asdf>notepad.exe a.a

C:\asdf>file a.a
a.a; Little-endian UTF-16 Unicode text, with no line terminators

C:\asdf>type a.a
aaa慡ൡ <-- though displayed aaa followed by some boxes in my cmd window
C:\asdf>

C:\asdf>xxd a.a
0000000: fffe 6100 6100 6100 6161 610d            ..a.a.a.aaa.

C:\asdf>

^^ The portion of the byte that stores the 61 is the lower value portion which with LE is stored first.

The dd command (a *nix command I run from cygwin within windows) can switch it

C:\asdf>xxd -p a.a
fffe6100610061006161610d

C:\asdf>file a.a
a.a; Little-endian UTF-16 Unicode text, with no line terminators

C:\asdf>dd if=a.a conv=swab of=a.a2
0+1 records in
0+1 records out
12 bytes (12 B) copied, 0 seconds, Infinity B/s

C:\asdf>type a.a2
a  a a aaa
C:\asdf>xxd -p a.a2
feff00610061006161610d61

C:\asdf>file a.a2
a.a2; Big-endian UTF-16 Unicode text, with no line terminators

C:\asdf>

And notepad itself can save as UTF-16 Big Endian or UTF-16 Little Endian or UTF-8

If you're a technical person or even just a notepad user, you're not bound to one encoding because of your OS!

I suppose UTF-8 makes more sense than UTF-16, UTF-16 would use 16 bits even for characters that should only need 8 bits. Also though, bear in mind that charmap shows the UTF-16 code.

Sublime(A windows text editor) saves unicode as UTF-8 by default.

I use Windows and sometimes unicode, and i'm using UTF-8 mostly.

And as Windows is that technically flexible, linux is at least as technically flexible!