Issues with files and folders with names containing umlauts

I have just noticed some issues browsing my external music collection using Finder, certain artists folders were missing.

Going to iTerm and lsing the folder I got this as the initial report:

ls Music
ls: Maná: No such file or directory
ls: Márta Sebestyén: No such file or directory
ls: Motörhead: No such file or directory
ls: Zoë Keating: No such file or directory

Note that the artists names use umlauts and Double acute accents.

This is a folder on a NAS box (ReadyNAS 204) via smb, I also have back up of the NAS running Ubuntu access via CIFS.

Using Finder on OS X El Capitan neither the NAS or backup show these folders. However using Yosemite I can see the Artists folder but the albums are missing.

Logging in directly to the boxes all files appear to be intact and SONOS can still play them!

Has some thing changed in OS X, I used max to rip these from CD's (Snow Leopard) so I had to be able to handle the files at some point ‽

NB: ‽ is an interrobang

Update with a little more info

Using rsync from the Mac to copy music on to USB (OS X Extended format) drive I see things like:

rsync -av --iconv=utf8-mac,utf8 /Volumes/share/Music/ /Volumes/usbdisk/Music/

file has vanished: "/Volumes/share/Music/.../Márta Sebestyén"
file has vanished: "/Volumes/share/Music/.../Maná"
file has vanished: "/Volumes/share/Music/.../Zoë Keating"
file has vanished: "/Volumes/share/Music/.../Rammstein-2005-Rosenrot-07-Zerstören.flac"

ssh'ing on to Ubuntu server and using convmv as described here:

cd /mnt/Music
convmv -r -f ISO-8859-1 -t UTF-8 . 

It returns :

Starting a dry run without changes...
Skipping, already UTF-8: ./Márta Sebestyén
Skipping, already UTF-8: ./Maná
Skipping, already UTF-8: ./Michael Bublé
Skipping, already UTF-8: ./Zoë Keating
Skipping, already UTF-8: ./Rammstein/1999-Live aus Berlin/Rammstein-1999-Live aus Berlin-10-Bück dich.flac
Skipping, already UTF-8: ./Rammstein/1999-Live aus Berlin/Rammstein-1999-Live aus Berlin-03-Weißes Fleisch.flac
No changes to your files done. Use --notest to finally rename the files.

It looks like the main files having issues are already utf-8 but note the file "Rammstein-2005-Rosenrot-07-Zerstören.flac" that rsync had issue with is not listed in the above report.

I also used brew to install the latest vesion of rsync 3+ rather than the 2.6 version that is used be default.

brew tap homebrew/dupes
brew install rsync

And then edit /private/etc/paths to put /usr/local/bin before /usr/bin

Trying out AFP

Enabling AFP on Ready NAS (OS 6.4.2), well this was a little freaky for me. Structure is Artist/Album/Tracks.flac. Artists are there, album was there, expand the album to see tracks the Artists folder disappeared.

ReadyNAS smb.conf

[global]
 dos charset = CP1252
 unix charset = UTF-8

Related Info

Folders not showing up in Finder?
CIFS & MAC problems with special characters in folder names .

Solution 1:

macOS and Linux use different UTF-8 styles. convmv can convert between those two with the options --nfc / --nfd. On your linux based NAS do this:

convmv -r -f UTF-8 -t UTF-8 --nfc  .

From the convmv man page:

HFS+ on OS X / Darwin Linux and (most?) other Unix-like operating systems use the so called normalization form C (NFC) for its UTF-8 encoding by default but do not enforce this. HFS+ on the Macintosh OS enforces normalization form D (NFD), where a few characters are encoded in a different way. On OS X it's not possible to create NFC UTF-8 filenames because this is prevented at filesystem layer. On HFS+ filenames are internally stored in UTF-16 and when converted back to UTF-8 (because the Unix based OS can't deal with UTF-16 directly), NFD is created for whatever reason. See http://developer.apple.com/qa/qa2001/qa1173.html for defails. I think it was a very bad idea and breaks many things under OS X which expect a normal POSIX conforming system. Anywhere else convmv is able to convert files from NFC to NFD or vice versa which makes interoperability with such systems a lot easier.

Btw: This only happens when you transfer your files from a HFS+ partition. APFS doesn't enforce NFD anymore.