Which file sharing protocol (SMB/AFS/NFS/SMB2/SMB3) is best for an mixed client environment within an ad?

SMB / CIFS / Samba

Seems to be the only real option in an real mixed environment:

• Accessible by all major platforms (Mac, Windows, Unix)
• Mac increases support for SMB3 a lot.

Maybe opinionated but from my experience:

• However poor performance (slow) compared to AFP / NFS / ZFS Replication.
• CPU performance matters. SMB is single threaded, meaning one thread per connection. Thus the speed of one core decides of the maximum data transfer performance of one connection. However, multiple clients with multiple connections result in an overall higher transfer rate, since multiple threads can be spread about multiple CPU cores.
• RAM hungry
• Scalability limited. Scaling through better, expensive hardware.
• Additional features and logging slows down samba alot.
• Rough configuration with a lot of try and error on a linux based server. Buggy rights on Linux based server. As Snellgrove described you're probably are better of hosting a file server on windows.

NFS (Network File System)

Originally comming from the unix/linux world, it has really poor integration into windows and mac. Neither of them supports it straig away and needs manual adjustments. Performance seems to be faster than SMB though.

It is better suitable for server to server file transfer. However, depending on the situation often file system replication like zfs replication, or rsync are better alternatives for this.

AFP (Apple File Protocol) - Winner for Max

Only accessible by Mac, thus not usable in a mixed environment, however if you have shares witch are only accessible from mac, you're better of with AFP for now. However this may change with future version of samba/cifs and mac's support for it

• Much faster transfer than SMB on single connections
• Server side search
• More stable saving to netdrives from mac applications
• Live change updates (no refresh of folders, changed and new files appear by magic)
• Time Machine Backup Support

If you have a heterogenous network, with Windows, Macs, *nix etc then SMB is probably your only real option as it is compatible with just about everything.

With regard to Active Directory, your file protocol selection has little bearing on it unless you're using share permissions and the like, with objects from yourdomain\userAccount

You can of course get away with local permissions in a domain environment - a Microsoft AD installation does not stop you doing that. I am guessing you want to leverage this though, or you wouldn't mention it. So I'm thinking you have a primarily Microsoft infrastructure.

Based on that, I'd aim it like this:

Use NTFS as your underlying file system if you can (I know you've said you're using FreeNAS so that's probably out the window, but I'm just stating it as it's probably best practise for this kind of thing. Personally I'd throw that away and put in a Microsoft Windows Server based file server) - it's very good and you can set the ACLs to use domain\useraccounts, use SMB for the actual sharing protocol and make sure your clients are compatible with this infrastructure, not the other way round.

I may sound biased towards Microsoft here but I don't work for them or anything, it just sounds to me that you are a Microsoft house, with non-microsoft clients. So use the best tool for the job.

In Summary: use SMB! if CPU becomes an issue, get better hardware.