How to get optical 5.1 surround audio output Retina MacBook Pro from Logic Pro X
Solution 1:
The standard TOSLINK Spec (without utilizing the extensions) was commercialized in the 1980s and only has a bandwidth of 3.1 Mbits/s at the ground floor. This can't quite manage 5.1 uncompressed audio with overhead, especially at >= 48k/16. So this isn't a DSP issue, its a bandwidth issue. The TOSLINK hardware link layer just can't manage the bandwidth I'm afraid. Extensions to TOSLINK are available to increase bandwidth, but they aren't widely supported, especially on audio equipment.
It can manage 5.1 with DTS/DD because of the compressed bitrate, but clearly this isn't an option when latency and quality are important.
Further more, TOSLINK has known clock recovery issues that give rise to jitter. Fairly acceptable for 1983, but 30 years later we have better options.
The MPB is perfectly capable of 5.1 uncompressed using the HDMI interface, which has superior bandwidth, FEC, and clock recovery when compared to TOSLINK.
In addition, there are plenty of external professional options available for this problem (pro tools, etc)
Bottom Line: You won't see AV professionals running a TOSLINK out of their MPB for any serious project.
Solution 2:
Alas, after much research, I've learnt that it just isn't possible for the current generation of MacBook Pro laptops (and most of the iMacs!) to output uncompressed 5.1 streams—even when using software capable of outputting them—without buying extra hardware, as the built-in DSP is only capable of stereo output and passing through pre-encoded DTS or Dolby Digital data.
My solution was to buy a Sound Blaster X-Fi Surround 5.1 Pro which I've connected up to my z906s using the 6-channel direct input. OS X picks this up instantly without requiring any extra software to be installed, and it was simply a case of configuring the various panels as follows:
Sound Preferences
Select the External Digital Audio Interface in the Sound Preferences' Output panel.
Audio MIDI Setup
Select the SB X-FI Surround 5.1 Pro 6-channel output in Audio MIDI Setup, and make sure that the Source is set to External Digital Audio Interface, Format is 48kHz 6-channel 24-bit integer.
Then, click Configure Speakers and select Multichannel, and 5.1 Surround in the drop-down. The defaults are the ITU channel settings, so front left, centre, and right are channels 1-3, channel 4 is the woofer, and 5 and 6 are the rear left and right respectively.
Logic Pro X
Finally, pop up the Audio preferences panel in Logic and select 5.1 Surround in the "Show as" drop-down and click the ITU button, which will set the channel assignments to the same ones as the defaults in the Audio MIDI Setup above.
Voila. Works perfectly. :)
It's a shame that native 5.1 functionality is missing from the MacBook Pro Retina range. :/ I mean, these are top-end laptops which are aimed at A/V professionals and graphical designers. A lot of people use MacBook as the core of their studios because you can take it places with you (eg, clients' offices, recording studios, etc), and it would've been nice to be able to plug the laptop straight into the optical in on the sound systems there and have instant 5.1 audio. Fortunately, the X-Fi interface is only slightly bigger than a phone, so you can chuck it in the bag with the laptop and just wire it in that way.
I hope this is of some help to other people who are in the same boat as me. :)