How can I use Audio Midi Setup utility to create a multi-output device with multiple AirPlay speakers?

Reposted from What is the difference between an "aggregate" and a "multi-output" device?

I am setting up a multi-output device using Audio Midi Setup utility, but I have an Apple TV and three different sets of speakers connected to different AirPort Express stations in different rooms of my house, instead of just the Mac and the Apple TV. How can I make all four (ATV + AEX + AEX + AEX) be part of a single device to receive audio together in sync?

The only way to do this without a third party app is directly from iTunes. In the top area of the iTunes windows you'll see the AirPlay icon. Click on that and then select Multiple to select multiple AirPlay targets to stream the audio (and/or video) to at the same time:

To stream to multiple AirPlay devices from outside iTunes you'll need to invest in some third party applications. The best approach I know of uses Airfoil for Mac from Rogue Amoeba. Airfoil allows you to send audio to many destinations from your Mac. From their page:

Multiple Outputs At Once You can send audio to multiple outputs around your house, all in sync.

It's an excellent app from an excellent Mac-centric development company and for $25 you get an excellent solution to your problem.