How to connect dualshock3 (PS3) controller when it cannot be found
According to the steps listed here, you have to connect it with a USB cable first, press the PS button until the blinking lights speed up, then disconnect the cable. The faster lights indicate it is trying to pair. Once the cable is disconnected, the Mac should request the pairing passcode which is 0000.
This pairing process should only be a one-time setup, and going forward you should be able to connect simply by turning the controller on.
Note that you may need to reset the DS3 before attempting to connect it to your Mac - to do so, use a paperclip to click the reset button, which is left of the L2 button.
(Image from Sony)
Apparently, Yosemite breaks the support for the PS3 controller. This can be easily seen when looking into /var/log/system.log. When connecting the PS3 controller via USB, you'll see something like:
Aug 8 09:08:19 somebody-macbookpro kernel[0]: [IOBluetoothHIDDriver][probe] Couldn't retrieve device properties
Aug 8 09:08:19 somebody-macbookpro.local blued[84]: addDeviceToHIDEmulationMode - unrecognized HID device; NOT storing the link keys to the module.