How to limit the bandwidth on a per-application (or per-process) basis in OS X Yosemite?

The feature you want is confusingly called Quality of Service or "QoS" for short.

These days it is usually configured for an entire network, or subnet, using the management interface on a smart switch (which costs a little more than an Ethernet hub with the same number of ports, but not much more--here is a representative model from Cisco for under $100).

The closest equivalent that runs on OS X directly is ThrottleD from IntraArts:

Throttled Pro [gives you] weighted network queues (WF2Q+) that guarantee bandwidth for all your essential network services like web browsing, email, and online gaming. In addition, it provides ACK packet priority which speeds up downloads when you are sending out a lot of data. This combination assures that you get the most out of your internet service.

The GUI version is $20, while the CLI version is donationware.

Charles Proxy can do throttling. The app functions as a proxy server on your machine. You can redirect traffic that you want to throttle to its way. The throttled app has to come with its own proxy settings.

The best way to throttle bandwidth on macOS is using Dummynet, which is built into the macOS kernel and works with the pf packet filter. The only way to effectively tune bandwidth on a per-process basis is to identify which local ports are bound by the process and then create the corresponding pf dummynet rules. As far as I know the only macOS app that offers this option is Scudo (I'm the developer). Scudo 1.0 beta 3 is currently available for free, more info about Scudo per-process bandwidth tuning can be found here: http://www.murusfirewall.com/forum/viewtopic.php?f=2&t=1919&p=3281#p3281