Yosemite randomly switches audio output to displayport

Ever since upgrading from Mavericks to Yosemite I'm experiencing this annoying bug, where OSX switched the default audio output to external display connected via DisplayPort. This display has no speakers nor audio port of any kind.

Is there a permanent work around for this? Can I somehow blacklist a device as audio output?

System info:

System Preferences builds a preferred list for your audio outputs and inputs based on which ones you pick over others.

To work around this you will need to manually select the audio output you would rather use once when the undesired one is available and again when it isn't. This method will work in most (but not all) cases.