How does the game prioritize hearts?

The normal priority excluding the blood machine is soul hearts, then white hearts, then regular hearts.

If you use the blood machine at more than half a regular heart, the priority becomes white hearts, then regular hearts, then soul hearts. At half a heart, it will consume white hearts, then soul hearts, then your last half a heart (killing you). This means if you have a white heart, it will be consumed first on a blood machine - watch out!

It depends on the source of damage.

Generally, from monster hits, spikes and others, the order is soul -> white -> normal.

The blood donation machines are an exception because they will always drain your white hearts first then your normal hearts, until you have half a hart remaining, THEN drain your soul hearts