What are trade offs for "busy wait" vs "sleep"?
Going to sleep until the scheduler wakes you is the normal/prefered thing to do.
Spinning (the alternative way to wait, without sleeping) is less usual and has the following effects:
Keeps the CPU busy, and prevents other threads from using the CPU (until/unless the spinning thread finishes its timeslice and is prempted)
Can stop spinning the very moment the thing which you're waiting for happens (because you're continuously checking for that event, and you don't need to take the time it takes to be woken up, because you're already awake)
Doesn't invoke the CPU istructions required to go to sleep and to wake up again
Spinning can be more efficient (less total CPU) than going to sleep, if the length of the delay is very short (e.g. if the delay is for only as long as it takes to execute 100 CPU instructions).
A Spin Lock burns CPU and the polled resource path for a continued waste of resources while the desired event does not occur.
A Blocking operation most importantly differs by leaving the CPU and associated resource path out and, installing a wait of some form on the resource from which the desired event is expected.
In a multitasking or multithreaded/processor environment (the usual case for a long time now), where there are other operations possible while the desired event has not arrived, burning CPU and resource access paths leads to an awful waste of processing power and time.
When we have a hyperthreading system (like I think you are referring to in your question), It is important to note that the granularity at which CPU threads are sliced is very high. I would stick my neck out to also observe that all events -- on which you would tend to block -- would take sufficient time to arise, compensating the small time-slice for which they had to wait extra before unblocking.
I think J-16s point is towards the condition where a sleeping (blocked) thread is leaving its code and data space unused while in blocked state. This could make a system relinquish resources (like data/code caches), which would then need to be refilled when the block is released. Therefore, subject to conditions, a block may effect in more resource wastage.
This is also a valid note and should be checked in design and implementation.
But, blocking is usually better than spin-locks in most conditions.
If in the use case of your application, context switching would be more expensive than eating a few CPU cycles because your condition would be guaranteed to get satisfied within a short time, then busy waiting may be good for you.
Otherwise, you can forcefully relinquish the CPU by sleeping or cond_wait()ing.
Another scenario that I can think of for forceful context switching out is as follows:
while(condition)
sleep(0);