"length in bytes" vs "length by the byte" and "paid in hours" vs "paid by the hour"

The two examples involve different roles for the unit. You could have, for example,

The variable 'len', which contains the buffer length in bytes, changes by the minute.

He is paid by the hour, in pound notes.

It's the difference between by what method? [incl. how often?], and in what units?.

The convention is different because the objects in the expressions are different.

Both "in bytes" and "in hours" mean in units of said. A buffer size is actually measured in bytes, while being paid in hours is ambiguous at best. Unless someone has created an avatar of time that is now a currency, that wouldn't even be payment, it would be barter!

The expression "to be paid by the hour" refers to how the payment is calculated. Instead of being set by number of commissions secured, their pay is set by the number of hours they work.