LG Ultrafine 5K volume too loud

This is fixed in the latest firmware from LG. Unfortunately you have to install it via their Screen Manager software, available here, but it seems to resolve the issue for the most part.

You can make fine adjustments to your volume level by holding OptionShift while you press F11 F12 (Volume Up/Down). this way, it won't make those big jumps.

As for the cause of the volume issue - it appears you aren't alone as evidenced by this Apple discussion thread and this thread. Everything that I have read says they are waiting on a software update

Desr Sir, I recently purchased the newest model of the Apple MacBook Air and the LG Ultrafine 4K. I had the same issue that the speakers went into overdrive as soon as I turned up the volume to more that 20%. I installed the latest version of the LG screen manager. After a few restarts it resolved this issue.