Is it better to upgrade Melee Attack or Flying Attack to boost BroodLord damage?

Definitely upgrade ground melee attack,

Not only does ground melee attack have better dps with a brood lord, it also complements the higher tiered zerg units as well (crack-lings and ultralisks become very common for rebuilding a late-game army after an encounter).

With that being said, if you are planning on going mass mutalisk early in the game, the air upgrades may provide a natural transition into brood lords but make sure you start researching ground attack as you transition. Prioritize ground melee, but don't forget that flyer attack helps as well. Zerg late game should provide you with plenty of resources to get both upgrades when your army is maxed.

Even though it is still probably better to get melee attacks for making brood lords better, what you are not taking into consideration in your math is if you are attacking armored units with brood lords.

For example, if you attack a unit with brood lord that has 2 armor, the BL attack does 20-2 = 18, which is 10% less than attacking a unit without the armor.

When a broodling attacks the same unit, it does 4-2 = 2, which is often times 50% (!) less than attacking a unit without the armor.

So against armored units the ratio of damage dealt from initial BL attack versus attack done by broodlings will be much higher than when attacking light units.

I think especially of for example attacking a couple of thors with a few brood lords. The broodling count will be so small that they all will almost die all the time, maybe hitting a thor just one time. In that case almost all of the damage to thors will be dealt from the BLs initial attack. (and in that case you should've probably gone for flyer attacks)

Just some things to think about. Still, without doing any exact math, going for melee attacks SEEMS better against most things.