How does damage calculation in Final Fantasy 12 work?

Ignoring scalar effects like Bravery/Faith/Protect/Last Stand/etc., every weapon attack is a product of the weapon's damage roll and its statistical bonus.

Damage rolls are generally calculated by multiplying the attacker's Attack score with a random number between 1 and 1.125 before subtracting the target's Defense. There are several exceptions:

• Axes, Hammers, and Hand-bombs don't use the normal 1-1.125 random value range, and instead use a range of 0-1.111 (which would seem to indicate that they're far worse than normal weapons, but they can make up for this in the statistical bonus below).
• Guns, Measures, and the Healing Rod and Rod of Faith ignore defense altogether: they take the attack value times the random value of 1-1.125 and square the result. However, these weapons have no statistical bonus, so they're best used by lower-level or statistically inferior characters.
• Pole attacks are reduced by the target's Magic Defense rather than physical Defense.
• Unarmed attacks default to an Attack value of 11 if the attacker has not yet acquired the Brawler license. Even with Brawler, the normal Attack value is ignored in favor of the average between the character's Level and Strength.

The statistical bonus of an attack depends on the weapon equipped. Each weapon type has a primary stat and a secondary stat which contribute to the damage bonus. The way the bonus is found is usually 1 + ( ( PrimaryStat + Level ) * SecondaryStat ) / 256. Because the statistical bonus multiplies the damage roll, both stats are important, but as your level increases, primary stats contribute more and more to the damage you do with that type of weapon. Stat breakdowns (and formulary exceptions) are as follows:

• The primary and secondary stat for any Sword, Greatsword, Spear, Crossbow, Rod, or Pole is Strength.
• The primary and secondary stat for an unarmed attack is also Strength, but the stats are not added to 1 after being divided by 256; unarmed attacks can only ever be more damaging than armed attacks if the attacker's combined strength and level are significantly higher than his Attack score with any weapon.
• The primary and secondary stat for a Mace is Magic.
• The primary stat for a Katana or Staff is Magic, but the secondary stat is Strength.
• The primary stat for a Dagger, Ninja Sword, or Bow is Speed, but the secondary stat is Strength. Additionally, the stats are divided by 218 after multiplication, not 256.
• The primary stat for an Axe, Hammer, or Hand-bomb is Vitality, but the secondary stat is Strength. Additionally, the stats are divided by 128 after multiplication, not 256.
• Guns, Measures, the Healing Rod, and the Rod of Faith do not calculate a statistical bonus. They are treated as having a statistical bonus of 1.

To apply this to your situation, it makes most sense to equip Balthier with a Rifle early in the game, unless his Strength is absurdly high, in which case you might be better off leaving him unarmed and giving the rifle to someone else. As his stats develop, you can use the info here to determine which weapon types you should pursue.

Credit where it's due: I arrived at this answer by reading Arthellinus's "Weapons Analysis" FAQ at GameFAQs and summarizing his somewhat lengthier and mathier analysis.

You should ignore the attack power when comparing guns to other weapons.

Guns bypass a monster's defense. So you'll notice that at your party's current level, Balthier (or whomever) equipped with a gun is no slouch as an offensive weapon. This becomes more and more significant as the monster's go up in defense power (and the guns available also increase in attack power). It's also possible to luck into some very powerful guns about midway through the story, making their advantage even more pronounced.

In fact, there are ways to power level by using guns (since you can damage enemies that a +30 attack sword wouldn't dent).

Guns cannot be parried or blocked. Guns add to your character's evasion as well. Additionally you can stack the status effects of various kinds of bullets and also do knockback attacks - adding a few more advantages to guns.

The only huge drawback is how long they take to load.