Does 'overcharging' a sniper increase its damage?

No.

Once the meter is full, the sniper rifle will deal maximum damage (normally 150 for bodyshots, 450 for headshots). Charging longer than this won't increase the damage any further.

You probably dealt 500 damage because of the Machina's +15% damage on fully charged shots. This would give you 518 damage (though 450 also kills a fully overhealed heavy).

On this page, you can read about the sniper Machina, which is a sniper that deals 15% extra damage when fully charged.

"A fully charged shot fired by the Machina delivers an extra 15% damage, can pierce through multiple enemies, and has a distinct firing sound."

This is the only sniper that gets bonus damage for being fully charged.