What do all of the Melee/Unarmed special attacks in V.A.T.S. do?
Once you have sufficient Melee/Unarmed skill, you unlock additional, special attacks with those combat skills, both in and out of VATS (to use them out of VATS, you need to do a power attack - hold the attack button down, in combination with moving in a direction for varied results).
Thing is, the actual effect of these special attacks is... horribly undocumented, and the Wiki isn't quite up to speed on them either.
I've figured out that the 'Back Slash' with sword/knife weapons just costs less AP, but are there any special effects? Is there any good reason not to just spam them in VATS? (Out of VATS, they tend to be substantially slower than normal attacks, for better or worse I've found).
I've looked into the GECK to find some information about the special attacks. They are a property of the weapon, so there is no nice overview. The AP costs vary per weapon, the other effects not, as far as I can see.
The Back Slash on a knife has an AP cost of 16 (with a knife) and deals 0.7x damage.
The Grand Slam on the baseball bat uses 35 AP, compared to 25 standard, and deals 2x damage.
The Long Cut for the Ripper has an AP cost of 70, compared to 65 standard, and deals 1.2x damage.
The Scrap Heap for the Thermic Lance has an AP cost of 70, compared to 65 standard, and deals 1.25x damage.
The Mauler for Super Sledge has an AP cost of 48, compared to 38 standard, and deals 50% damage. This attack knocks down the enemy (if Stonewall perk is not present).
The Lights Out for Police baton has an AP cost of 29, compared to 23 normal, and deals 1.25x damage.
The Wiki has been updated to include this info since this question was asked.
The vast majority offer simply adjustments to the damage and AP cost (as well as modified animations that may affect attack speed). Notable exceptions are Fore! and Grand Slam, which offer a guaranteed knockdown, and Mauler, which offers a chance of a knockdown. Additionally, Flambe Cut offers a fire damage DoT effect. Consult the full table, or the individual weapon pages on the Wiki for exact damage modifier and Damage/AP numbers for various weapons. Many of the special attacks do less damage than a standard attack; in general, these are not worth using unless the reduced cost allows you to use an additional attack per VATS round.
Outside of VATS, the various special attacks are just fancy animations for a standard power attack, for better or worse.
For Unarmed attacks, Stomp deals double damage to knocked down opponents, and Upper Cut and Cross offer small damage bonuses, which give them substantially better AP/Damage ratios than a standard attack. Additionally, Cross yields a bonus to limb damage.