How to subtract durability from an item the player is holding in their hand?

You should use execute store result entity @s SelectedItem.tag.Damage int 1 run <some command that returns a value>. This will set the Damage of the tool (how much it has been used) to the value of whatever command you put after the run. The 1 after the int is a scale factor of 1.

You can make a system where after the item is used, you store the durability somewhere with execute store result score <entity> <objective> run data get entity @s SelectedItem.tag.Damage 1,then use scoreboard players add <entity> <objective> 1 to add one to the value (because you are adding damage, effectively removing durability), and then use the above execute store result entity @s SelectedItem.tag.Damage int 1 run scoreboard players get <entity> <objective> to store that modified value back into the Damage tag.