What is the Math Formula on Fortify Restoration Potion Improves Fortify Other Skills

The math is relatively unimportant here -- it's straight math that is dependent on your input -- so a better question to answer would be, "what is going on, and why does this work?"

The core loop of this bug (and it is considered as such and fixed by the Unofficial Skyrim Patch!) involves the following 6 steps:

1. Equip as many "Fortify Alchemy" equipment as you can.
2. Make a potion of "Fortify Restoration".
3. Unequip your "Fortify Alchemy" equipment.
4. Drink the potion created in step 2.
5. Goto 1, or if you're satisfied with the numbers, Goto 6.
6. Using your exponentially-enhanced Alchemy skill, create a potion of "Fortify Enchantment", then drink the potion and enchant as many replacement pieces of equipment with "Fortify Alchemy" as you can.

The base skyrim game has a unique quirk with the "Fortify X" effects. "Fortify X" is considered a Restoration effect, and the alchemical version of "Fortify Restoration" increases the magnitude of all Restoration effects. (The enchanting version merely reduces cost) Equipping something enchanted with "Fortify X" is treated as casting a permanent "Fortify X" to your character, which means it benefits from the alchemical effect!

Unequipping and re-equipping an item will re-apply the "Fortify X" effects, which is why you will lose the bonus if you re-equip without the buff to "Fortify Restoration".

So to answer your original question, if you have a +30 "Fortify Enchanting" helm, quaff a potion of "Fortify Restoration -- Restoration spells are 20% stronger for 60 seconds.", your character should actually benefit from +36 (30 * 1.2) Fortify Enchantment , which you should be able to verify by looking at the "Active Effects" subsection of your magic menu. (Since the UI does not show fractional values by default, it may be hard to notice at low values).

"Fortify Enchanting" increases your raw Enchanting skill by the magnitude of the effect -- +36 Fortify Enchanting is the same as having an Enchanting skill 36 levels higher.