Why does this trick on the Arch-vile work?

I think it is because the line of sight is technically obstructed. On the Wiki page, the way the Arch-Vile's explosive attack works is it does a minimum base damage amount of 20, and then additional damage is done via blast damage.

The Arch-vile's attack inflicts a base 20 points of damage, plus up to 70 additional points of blast damage. The blast damage against immobile targets is often around 63 or 67 points, as the center of the attack is not normally placed at the center of the target.

For blast damage, one of the things that is examined is the line of sight. If the object is out of sight from the blast, it will take no damage:

The line-of-sight is checked, and if the view of the object from the explosion is obstructed, no damage is inflicted

This image from the Wiki page illustrates this:

In this case, lets pretend you are object A. The Arch-Vile is on the opposite side of that wall, which is just high enough (24 units according to that video) to obscure line of sight (for the blast damage at least). So, you take the Arch-Viles base damage of just 20 only.

It's important to note that the blast damage is not usually placed in the center of the target (the player in this case). I think that is why you cannot be on the same side of the Arch-Vile for this trick. I'm not entirely sure on this, but I'm guessing the center of the blast damage is in front of the player some when standing opposite of the Arch-Vile. This would place it inside of the wall that the player was standing against in the video, which I would think also obscures line of sight. If you are on the same side as the Arch-Vile, the blast damage may be centered behind you and you get the full brunt of the blast.