Did I just blow up my mainboard, or is it something else?

First of all you should check the power supply and whether it's suitable for your hardware. Most Radeon HD 5770 come with a 6-pin PCIe power plug. This plug is not optional; it really needs to be connected. Either connecting it to the PCIe plug on your power supply or using an adapter (most cards come with an included adapter). Such adapters usually are connecting the 6-Pin PCIe plug to two Molex 4-Pin connectors (the same 4-pin plug which is used for ODD and IDE HDD).
If you use the adapter then make sure to connect the adapter to two independent power cables, not connecting both 4-Pin Molex connectors to the same cable - this is due to specification on maximum current per cable.

If this does not help or you already connected everything correctly then check your power supply. The card takes around 100W peak from the 12V rails. So make sure your power supply can deliver this power. Many OEM machines are equipped with a power supply which is sized exactly for the shipped hardware without big safety margins for upgrades. As I can see your model seems to be equipped with Pentium 4 class CPUs. These CPUs are well known to take a lot of power as well (P4 was shipped with TDP up to 140W). Adding this graphics card would require at least about 250W on 12V rails (assuming ~100W CPU, Chipset, ~100W GPU, HDD, ODD). Well, with some safety margin I would say a typical high-quality 350W power supply should be sufficient. As almost everything is powered from 12V in modern machines this would require a PSU which is able to deliver around 20A on the 12V rail(s). Read the stickers on your PSU whether it states anything about maximum current. If it's not sufficient you might try replacing the PSU. But read on first, maybe it won't even work when replacing the PSU.

OK, the following is a guess; not 100% secured by facts.

I've recently tried to upgrade an older machine with an nVidia chipset (Asus M2N mainboard) with an ATi Radeon HD 5770 card.

After installing the card I've experienced similar issues. The machine did not POST at all. Fans spinning normally (PSU on, 12V OK). The PSU was sufficient to power the card of course. I think it was powered by a Silverstone ST50F 500W unit.
After googling around I found that many users had similar issues with Radeon HD 5000 series card in older machines. At the end it seems that it all tracks down to a bug/feature in the BIOS of the cards. Somehow the cards seem not to initialize properly in PCIe 1.x slots (I think this applies to your mainboard as well). The cards do only work in PCIe 2.x slots So the usual effect is that the machine won't recognize the card or not boot at all (no beep). However it's common that fans turn on as the power supply is properly powered on.

Unfortunately it seems that there is no solution for this issue. I was already preparing to flash the BIOS of the card with a fixed one. Unfortunately I was unable to find any fixed ROM image on the internet.

Finally I ended up returning the card and replacing it by an nVidia GeForce GTS250 which finally worked perfectly.

I am not sure if the Radeon HD 6000 series still faces the same issues or whether it has been fixed. Up to now I did not attempt to upgrade an old PCIe 1.x-only machine with such a card.