How can I level up in magic when I can't cast any spells?

I tried all 9 single-rune spells, and they all "fizzle" or it says I don't have enough skill to cast them. I am assuming that the single-rune spells are the simplest, low-level spells.

So my question is, how can I improve my magic skill when I can't even cast spells?

In order to cast spells, you need skill points allocated to the spell trees. I believe you can allocate some basic points at character creation, and at level up. The most basic spell is in the Fire tree, and you get it after only 2 skill points in this skill.

Your magic is only going to get you so far in some of the longer fights, so being without magic is not all that unusual for a mage. If you can't cast spells and you don't want to reroll, you can always use back row attacks, like throwing rocks, using a bow, or using the reach weapon, this spear.

Runes don't stand for individual spells. Those are just buttons you need to press as a code to activate spells (with some only requiring 1 rune and the largest combos needing 4 runes)

In order to be able to USE certain combinations of runes, you need to have trained the appropriate skill high enough. For example, the Air school has one spell that unlocks when it's skill is at 4, and another at 9. (Note that there is a panel on the right side of the page that links to all the other schools)

Yep, that's pretty much it: Knowing the combination and having a skill high enough to use it. You can find the combinations on scrolls you happen to come across while exploring Grimrock.