Besides perks, what other permanent abilities can I get?

In another question, it was mentioned there were other permanent abilities besides perks.

What are they, and where can they be found?

There are a number of different permanent abilities besides the normal perks that you can get:

Guardian stones

There are the 13 guardian stones, found at locations throughout the world:

Warrior: Combat skills increase 20% faster
Thief: Stealth skills increase 20% faster
Mage: Magic skills improve 20% faster
Lover: All skills improve 15% faster
Apprentice: 2x magicka regeneration, twice as vulnerable to magic
Atronach: +50 magicka, 50% magic absorb, -50% magicka regeneration
Lady: +25% health and stamina regeneration
Lord: +50 damage resist, 25% magic resist
Ritual: All dead things around you become undead and fight for you
Serpent: Attacks have a chance to paralyze targets for 5 seconds and 25 damage
Shadow: 60 second invisibility power
Steed: +100 weight capacity, no movement penalty from armor
Tower: You can unlock one expert-level or lower locked door/chest per day without lockpicks

These are mutually exclusive: you can only have one active at any time. If you have the Aetherial Crown, you can activate 2 Standing Stones.

Quest-related "perks"

There are a few permanent abilities you get as a result of completing various side quests:

Ancient Knowledge (+25% Armor if wearing all Dwarven armor, smithing skill increases 15% faster) received for completing Unfathomable Depths
Sailor’s Repose (10% more health from healing spells) for completing What Lies Beneath
Sinderion’s Serendipity (25% chance of creating an extra potion during alchemy) for completing A Return to Your Roots in Blackreach

Guild "perks"

These are received while completing quests for each of the "guilds" or organizations in the world:

Assassin's Aegis (+25% Armor if wearing all Dark Brotherhood Armor) for completing the Dark Brotherhood quest, Sanctuary
Dragon Infusion (25% less melee damage from Dragons) for completing the Blades quest, Dragon Research
Prowler’s Profit (chance of finding additional gems in chests) for completing the Thieves' Guild quest, No Stone Unturned
Nightingale’s Aegis (+25% armor if wearing all Nightingale set pieces) for completing the Thieves' Guild quest, Trinity Restored

Permanent blessings of the Divine

While all the Divines have temporary blessings you can get for visiting their shrines, Dibella and Mara have quests that allow you to get a permanent blessing from them:

Agent of Mara (+15% Resist Magic) received for completing The Book of Love
Agent of Dibella (+10% melee damage to the opposite sex) received for completing The Heart of Dibella

Meditations

After completing the initial Greybeards quests that are part of the main storyline, you can speak to Paarthurnax who will give you one of three meditations:

The Fire Within: the Fire Breath shout does 25% more damage
Force Without Effort: You stagger 25% less while enemies stagger 25% more
Eternal Spirit: +25% health regeneration while ethereal

You can only have one meditation active at any given time, but you can switch at any time by speaking to Paarthurnax again.

Nightingale Agent status

You can select one of three gifts after completing "Darkness Returns":

Shadowcloak of Nocturnal: For 120 seconds you automatically become invisible while sneaking.
Nightingale Subterfuge: People and creatures in the spell's area of effect will attack anyone nearby for 30 seconds.
Nightingale Strife: Instantly absorbs 100 points of Health from the target.

These are permanent, activated powers that can only be used once per day. Similarly, you can swap between the three gifts by revisiting the Twilight Sepulcher, but only once per day.

Unimplemented abilities

These abilities are mentioned in the official strategy guide, but do not appear to be implemented in the game (yet?):

Archmage's Authority (Better prices from members of the College) for completing the final College of Winterhold quest, The Eye of Magnus
Gift of the Gab (speech skill increases 15% faster) for completing the Bards College quest, Tending the Flames