How does spell cooldown system works?

Solution 1:

In D&D systems spells are uses / day. They return when you rest only. You can change the spells you are going to use for the next day in your spellbook, but you will need to rest to be able to use them.

Solution 2:

More specifically, Baldur's Gate (including 2 and add-ons) uses the rules of 2nd edition AD&D, where spells work more or less identically for each spell-casting class.

Depending on your class level and primary attribute (intelligence for wizards and bards, wisdom for most others -- see descriptions), you get a number of spell slots for each spell level. These slots can be filled by spells from your spellbook or scroll when you rest -- this is called memorising spells. Once a memorised spell is cast, it's gone from the caster's memory (but not his spellbook/scroll, meaning it can be memorised again when you rest). The same spell can be memorised more than once, taking one level-appropriate slot each time, allowing you to cast it that many times before resting. If you're curious, this ruleset is called Vancian Magic, after Jack Vance's Dying Earth series of books.

As far as I know, there are no penalties for resting often, but resting may cause some encounters, but these are only affected by where the party is and what you have done before.

As a final tip, be sure to check your spellbook after levelling up, increasing your primary attribute, or otherwise getting more spell slots (are there items that grant them in BG? I don't remember)! New slots start out empty, so if you forget to check your spells and rest, you'll miss out!