How does the effect "Counter target spell" work?
There is an instant card called "Cancel" with this effect and also a creature whose name I forgot. I don't really understand the effect of "counter target spell" and how (and when) to use this card. I tried to use it against an instant card from my opponent, but apparently "counter target spell" doesn't destroy the enemy instant which I thought would happen...
Could someone please explain how this card is used?
When you "Counter" a spell it essentially never happens. This can only be done during the time the spell is being cast.
A spell that is countered is put into the graveyard instead of doing its effect.
In Magic the Gathering, everything but land is a spell. So whether your opponent is playing a creature, instant, sorcery, enchantment, etc. it can be countered.
As far as why your specific counter did not work, perhaps you targeted the wrong card? Or maybe your opponent used a counter on your Cancel (countering your counter). Another reason could be that the spell specifically cannot be countered. It is also possible that you attempted to counter first and the instant was play on the stop of the stack, meaning it resolves first. In order to know for sure, we would need to know what spell the opponent was playing as well as anything on the field.