Any drawback to keep many prisoners?

The longer I play the more prisoners are in my prison. Some I can ransom, but already more than 20 are sitting that I can't ransom...

Is there any drawback keeping them alive?

There is no limit to the number of prisoners you can keep. The chance of each individual prisoner to escape or die is not related to how many prisoners you have in total.

Whether it's better for you to keep them imprisoned, execute them or just let them go is situation-dependent.

Sometimes you might actually help the enemy by executing a prisoner. For example if that prisoner is a landed noble. A duke sitting in prison can often be worse for a realm than a duke dying and getting replaced by their heir. Or a prisoner might be a close relative of someone who might declare war on you. They can't do that as long as you have a hostage. They can (and likely will) after you killed the hostage.

On the other hand, killing a prisoner might also cause some good stuff to happen. When their heir is a child (even better a girl) you can cause a bloody succession crisis among your enemies. When it is a commander with very high martial skill, you will likely want them dead so you won't ever meet them again on the battlefield.

Then there are a few other fun things you can do with prisoners:

• Brainwashing children: When you have children in your prison cells, you can assign guardians to them. When they have a different religion, then the guardian might convert them (use guardians with "Zealous" and "Diligent" to increase the odds of this happening). When they become adults, release them. They will then return to their court where they will now hate everyone for religions reasons. That means they will agree to join most murder plots.
• Discount Ransom: They / their liege want to pay ransom, but they can't afford it? Consider giving them a gift so they can. You often still end up with a profit that way and they get the opinion bonus from the gift.
• Stealth Executions: You have one of your own vassals in your prision you want dead, but you are not allowed to execute them without getting a tyranny opinion penalty? Then plot to murder them. Murdering people in your prison is far easier than murdering free people.
• Stealing women: When you are a man of a religion which allows concubinage, you can force any adult woman in your prison to become your concubine. I haven't tried, but I think it might also work with male prisoners if you are a woman and your religion allows you to take men as consorts. Yes, forcing people into your harem is really creepy, but there is a non-sexual use for that: If you then set her aside, she won't return home but will stay in your court as a regular vassal.
• Enjoy sadism: With the "Reaper's Due" DLC, you have the options to "Humiliate", "Torture" or "Mutilate" prisoners. These actions will also free the prisoner. There is usually little practical reason to choose any of this over just executing or freeing them regularly, but it satisfies your secret sadist urges :)

Well, the person you've imprisoned will hate you. Same goes for their parents/spouse(s)/children/friends. This may lead to some issues down the line.

Furthermore, there is always the risk that they will escape, and as such not capitalizing on the money for ransoming or relations bonus for releasing them may be seen as a waste. Likewise, if you really needed them out of the picture, killing or banishing could be a viable alternative if you don't want to risk them escaping.