I'm 2 to 3 hours into The Witcher 3 and drowners are impossible to kill. What am I doing wrong?

You’re likely using the wrong sword. Geralt carries two swords, one is made of steel and the other is made of silver. The steel sword is for human enemies and beasts, while the silver sword is used for fighting monsters.

Your Witcher will automatically draw the correct sword for an engagement, but you can overwrite that by manually drawing your sword before moving into combat range. You may also be switching to the wrong sword because you perhaps think it’s better than the one he unsheathed.

Lastly, check both your game difficulty and the monster bestiary to see if you can make combat easier for yourself by both lowering the difficulty and by using oils, poisons, and other consumables before the fight.

In addition to what @PausePause said, you should make sure the enemies you're fighting aren't too high-level for you.

You can easily check by looking at the color of the number to the left of the enemy's health bar. According to the game manual:

8. Enemy Health Bar: Displays the currently targeted enemy’s name, level, and remaining health. The health bar’s color indicates the targeted enemy’s type: silver indicates an enemy of magic origin, red indicates all other enemy types. The number to the left of the bar shows the enemy’s level. If you are 5 or more levels above the enemy, the number will be gray. If you are within 4 levels of the enemy, the number will be green. If the enemy is 5 or more levels above you, the number will be red. A skull near the health bar indicates the enemy is a major threat.

Emphasis mine.

Here are a few examples of what this looks like:

Green

Red

Skull

Enemies far above your own level receive very little damage, unless you're striking their weakness. Drowners are supposed to be among the weakest of monsters, so you may have wandered into an area that's beyond your current level. This means that every enemy — not just Drowners — will be difficult to defeat.

You have 2 swords: the steel one and the silver one. The silver sword is for killing monsters. Geralt should pull the correct sword out automatically. Check if you manually pulled out the wrong weapon.

The start of The Witcher 3 is pretty rough. Basically you need to follow the main story line until you have gained some levels; the level requirement of the next main line quest may be too high for you. Before this point, every monster is tough for Geralt.