How to capture a city (take over another civilisation) and what is a warmonger penalty?

You can capture a city simply by melee attacking it, like you would do with a normal unit. Remember that you can only take cities with a melee attack, you cannot take cities with ranged or air units. You can take cities with naval melee units, such as the Privateer or Destroyer.

To entirely wipe out another civilization, you will have to conquer all of their cities. But in order to win a domination victory, you only have to be in control of the capital cities of all civs in the game.

About the warmonger penalty; this is diplomatic penalty you will get with other civilizations in the game. This means that they will like you less, resulting in them giving you less money for luxury and strategic resources, or if it gets worse, they will denounce you or even declare war on you. Taking the last city of an enemy will result in a much larger penalty. You will not lose any money once you capture the city, if anything, you will gain income, but only after the city gets out of resistance. You will lose happiness once you conquer an enemy city, so make sure you don't drop too much into negative happiness.

I can also refer you to the Civilization Wikia page about city combat for more information.