Can a website know about my browsing history?

The short answer is yes, though it is not as easy as you might think.

The browser stores cookies independently for each domain. That means that www.foo.com cannot access the cookies made by www.bar.com and vice versa.

The vulnerability (or loophole) is in included pages. Most advertisements come from a different domain than the page itself, so they create their own sets of cookies. When another site includes an ad from the same ad provider, they can read their own cookies created earlier and they know you visited that page earlier. This way they can only track you on sites that host their ads. This is the strategy Google uses to serve relevant ads.

Also Facebook and other social networks can do this because of their ubiquitous like, tweeet, pin etc. buttons, which are also included content. This is not avoidable without disabling cookies altogether or using private browsing, but that could be a major burden (Cookies do make the Internet convenient). I personally choose not to be paranoid of these things.

A third-party cookie placed on your machine at one site can be detected at a later time by the third party, so your browsing history is to some extent public. There are both HTML and Flash cookies, so if you wish to remove them delete both types.

Browsers such as Firefox can be set to a "private" mode in which cookies are deleted at shutdown. Cookies can also be deleted by the browser or by free tool such as CCleaner. Deleting cookies may require you to log into sites which had opened automatically before.