What is DOCTYPE?
Basically, the DOCTYPE describes the HTML that will be used in your page.
Browsers also use the DOCTYPE to determine how to render a page. Not including a DOCTYPE or including an incorrect one can trigger quirks mode.
The kicker here is, that quirks mode in Internet Explorer is quite different from quirks mode in Firefox (and other browsers); meaning that you'll have a much harder job, trying to ensure your page renders consistently with all browsers if the quirks mode is triggered, than you will if it is rendered in standards mode.
Wikipedia has a more in-depth summary of the differences in rendering when using various DOCTYPEs. XHTML is enabled by certain DOCTYPEs, and there is quite a bit of debate about the use of XHTML which is covered well in XHTML — myths and reality.
There are subtle differences between different "standards compliant" rendering DOCTYPEs, such as the HTML5 DOCTYPE (<!DOCTYPE html>, prior to HTML5, only known as the "skinny doctype" which does not trigger standardized rendering in older browsers) and other DOCTYPEs such as this one for HTML 4.01 transitional:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
The DOCTYPE tells the consuming user agent (web browsers, web crawlers, validation tools) what type of document the file is. Using it ensures that the consumer correctly parses the HTML as you intended it.
There are several different DOCTYPES for HTML, XHTML, and Framesets and each of these has two modes Strict and Transitional. Strict says that your markup is using the defined standards exactly. See W3C DTDs page for further details.
Quirksmode is basically the layout method from the browser wars days when the standards were much less respected and defined. Generally a standards mode page, that is valid, will layout more consistently across various browsers, but may lack certain features that you require. One such features is the anchor tag's target attribute. The Quirksmode site is a great resource for these differences.
One final thought is that the new HTML5 standard proposes using a very simple DOCTYPE:
<!DOCTYPE html>
Using this DOCTYPE is a forward compatible way to specify that your pages are in standards mode, and are HTML. This is the method that Google uses, and is reasonably easy to remember. I recommend using this DOCTYPE unless you plan to use XHTML.
A doctype defines which version of HTML/XHTML your document uses. You would want to use a doctype so that when you run your code through validators, the validators know which version of HTML/XHTML to check against. This page provides a good overview:
Don't forget to add a doctype
Common doctypes you can use are listed here:
Recommended list of DTDs
Which doctype you should go with depends on the code you're using, but to get an idea, try running your code through the W3C validator and use the Document Type drop-down menu in the "More Options" menu to try different doctypes out.
W3C Markup Validation Service
In HTML (including XHTML) as used on web pages, DOCTYPE is a string that triggers one of a few browser modes (quirks mode, standards mode, almost standards mode), depending on the exact spelling of the DOCTYPE. You want to use it to select a browser mode that best suits your page.
Formally, in SGML and XML, a DOCTYPE declaration is a reference to a Document Type Definition (DTD), which specifies the formal syntax rules of the markup language. No browser has ever used DTDs for anything or even accessed them. However, they are used by SGML and XML markup validators such as the W3C Markup Validator, except in HTML5 mode. Therefore, the choice of DOCTYPE determines how a validator works if the document is submitted to it. However, the validator mode of operation can also be selected in its user interface. (SGML and XML processors may use DOCTYPEs in different other ways, too, but the question is apparently meant to be limited to the HTML context and to web browsers and closely related software.)
There is no authoritative list of DOCTYPEs. Each HTML specification or draft defines its own DOCTYPE, or DOCTYPEs. The set of DOCTYPEs recognized by browsers when selecting mode varies by browser. In practice, there is no reason to use a DOCTYPE other than <DOCTYPE html> as defined in HTML5, though HTML5 also lists a few “legacy DOCTYPEs”. You can use that DOCTYPE if you want standards mode (recommended for new pages) and use no DOCTYPE if you want quirks mode (which you may need for legacy pages).
“Standards mode” generally means the mode of operation where a browser follows HTML, CSS, DOM and other specifications the best it can. It does not usually mean full conformance. “Quirks mode” is different in different browsers, but generally it means an attempt at imitating the behavior of very old browsers like IE 5. The purpose is to keep old pages working, under the assumption that they may rely on features and bugs in the old browsers. See the description What happens in Quirks Mode? Note that there is a rather different, more limited concept of “quirks mode” in HTML5, which closely resembles the document called Quirks Mode Living Standard.
A typical issue is that element widths are calculated differently in quirks mode and in standards mode. This means that the layout of a page may be more or less changed or even totally messed up, if a page designed to work in quirks mode is viewed in standards mode (or vice versa).
So you should use <!DOCTYPE html> for new pages and keep whatever DOCTYPE (if any) you have been using for old pages.
However, quirks mode means, in some browsers, that many new features of CSS are not supported. This means that if you want to enhance an old page with some CSS3 feature, it may well be necessary to switch to a DOCTYPE that triggers standards mode. In such a case, you need to review and test the page to see whether it will run in standards mode.
A doctype is a document that describes how the contents of a xhtml-like document can look like (like a webpage). Note: this defines only the syntax of said page, the rendering of the page is NOT defined by the DTD!
For example, a doctype could define how the <table>-tag can look like - which attributes it accepts, and which values/valuetypes are accepted for each attribute. Think of it as a lexicon for your current webpage.
Wikipedia has an informative page on the various Doctypes that are in common use. Mind you - there's nothing stopping you from creating your own doctype. The chances are, however, that the browser probably doesn't know how to render your document.
Which DTD to use depends on what you are going to write. XHTML has a whole different DTD than HTML, for example.