Why is quadratic integer ring defined in that way?

Quadratic integer ring $\mathcal{O}$ is defined by \begin{equation} \mathcal{O}=\begin{cases} \mathbb{Z}[\sqrt{D}] & \text{if}\ D\equiv2,3\ \pmod 4\\ \mathbb{Z}\left[\frac{1+\sqrt{D}}{2}\right]\ & \text{if}\ D\equiv1\pmod 4 \end{cases} \end{equation} where $D$ is square-free.

I understand $\mathbb{Z}\left[\frac{1+\sqrt{D}}{2}\right]$ is not closed under multiplication if $D\equiv2,3\pmod 4$. But still, isn't it more natural to define $\mathcal{O}=\mathbb{Z}[\sqrt{D}]$ for all square-free $D$? (In that case, it really seems like 'quadratic integer'.) I wonder what is the motivation of this definition.

Note: this answer has been edited to put the punchline first so that the rest is more motivated)

The Punchline I think the definition they gave you was a bad one, since usually one starts the study of integer rings (quadratic and otherwise) with conditions like "integrally closed Nötherian domain in which every prime ideal is maximal," or "every ideal factors uniquely into prime ideals," or something else more conceptual or based on properties. After that you usually do some (often-times very hard) work to determine the explicit description. In your case you are just given the explicit description (for the quadratic case) right off the bat, and the lack of symmetry--as you have noted--can be quite striking, until you later realize there's a good reason for it. In your case the problem is that those sets you suggest make more sense fail to satisfy that "integrally closed" part of the definition, eg $\Bbb Z[\sqrt{5}]$ is not integrally closed. The most likely reason is that you're in a basic class where the concepts involved in integer rings have not been taught yet, but since they are such a basic case, they're giving you some practice without the motivation (a confusing, but useful practice if their goal is to prep you for eventually studying the subject more thoroughly).

Since integral closure is about things in your field satisfying monic polynomials, you do some computations to get the description you were given as a definition.

The computations behind the definition you were given

$$\mathcal{O}_K=\left\{\alpha=a+b\sqrt{D} : a,b\in\Bbb Q, \exists \text{ monic } p(x)\in\Bbb Z[x]\text{ s.t. } p(\alpha)=0\right\}.$$

We note that the minimal polynomial for $a+b\sqrt{D}$, assuming $b\ne 0$ is just

$$p_{\alpha}(x)=x^2-2ax+(a^2-Db^2)$$

because it is monic, and has $a+b\sqrt{D}$ as a root by inspection (the other root is $a-b\sqrt{D}$). In order for this to be integral, we need $2a\in\Bbb Z$ and $a^2-Db^2\in\Bbb Z$, and nothing more.

if $2a=2j+1$ is odd, then we compute

$$a^2-Db^2 = {4j^2+4j+1-4Db^2\over 4}$$

If $b\in\Bbb Z$ this is not an integer, since $4j^2+4j-4Db^2+1\equiv 1\mod 4$, so the numerator is not divisible by the denominator, so it must be that $2b=2k+1$, hence we have

$$a^2-Db^2={4(j^2+j-Dk^2-Dk)+1-D\over 4}$$

and for this to be an integer, it is necessary and sufficient that $D\equiv 1\mod 4$. We see then that $\mathcal{O}_K$ is the ring of integers exactly as in your definition, since in all other cases $a,b\in\Bbb Z$--remember, the only way $2a\in\Bbb Z$, but $a\not\in\Bbb Z$ was in the case where we concluded $2b\in\Bbb Z, b\not\in\Bbb Z$ and $D\equiv 1\mod 4$.

For the ambitious student

Why we care about integral closure

The most essential reason is because you get the (generalized) fundamental theorem of arithmetic. Without integral closure, you cannot guarantee the unique factorization because the prime ideals aren't always maximal--i.e. without integral closure your ring is not a Krull domain. We cannot take just any ring that looks natural, we have to do the one that the mathematics definitions lead us to. (Again, the general definitions, not the very specific one you got in this case.)

The ring of integers of a quadratic number field $\,\mathbf Q(\sqrt D)=\bigl\{x+y\sqrt D\mid x, y\in\mathbf Q\bigr\}\enspace $ ($D$ square-free integer) is not defined as the subset with $x, y\in\mathbf Z$, but as the subset of elements in $\mathbf Q(\sqrt D)$) which are integral over $\mathbf Z$ — the integral closure of $\mathbf Z$ in $\mathbf Q(\sqrt D)$, i.e. elements which are roots of a monic polynomial of degree $2$ with integer coefficients.

This integral closure is a free $\mathbf Z$-module. While a basis of $\mathbf Q(\sqrt D)$ over $\mathbf Q$ is $(1,\sqrt D)$, it happens that it is a basis of the integral closure only if $\,D\equiv 2,3\bmod4$, and if $D\equiv 1 \bmod 4$, a basis is $1,\dfrac{1+\sqrt D}2$.

The definition is a consequence of $\mathcal{O}_{\mathbb{Q}(\sqrt{D})}$'s contents, not the other way around. Clearly $\mathbb{Q}(\sqrt{D})$ consists of algebraic numbers, and some of those are algebraic integers. But without knowing what $D$ is, we can't be sure if there are algebraic integers in $\mathbb{Q}(\sqrt{D})$ that are not of the form $a + b \sqrt{D}$.

At this point I think it's a good idea to review the distinction between algebraic numbers and algebraic integers. An algebraic number of degree 2 has a minimal polynomial of the form $a_2 x^2 + a_1 x + a_0$, with each $a_i \in \mathbb{Z}$. An algebraic integer of degree 2 has a minimal polynomial of the form $x^2 + a_1 x + a_0$, or we could say that $a_2 = 1$.

Obviously the minimal polynomial of $\sqrt{D}$ is $x^2 - D$. But if $D \equiv 1 \pmod 4$, then $\frac{1}{2} + \frac{\sqrt{D}}{2}$ has a minimal polynomial of $x^2 - x - \frac{D - 1}{4}$. Of course $\frac{D - 1}{4}$ is not an integer if, say, $D \equiv 3 \pmod 4$, but in that case the minimal polynomial becomes $2x^2 - 2x - \frac{D - 1}{2}$, which means we still have an algebraic number, just not an algebraic integer. (You can work out what happens with $D$ singly even if you want).

This number $\frac{1}{2} + \frac{\sqrt{D}}{2}$ is important and is sometimes designated $\omega$, or, worse, $w$. But I suggest only using $\omega$ for the case $D = -3$. Still, it can come in handy to assign this number to a single symbol, but there are plenty of letters in either alphabet to accomplish this. Let's say you pick $\theta$.

So then, instead of representing the algebraic integers of $\mathbb{Q}(\sqrt{D})$ with $D \equiv 1 \pmod 4$ as $\frac{a}{2} + \frac{b \sqrt{D}}{2}$ and having to worry about $a$ and $b$ having the same parity, you can rewrite the numbers as $c + d \theta$. A downside of this, in my opinion, is that in the case of $D$ negative it's a little bit harder to figure out $\Re(c + d \theta)$ and $\Im(c + d \theta)$, whereas $\Re\left(\frac{a}{2} + \frac{b \sqrt{D}}{2}\right) = \frac{a}{2}$ and $\Im\left(\frac{a}{2} + \frac{b \sqrt{D}}{2}\right) = \frac{b \sqrt{D}}{2}$.

If you go on to study algebraic domains of higher degree, you need to be ready for algebraic integers that don't necessarily look like $a + b \theta$. For example, $$\frac{1}{3} + \frac{\root 3 \of{19}}{3} + \frac{(\root 3 \of{19})^2}{3}$$ is an algebraic integer in $\mathbb{Q}(\root 3 \of{19})$, as it has $x^3 - x^2 - 6x - 12$ for a minimal polynomial.

It turns out to be significant to identify an integer with a root of a monic polynomial, and the quadratic definition is a by-product of that. Look at the roots of $x^2-x-1$, for example - in the general theory it makes sense to call them integers.

The context for this is a broader understanding of the ideas encompassed by "Noetherian" and "finitely generated".

However in the quadratic world there remains an ambiguity in what is counted as an integral quadratic form - is it a form which always takes integer values for integer inputs, or a form with integral coefficients? Serious mathematicians and texts take different views. That is not quite your question, but closely related.