Prove that $I \subseteq R$ is prime if and only if $R/I$ is an integral domain.

We say that an ideal $I \subseteq R$ is prime if for all $a, b \in R$, $ab \in I$ implies that $a \in I$ or $b \in I$.

(a) Prove that $I \subseteq R$ is prime if and only if $R/I$ is an integral domain.

Proof. First note that the zero element of the ring $R/I$ is $0 + I = I$ and that $a + I = I$ if and only if $a \in I$. Now suppose that $I \subseteq R$ is a prime ideal and consider nonzero cosets $a + I$ and $b + I$ in $R/I$ (i.e. consider $a \not \in I$ and $b \not \in I$). Since $I$ is prime this implies that $ab \not \in I$, hence $ab + I \neq I$ and we conclude that $R/I$ is an integral domain.

Conversely, let $R/I$ be an integral domain and consider $a, b \in R$ with $ab \in I$ (i.e. consider $ab + I = I$). Since $(a + I)(b + I) = ab + I = I$ and $R/I$ is an integral domain we conclude that either $a + I = I$ (i.e., $a \in I$) or $b + I = I$ (i.e., $b \in I$). Hence $I \subseteq R$ is a prime ideal.

My question is: how can they conclude that since $a,b$ don't belong to $I$ then it is an integral domain then prove the other way by stating that since it is an integral domain then $ab$ belong to $I$?

So you need to prove that $I \subseteq R$ is prime iff $R/I$ is an integral domain.

To prove this, you need to prove the $\implies$ direction and the $\impliedby$ direction.

For the $\implies$ direction, you suppose $I \subseteq R$ is a prime ideal, and prove that $R/I$ is an integral domain. What does it mean to be an integral domain? It means you are a commutative ring with multiplicative identity, and no zero divisors. But $R/I$ is already a commutative ring with identity if $R$ is. So we only need to show that it has no zero divisors, i.e., that if you have two nonzero elements, then their product will be nonzero. So let $a + I$ and $b + I$ be nonzero elements in $I$ (i.e., $a, b \not \in I$, as you stated). Then since $I$ is prime, $ab \not \in I$ (since if $ab \in I$, we must have $a \in I$ or $b \in I$ because $I$ is prime). But $ab \not \in I$ precisely means $ab + I$ is nonzero, as you stated. So any two nonzero elements of $R/I$ have nonzero product, which means there are no zero divisors in $R/I$, and so $R/I$ is an integral domain.

Now for the $\impliedby$ direction, we assume $I \subseteq R$ such that $R/I$ is an integral domain. We need to prove $I$ is a prime ideal. So we need to show if $ab \in I$, then either $a \in I$ or $b \in I$. But if $ab \in I$, then $ab + I = I$. But we know $ab + I = (a + I)(b + I)$. But $ab + I$ is zero in $R/I$, so that means the product $(a + I)(b + I)$ is zero in $R/I$. But $R/I$ is an integral domain by assumption, so it cannot have zero divisors. That means either $a + I$ or $b + I$ is zero. But then that means either $a \in I$ or $b \in I$ since if $a + I$ is zero, $a \in I$ (similarly for $b + I$).

Does this clear things up?