Showing $F[x,y]/(ax^2+by^2-1)$ is a Dedekind domain
Solution 1:
This ring is clearly noetherian (as a quotient of a polynomial ring) and a $1$-dimensional domain, since we divide a principal prime ideal out of a two-dimensional domain.
For the last step, we can pass over to geometry:
For a one-dimensional noetherian domain, integrally closed is the same as regular, and we can test this with partial derivatives. Let $f=ax^2+by^2-1$, then $\partial_x f = 2ax, \partial_y f=2by$ and both vanish iff $x=y=0$ but the point $(0,0)$ is not contained on this ellipsoid, hence our ring is regular and thus integrally closed.
Solution 2:
To show that $R$ is a UFD is not a good idea. For instance, if $F=\mathbb R$, and $a=b=1$ this is not true; see here.
Instead, you can prove that $R$ is integrally closed. This follows from a general result which I proved here. See also the (solved) exercise 4.H from these notes.
For $\dim R=1$ you have to recall that $\dim F[X,Y]=2$.