Is this quotient ring finite?

$r^{10}=r^2$ for all $r \in R$ so the same is true in the quotient $R/P$. Since $P$ is prime the quotient $R/P$ is an integral domain and thus has a field of fractions $K$ which inherits the property $x^{10} = x^2$ for all $x \in K$ which implies $x^8=1$ for all $0 \neq x \in K$ (there are no zero divisors).

Since a polynomial equation over a field has at most as many zeros as the degree of the polynomial the field $K$ consequently has at most $9$ elements and $|K^\times| = |K|-1$ divides $8$ since $K^\times $ is a cyclic group.

By simply checking we end up with the list $|K| \in \{2,3,5,9\}$. Finally, $R/P$ is a subring of $K$ whose field of fractions is $K$ so $R/P$ already coincides with $K$.

Let $D=R/P$. Then $r^{10}=r^2$ for all $r \in R$ implies $z^{10}=z^2$ for all $z \in D$ because of the canonical projection $R \to D$.

In a field, a polynomial equation of degree $n$ has at most $n$ roots. Thus, there are at most $10$ solutions for $z^{10}=z^2$ in the quotient field of $D$. Therefore, $D$ has at most $10$ elements.