What does $\!\bmod(n,x^r-1)$ mean? [in AKS primality test]

Informally $\!\bmod(n,\ x^r - 1)\,$ denotes polynomial arithmetic modulo the that assumptions $\,n = 0,\ x^r -1 = 0.\,$ Using these assumptions we can reduce an integer coefficient polynomial to normal form form as follows. Reduce all coefficients $\!\bmod n,\,$ then substitute $\,x^r = 1,\, $ so $\ x^{r+1} =x,\,\ x^{r+2} = x^2,\ \ldots\,$ hence we can reduce all exponents to be $< r.\,$ Indeed, we can replace all of the exponents of $\,x\,$ by their remainders $\!\bmod r,\, $ by modular order reduction.

This yields a polynomial of degree $ < r\,$ with coefficients between $\,0\,$ and $\,n-1.\,$ Two polynomials are congruent $\!\bmod (n,\ x^r-1)\,$ iff they have the same reduced normal forms (or, equivalently, if their difference reduces to $0\,$). This reduction is the same as using the the Division Algorithm to reduce a polynomial by mapping it to its remainder after division by $\, x^r-1\,,\,$ doing coefficient arithmetic $\bmod n,\,$ i.e. by considering the polynomials as being over the coefficient ring $\,\mathbb Z/n =\, $ integers $\!\bmod n$.

Just as we use integer congruences $\!\bmod n$ to mimic the quotient ring $ \,\Bbb Z_n = \Bbb Z/n,\,$ the above is using congruences to mimic the quotient ring $\,\Bbb Z[x]/(n,x^r-1)$ where the notation $\,(n,x^r-1)$ denotes the ideal $\,n\Bbb Z[x]+(x^r-1)\Bbb Z[x]\subset \Bbb Z[x]$. Similarly, in general, a congruence $ \bmod\!\!(a,b)\,$ on $ \,R\,$ mimics the quotient ring $\,R/(a,b)$.

I will address the two question that are directly stated. If you ask more, I can change this answer.

To mod by two things, in this case a number and a polynomial, is not as bad as it might seem. Modding by the number n is just as before - treat every coefficient separately. So $16x^2 + 10x + 7 \equiv x^2 + 2 \mod 5$. The polynomial modulus is a little stranger. But in the case where it's $x^k - 1$, this just means that you reduce all exponents greater than $k$ by $k$, effectively dividing polynomials that are "too large" by a smaller polynomial.

Many more sources exist. Here is a great primer. This is a short FAQ aimed at the AKS. And here is a list of other sources or learning about the AKS algorithm.

I'm not sure it's suitable for the layman, nor indeed whether there is an account of AKS that is suitable for the layman, but the book Primality Testing in Polynomial Time by Dietzfelbinger is very nice.