faithfully flat ring extensions where primes extend to primes

This question might be easier to consider geometrically.

You want the morphism $\operatorname{Spec}S \to \operatorname{Spec}R$ to be faithfully flat, and the fibre over each point $\mathfrak p \in \operatorname{Spec}R$ (which equals $\operatorname{Spec}S\otimes_R \kappa(\mathfrak p)$, where $\kappa(\mathfrak p)$ is the residue field of $R/\mathfrak p$) should be integral. (Actually, the question asks that $S/\mathfrak p S$ be integral, but since $S$ is flat over $R$ by assumption, the embedding $R/\mathfrak p \hookrightarrow \kappa(\mathfrak p)$ induces an embedding $S/\mathfrak p S \hookrightarrow S\otimes_R \kappa(\mathfrak p).$ Thus if the target is a domain, so is the source, and conversely, since the target is a localization of the source.)

So for example, any smooth morphism of affine varieties all of whose fibres are connected will give an example. (Since a smooth connected variety is irreducible.)

This will give an enormous class of examples.

E.g. as a slight variation on the preceding remark, let $k$ be an algebraically closed field, and let $A \subset B$ an inclusion of f.g. $k$-algebras that are domains, such that, if $K$ denotes the algebraic closure of $\operatorname{Frac}(A)$, then $K\otimes_A B$ is a domain. (In other words, the generic fibre of the map $\operatorname{Spec}B \to \operatorname{Spec}A$ is geometrically irreducible.) Then over some non-empty open subset of $\operatorname{Spec}A$ the restriction of $\operatorname{Spec}A \to\operatorname{Spec}B$ will be faithfully flat and will have geometrically irreducible fibres, and hence we can find $a \neq 0$ such that $A[1/a] \to B[1/a]$ will have the desired property.

A concrete example is given by the inclusion $\mathbb C[t] \subset \mathbb C[x,y,t]/(y^2 - x(x-1)(x-t)),$ but there is nothing particularly special about this example; it is just my favorite example of a flat family of irreducible curves.

Added: This is an example taken from the discussion in comments below. It is intended to illustrate why the above example is not particularly special or limited in scope.

Let $A$ be any f.g. $k$-algebra, and let $f(x_1,\ldots,x_n)$ be a polynomial whose coefficients are elements of $A$, i.e. $f \in A[x_1,\ldots,x_n]$. Now consider the ring $B = A[x_1,\ldots,x_n]/(f)$.

Geometrically, $f$ is the equation for a family of hypersurfaces, parameterized by $\operatorname{Spec}A$.

Again, suppose that $A$ is a domain (so $\operatorname{Spec}A$ is a variety), and let $a \in A$ be the discriminant of $f$. Unless you are very unlucky in your choice, $a$ will be non-zero. (Geometrically, this is saying that the family of hypersurfaces $f=0$ doesn't have every member being singular. Of course it could, if you chose it that way, but if you choose it generically, then it won't.) So if we pass to $A[1/a]$ and $B[1/a]$, we have a family of smooth hypersurfaces. Provided that $n \geq 2$, a smooth hypersurface is irreducible, so $A[1/a] \subset B[1/a]$ will satisfy the conditions of the problem.

[In the example with cubic curves, I didn't invert the discriminant, because I happened to choose a flat family of curves which remain irreducible even when they become singular.]