Underlying set of the scheme theoretic fiber

Instead of giving a vague a priori reason or the usual proof, let me give a quite unknown direct proof, which even works in the larger category of locally ringed spaces:

If $X \to S$ and $Y \to S$ are morphisms of locally ringed spaces, then you can construct their fiber product $X \times_S Y$ explicitly, see here for a summary. The set consists of all $(x,y,s,\mathfrak{p})$, where $x \in X, y \in Y$ lie over $s \in S$ and $\mathfrak{p}$ is a prime ideal of $\kappa(x) \otimes_{\kappa(s)} \kappa(y)$. The topology is generated by sets of the form $\{(x,y,s,\mathfrak{p}) : x \in U, y \in V, s \in W, \alpha(x,y) \notin \mathfrak{p}\}$ for $U \subseteq X, V \subseteq Y$ open which map into $W \subseteq S$ open, and $\alpha \in \mathcal{O}_X(U) \otimes_{\mathcal{O}_S(W)} \mathcal{O}_Y(V)$. The structure sheaf is defined in such a way that the stalk at $(x,y,s,\mathfrak{p})$ is the localization of $\mathcal{O}_{X,x} \otimes_{\mathcal{O}_{S,s}} \mathcal{O}_{Y,y}$ at the preimage of $\mathfrak{p}$.

Now let $s \in S$ and consider $Y=\mathrm{Spec}(k(s))$ with the canonical morphism $Y \to S$. Let $f : X \to S$ be arbitrary. As a set, we have $Y=\{s\}$, with structure sheaf $\mathcal{O}_{Y,s}=\kappa(s)$. Given a point $(x,y,s,\mathfrak{p})$ in $X \times_S Y$, we then have $y=s$ and $\mathfrak{p}$ is a prime ideal in $\kappa(x) \otimes_{\kappa(s)} \kappa(s)=\kappa(x)$, i.e. $\mathfrak{p}=0$. Thus, $X \times_S Y \cong f^{-1}(s)$ as sets. The topology is generated by sets of the form $\{x \in f^{-1}(s) : x \in U, r(x,s) \neq 0\}$ for $U \subseteq X$, $s \in W \subseteq S$ open and $r \in \mathcal{O}_X(U) \otimes_{\mathcal{O}_S(W)} \kappa(s)$. But these sets are open in the usual topology of $f^{-1}(s)$ (and the converse is clear from $r=1$): If $r$ as above satisfies $r(x,s) \neq 0$ in $\kappa(x)$, then $r_x \in \mathcal{O}_{X,x} \otimes_{\mathcal{O}_{S,s}} \kappa(s)$ is invertible, but the inverse extends to an open neighborhood of $x$ and we see that there is an open subset $x \in U' \subseteq U$ such that $r(x',s) \neq 0$ for all $x' \in U'$.

Edit. The underlying set $|X|$ of a locally ringed space $X$ (or just scheme) can be recovered from its $k$-valued points where $k$ runs through all fields as follows: $$|X| \cong \mathrm{colim}_k ~X(k)$$ From this and $(X \times_S Y)(k) = X(k) \times_{S(k)} Y(k)$ (which is the universal property of the fiber product) one easily deduces the description of $|X \times_S Y|$ given above.