Is Set "prime" with respect to the cartesian product?

Suppose you have an equivalence $Set\cong C\times D$ where $C$ and $D$ are both nontrivial. Let $(0_C,0_D)$ be a pair of objects corresponding to the empty set. Then $0_C$ and $0_D$ must be initial and we must have that if there exist maps $c\to 0_C$ and $d\to 0_D$, then $c\cong 0_C$ and $d\cong 0_D$ (since no nonempty set can map to the empty set). In particular, taking $d=0_D$, we find that every object of $C$ that maps to $0_C$ is initial, and similarly for $0_D$. Since $C$ and $D$ are both nontrivial, we can choose objects $c\in C$ and $d\in D$ which are not initial. Note that there are then no maps $(c,0_D)\to(0_C,d)$, since there are no maps $c\to 0_C$. But this is impossible, because $(0_C,d)\not\cong (0_C,0_D)$ so $(0_C,d)$ must correspond to a nonempty set, and every set can map to every nonempty set.

Here's another argument, which is more similar in spirit to the arguments for other categories I sketched in comments below. Let $(0_C,0_D)$ be as above and let $(1_C,1_D)$ be the terminal object. Note that $(1_C,1_D)$ is the coproduct of $(1_C,0_D)$ and $(0_C,1_D)$. Since the terminal object of $Set$ cannot be written as a coproduct unless one of the summands is initial, this implies either $1_C\cong 0_C$ or $1_D\cong 0_D$; suppose $1_C\cong 0_C$. Then for any $c\in C$, note that there is exactly one map from $(0_C,1_D)\cong (1_C,1_D)$ to $(c,1_D)$. But this means that as a set, $(c,1_D)$ has one point, so it is terminal, so $c\cong 1_C$. Thus every object in $C$ is terminal and $C$ is trivial.