Prove that the language $\{bin(p) \mid p\ \text{is prime}\}$ is not regular (prime numbers)

The answer is negative: the set of primes is not $2$-recognizable. A proof is given in [1], example 4 and in [2], Corollary 1.43. The proof relies on a property of $k$-recognizable languages due to Cobham (1972).

Let $(x_n)_{n \geqslant 0}$ be an increasing sequence of natural numbers. If the set $\{x_n \mid n \geqslant 0\}$ is $k$-recognizable, then either $$ \limsup_{n \to \infty}\ (x_{n+1} − x_n) < \infty \quad \text{or}\quad \limsup_{n \to \infty}\ \frac{x_{n + 1}}{x_n} > 1. $$ Let $p_n$ be the $n$-th prime number. Since $n! + 2, n!+3, \dotsm, n! +n$ are consecutive composite numbers, one has $\limsup_{n \to \infty}\ (p_{n+1} − p_n) = + \infty$. Moreover, it is a consequence of the prime number theorem that $\limsup_{n \to \infty}\ \frac{p_{n + 1}}{p_n} = 1$ (see Prime gap on Wikipedia).

[1] N. Rampersad, Abstract Numeration Systems in Language and Automata Theory and Applications, LNCS 6638, 65-79 (2011)

[2] M. Rigo, Formal Languages, Automata and Numeration Systems, Volume 2, John Wiley & Sons, 2014.

Conclusion. This proof answers the EDIT part of the question, but not the first part. Finding a proof relying on the pumping lemma is still an interesting challenge.