Probability of winning the game "1-2-3"

For $i,j\in\{1,2,3\}$, let $a_{i,j}$ denote the number of $i$ cards being dealt with number $j$ spoken. We have $\sum_j a_{i,j}=4$ and for a winning game $a_{i,i}=0$. The number of winning positions for a given $(a_{i,j})$ is $$\frac{18!}{a_{2,1}!a_{3,1}!(18-a_{2,1}-a_{3,1})!}\cdot\frac{17!}{a_{1,2}!a_{3,2}!(17-a_{1,2}-a_{3,2})!}\cdot\frac{17!}{a_{1,3}!a_{2,3}!(17-a_{1,3}-a_{2,3})!}. $$ We need to sum this over all $(a_{i,j})$ and divide by the total count $$ \frac{52!}{4!4!4!40!}.$$ (Actually, we need just let $a_{1,2}, a_{2,3}, a_{3,1}$ run from $0$ to $4$ and this determines $a_{1,3}=4-a_{1,2}$ etc.) The final result is $$p=\frac{58388462678560}{7151046448045500}=\frac{24532967512}{3004641364725}\approx 0.008165 $$ (I just noted that Harold has performed a Monte Carlo simulation with matching result)

Another update:

As explained in the paper below, you can use rook polynomials to solve such problems. Playing with a full deck of 52 cards we will call "one" 18 times, we will call "two" 17 times, and we will call "three" 17 times. The forbidden positions in the 52 by 52 board consist of three "independent" complete rectangles; one $18\times 4$ and the other two $17\times 4$.

The rook polynomial for a full $m\times n$ rectangle with $m\geq n$ is $$\sum_{k=0}^n{m\choose k}\, {n!\over (n-k)!}\, x^k. $$

Multiply the polynomials for these three rectangles to give us the rook polynomial for our problem
$$R(x)=(1+73440x^4+19584x^3+1836x^2+72x)(1+57120x^4+16320x^3+1632x^2+68x)^2.$$

The number of winning deck orders is $$\int_0^\infty x^N R(-1/x) \exp(-x)\,dx $$ so the probability is this divided by $N!$, i.e. $$\mathbb{P}(\text{win})= 24532967512/3004641364725= 0.008165023553.$$

Reference: F. F. Knudsen and I. Skau, On the Asymptotic Solution of a Card-Matching Problem, Mathematics Magazine 69 (1996), 190-197.

Update: The solution below is for a simplified version of the problem where you work with a deck of size 12: four each of ace, deuce, and trey.

This is a problem in generalized derangements and joriki's answer here tells you what to do. In general, the number of deck orders that lead to a win is $$\int_0^\infty L_{n_1}(x)\cdots L_{n_r}(x)\,\mathrm e^{-x}\mathrm dx.$$

In this problem, we have $r=3$ and $n_1=n_2=n_3=4$. The fourth Laguerre polynomial is $L_4(x)=(x^4-16x^3+72x^2-96x+24)/24$. Raising this to the third power and integrating against $\exp(-x)$ gives $346$. That is, there are $346$ ways to order the deck that give a win.

Divide this by the total number of orders $12!/(4!)^3$, to give $$\mathbb{P}(\text{win})=173/17325=0.00998.$$