Assume $L$ is finite, say $\max L=m$. Then For any $n>m$ there exists $p$ with $n-p\in L$. As this implies that all prime gaps are $<m$, it is absurd. Hence $L$ is infinite.


Suppose there are about $f(N)$ losing points in $[1,N]$. There would be around $f(N+1)\approx f(N)+1f'(N)$ losing points in $[1,N+1]$, so $f'(N)$ is the chance that $N+1$ is a losing point.
On the other hand the odds that $N+1$ is a losing point is the chance that all $N+1-a$ are composite.
Pretend that the $N+1-a$ are all $O(N)$, then this chance would be $$\left(1-\frac1{\ln N}\right)^{f(N)}=f'(N)$$
Let $f(N)=(\ln N)^2g(N)$.
$(1-1/\ln N)^{\ln N}$ is roughly $1/e$, so the equation is approximately $$N^{-g(N)}=\frac{2\ln Ng(N)}N+\ln(N)^2g'(N)$$ If $g(N)\approx 1$, then the left-hand side is too small; if $g(N)\approx1-\epsilon$ then the left-hand side is too big. So I believe $f(N)\approx(\ln N)^2$
EDIT : It turns out this is wrong because the numbers are mostly odd. That makes it much easier for a new odd number to make the list. If say a fifth of the odd numbers are in the list, then the chance of a new even $N$ to make the list would be $(1-1/\ln N)^{N/10}$. This goes to zero much faster than $1/N^2$, so has a finite sum, and I expect a finite number of even $N$ to make the list.