RejectedExecutionException inside single executor service

In one of our services, someone added such (simplified) a piece of code:

public class DeleteMe {

    public static void main(String[] args) {

        DeleteMe d = new DeleteMe();
        for (int i = 0; i < 10_000; ++i) {
            d.trigger(i);
        }
    }

    private Future<?> trigger(int i) {

        ExecutorService es = Executors.newSingleThreadExecutor();
        Future<?> f = es.submit(() -> {
            try {
                // some long running task
                Thread.sleep(10_000);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
        });

        return f;
    }
}

This fails sometimes with:

Exception in thread "main" java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3148f668 rejected from java.util.concurrent.ThreadPoolExecutor@6e005dc9[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
    at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
    at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
    at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678)
    at com.erabii.so.DeleteMe.trigger(DeleteMe.java:29)
    at com.erabii.so.DeleteMe.main(DeleteMe.java:22)

Most of the time the error is OutOfMemoryError - which I perfectly understand. The person writing the code never invoked ExecutorService::shutDown, thus keeping it alive too much. Of course creating a separate executor service for each method call is bad and will be changed; but this is exactly why the error is seen.

The point that I do not understand is why RejectedExecutionException would be thrown, specifically it is being thrown here.

Code comments there make some sense:

3. If we cannot queue task, then we try to add a new thread. If it fails, we know we are shut down or saturated and so reject the task.

If this is indeed the case, how come the documentation of execute does not mention this?

If the task cannot be submitted for execution, either because this executor has been shutdown or because its capacity has been reached, the task is handled by the current RejectedExecutionHandler.

To be frank initially I though that ExecutorService is GC-ed - reachability and scope are different things and GC is allowed to clear anything which is not reachable; but there is a Future<?> that will keep a strong reference to that service, so I excluded this.

Solution 1:

You wrote

To be frank initially I though that ExecutorService is GC-ed - reachability and scope are different things and GC is allowed to clear anything which is not reachable; but there is a Future<?> that will keep a strong reference to that service, so I excluded this.

But this is actually a very plausible scenario, which is described in JDK-8145304. In the bug report's example the ExecutorService is not held in a local variable, but a local variable does not prevent garbage collection per se.

Note that the exception message

Task java.util.concurrent.FutureTask@3148f668 rejected from  
    java.util.concurrent.ThreadPoolExecutor@6e005dc9[Terminated,
        pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]

supports this, as the state of ThreadPoolExecutor@6e005dc9 is specified as Terminated.

The assumption that futures hold a reference to their creating ExecutorService is wrong. The actual type depends on the service implementation, but for the common ones, it will be an instance of FutureTask which has no reference to an ExecutorService. It's also visible in the exception message that this applies to your case.

Even if it had a reference, the creator would be the actual ThreadPoolExecutor, but it is the wrapping FinalizableDelegatedExecutorService instance which gets garbage collected and calls shutdown() on the ThreadPoolExecutor instance (Thin wrappers are generally good candidates for premature garbage collection in optimized code which just bypasses the wrapping).

Note that while the bug report is still open, the problem is actually fixed in JDK 11. There, the base class of FinalizableDelegatedExecutorService, the class DelegatedExecutorService has an execute implementation that looks like this:

public void execute(Runnable command) {
    try {
        e.execute(command);
    } finally { reachabilityFence(this); }
}