Spring Scheduler stops unexpectedly

We have a Spring 3 web application on Tomcat 6 that uses several scheduled services via @Scheduled (mainly for jobs that run every night). Now it appears that sometimes (rarely, perhaps once in two months or so) the scheduler thread stops working, so none of the jobs will be executed in the following night. There is no exception or logging entry in our log files.

Has anybody a clue why this is happening? Or how to get more information about this problem?

Is there a way to detect this situation within the application and to restart the scheduler?

Currently we are solving this by having also a logging job that runs every 5 minutes and creates a log entry. If the log file stops being updated (monitored by nagios), we know it is time to restart tomcat. It would be nice to restart the jobs without a complete server restart.

Solution 1:

Since this question got so many votes, I'll post what the (probably very specific) solution to my problem was.

We are using the Apache HttpClient library to make calls to remote services in the scheduled jobs. Unfortunately there are no default timeouts set when performing requests. After setting

connectTimeout
connectionRequestTimeout
socketTimeout

to 30 seconds the problem was gone.

int timeout = 30 * 1000; // 30 seconds
RequestConfig requestConfig = RequestConfig.custom()
        .setConnectTimeout(timeout)
        .setConnectionRequestTimeout(timeout)
        .setSocketTimeout(timeout).build();
HttpClient client = HttpClients.custom()
        .setDefaultRequestConfig(requestConfig).build();

Solution 2:

This is pretty easy to find out. You would be doing this with a stack trace. There are many posts on how to get a stack trace, on unix system you do 'kill -3 ' and the stack trace appears in the catalina.out log file.

Once you have a stack trace, find the scheduler thread and see what it is doing. Is it possible that the task it was executing got stuck?

you can also post the stack trace here for more help.

what is important to know is what scheduler you use. if you use the SimpleAsyncTaskExecutor, it will start a new thread for each task, and your scheduling will never fail. However, if you have tasks that don't finish, you will run out of memory eventually.

http://docs.spring.io/spring/docs/3.0.x/reference/scheduling.html