Quartz.NET scheduler doesn't fire jobs/triggers once deployed
INTRODUCTION
I'm using Quartz.Net on an ASP.Net framework 4, webforms web site.
Basically, the user should have the hability to fire manually a batch script that asynchronously processes thousands of records stored on a database. The user can stop or pause at any time, adjust some variables, and continue if needed with the process (remaining records).
The code is done and working locally (developer machine, win7, vs2010, sql server express 2008 R2).
It was also tested on a local server (win server 2008 R2, sql server express 2008 R2).
It works fine on both enviroments, tested with all the code precompiled.
The problem is that, once deployed on a remote server (win server 2008 R2), where it actually should be running on (hosting enviroment, not shared, not clustered), it doesnt completely works (se details below). Scheduler gets created, but the trigger, hence the job, doesn't fire.
(Note: I know some of you would suggest to use Quartz as windows service, but despite the benefits of doing so, I really would like to find out why it doesn't work as an embedded solution, since it should be working just fine like does locally)
DETAILS
Quartz 2.1.2
Common.Logging 2.1.2
Common.Logging.NLog 2.0.0
NLog 2.0.1.2
global.asax
public static ISchedulerFactory SchedulerFactory;
public static IScheduler Scheduler;
void Application_Start(object sender, EventArgs e)
{
SchedulerFactory = new StdSchedulerFactory();
Scheduler = SchedulerFactory.GetScheduler();
// Define a durable job instance (durable jobs can exist without triggers)
IJobDetail job = JobBuilder.Create<MyJobClass>()
.WithIdentity("MyJob", "MyGroup")
.StoreDurably()
.Build();
Scheduler.AddJob(job, false);
Scheduler.Start();
}
void Application_End(object sender, EventArgs e)
{
Scheduler.Shutdown(true);
}
process.aspx.cs (start button click)
// get records from DB, iterate, process, etc
...
IJobDetail job = ASP.global_asax.Scheduler.GetJobDetail(new JobKey("MyJob", "MyGroup"));
job.JobDataMap.Put("something1", 1);
job.JobDataMap.Put("something2", somevar);
ITrigger trigger = TriggerBuilder.Create()
.WithIdentity("MyTrigger", "MyGroup")
.StartNow()
.WithSimpleSchedule(x => x.WithIntervalInSeconds(5).RepeatForever())
.Build();
var triggersSet = new Quartz.Collection.HashSet<ITrigger> { trigger };
ASP.global_asax.Scheduler.ScheduleJob(job, triggersSet, true);
LOG OUTPUT
local log
Default Quartz.NET properties loaded from embedded resource file
Using default implementation for object serializer
Using default implementation for ThreadExecutor
Initialized Scheduler Signaller of type: Quartz.Core.SchedulerSignalerImpl
Quartz Scheduler v.2.1.2.400 created.
RAMJobStore initialized.
Scheduler meta-data: Quartz Scheduler (v2.1.2.400) 'DefaultQuartzScheduler' with instanceId 'NON_CLUSTERED' Scheduler class: 'Quartz.Core.QuartzScheduler' - running locally. NOT STARTED. Currently in standby mode. Number of jobs executed: 0 Using thread pool 'Quartz.Simpl.SimpleThreadPool' - with 10 threads. Using job-store 'Quartz.Simpl.RAMJobStore' - which does not support persistence. and is not clustered.
Quartz scheduler 'DefaultQuartzScheduler' initialized
Quartz scheduler version: 2.1.2.400
Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED started.
Batch acquisition of 0 triggers
Batch acquisition of 0 triggers
It continues logging Batch acquisition of 0 triggers until button click occurs:
Default Quartz.NET properties loaded from embedded resource file
Batch acquisition of 1 triggers
Producing instance of Job 'MyGroup.MyJob', class=MyJobClass
Batch acquisition of 0 triggers
Calling Execute on job MyGroup.MyJob
Trigger instruction : NoInstruction
Batch acquisition of 1 triggers
Producing instance of Job 'MyGroup.MyJob', class=MyJobClass
Batch acquisition of 0 triggers
Calling Execute on job MyGroup.MyJob
Trigger instruction : NoInstruction
Batch acquisition of 1 triggers
deployed log
Default Quartz.NET properties loaded from embedded resource file
Using default implementation for object serializer
Using default implementation for ThreadExecutor
Initialized Scheduler Signaller of type: Quartz.Core.SchedulerSignalerImpl
Quartz Scheduler v.2.1.2.400 created.
RAMJobStore initialized.
Scheduler meta-data: Quartz Scheduler (v2.1.2.400) 'DefaultQuartzScheduler' with instanceId 'NON_CLUSTERED' Scheduler class: 'Quartz.Core.QuartzScheduler' - running locally. NOT STARTED. Currently in standby mode. Number of jobs executed: 0 Using thread pool 'Quartz.Simpl.SimpleThreadPool' - with 10 threads. Using job-store 'Quartz.Simpl.RAMJobStore' - which does not support persistence. and is not clustered.
Quartz scheduler 'DefaultQuartzScheduler' initialized
Quartz scheduler version: 2.1.2.400
Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED started.
Here stays like this. As you see, compared to the other log, it's not trying to acquire triggers (line Batch acquisition of 0 triggers does not appear at all). If you click the process button anyway, the log adds one line:
Default Quartz.NET properties loaded from embedded resource file
But nothing else happens. The records are not processed (I know since every time a record is proccessed, is marked in the database). No errors occur, but the trigger is not fired, and the job is not executed. Also, the CPU usage run up to 50% or more on button click, and doesnt gets down unless you go to IIS, stop and restart the application pool. This cpu consumption doesn't happen locally.
update 1
Changed use of scheduler for a singleton, as suggested by LeftyX, but still get same behavior on remote server.
update 2
I also tried to use ADOJobStore (instead of RAMJobStore which I was using). Now it still works perfectly locally; but still doesn't execute the trigger (hence job) online. The only difference is that online the CPU usage doesn't run up to 50%. And now I can see that the job and trigger are created (I query the tables and see that those records exists), but never gets executed.
Solution 1:
There are nothing wrong with Quartz, all because of IIS app pool recycling.
I fixed the bug by stopping the pool that is used for Quartz from recycling:
- Go to IIS manager -> Application Pools -> Create a new pool, I named it Scheduler (any name is ok)
- Select Scheduler pool -> advanced Settings
- In General section, at Start Mode, Select AlwaysRunning (IIS 8.5) or true for (IIS 7.5, 8)
- In Process Model Section-> Idle Timeout(minutes) set to 0 (meaning: No Idel timeout)
- In Recycling section -> Regular time Interval set to 0 (meaning: no recycling)
3. Deploy your Quartz site into that application pool. And send one request to the pool to "wake your app up" and it will run until u stop it.
That's it.
Update: Another solution to keep your app pool always alive is using Auto-Start ASP.NET Applications
Another option: Using some third party pinging tool (like uptimerobot or diy one ) to keep refreshing your site each and every x seconds (or minute)
Solution 2:
One thing that I have noticed is the use of the Scheduler in your asp.net application.
You should use singleton objects.
in your process.aspx.cs this line
IScheduler scheduler = new StdSchedulerFactory().GetScheduler();
creates a new scheduler but you should use the one you've created as static in Application_Start.
If you want to get access to the singleton instance use a public memeber in your Global.asax.cs:
public static ISchedulerFactory SchedulerFactory;
public static IScheduler Scheduler;
and you can reference it in your process.aspx.cs:
MvcApplication.Scheduler.ScheduleJob(job, triggersSet, true);
Another solution is to use dependency injection. You can find some info here using StructureMap and here for Unity.
UPDATE:
You can download a sample application (asp.net 4.0) called AspNet_Quartz here and see how it works here.
Solution 3:
The problem is related to IIS rather than the schedulers Quartz.NET, Hangfire, etc. On the other hand, there are lots of solution methods posted on the web, but only some of them is working. In my opinion, there is no need to apply lots of configuration settings. Just install Keep Alive Service For IIS 6.0/7.5 on the server to which you publish your application and enjoy. After that, your published application will be alive after application pool recycling, IIS/Application restarting, etc. Hope this helps...
Solution 4:
I just ran into a similar issue that might bite someone else - it brought me to this SO question after turning on debug and getting that 'batch acquisition of 0 triggers' message.
I had a job that was every 2 hours like this:
"0 0 0/2 * * ?"
And I wanted to make it more often, so every 2 minutes like this:
"0 0/2 0 * ?"
I even tried https://cronexpressiondescriptor.azurewebsites.net/ which gave me a big clue I should have read more carefully: Every 2 hours, on day 0 of the month which eventually made me realize what I really meant was:
"0 0/2 * * ?"
So the lesson was, when 'shifting left' your cron, back-fill with *, not 0.
Hope that helps someone else.