Background tasks in Meteor
I'm wondering, is there is way to implement background taks, maybe with workers pool. Can you show me direction, i'm thinking about writing package for this?
Solution 1:
2019 update
Before thinking about writing a package for anything, first look if there are existing packages that do what you need. In the Meteor world, this means looking on Atmosphere for "job/queue/task/worker management/scheduling" packages, then on npm for the same search terms. You also need to define your requirements more precisely:
- do you want persistence, or would an in-memory solution work?
- do you want to be able to distribute jobs to different machines?
Meteor-specific
- job-collection - reliable (I used it in 2014 in production at a startup), but currently in maintenance mode. Lets you schedule persistent jobs to be run anywhere (servers, clients).
- SteveJobs - actively maintained by Max Savin, the author of several powerful Meteor tools
- littledata:synced-cron - "A simple cron system for Meteor. It supports syncronizing jobs between multiple processes."
Abandoned packages:
- artwells:queue - priorities, scheduling, logging, re-queuing. Queue backed by MongoDB. Last code commit: 2015-Oct.
- super basic cron packages: easycron. Last update: Dec 2015.
- differential:workers - Spawn headless worker meteor processes to work on async jobs. Last code commit: Jan 2015
- cron (since 2015)
- PowerQueue - abandoned since 2014. Queue async tasks, throttle resource usage, retry failed. Supports sub queues. No scheduling. No tests, but nifty demo. Not suitable for running for a long while due to using recursive calls.
Npm packages
Meteor has been able to use npm packages directly for several years now, so this question amounts to finding job/worker/queue management packages on NPM. If you don't care about persistence:
-
Async "provides around 70 functions that include the usual 'functional' suspects (
map,reduce,filter,each...) as well as some common patterns for asynchronous control flow (parallel,series,waterfall...)" - d3-queue - minimalistic, written by D3 author Mike Bostock
If you do want persistence, since Meteor uses MongoDB already, it may be advantageous to use a job scheduling package with persistence to MongoDb. The most powerful and popular seems to be Agenda, but unfortunately it hasn't been maintained in months, and it has a significant backlog of issues.
If you're willing to add a dependency backed by redis to your project, there are more choices:
- bull - the most full-featured job queue solution for Node, backed by Redis
- bee - simple, fast, robust. Does not suffer from a memory leak that Bull exhibits
- Kue - priority job queue for Node
Like MongoDB, Redis can also provide high-availability (via Redis Sentinel), and if you want to distribute jobs among multiple worker machines, you can point them all at the same Redis server.
Solution 2:
There is a package based on Cron jobs which can be used to schedule tasks on certain intervals or dates. Here is the package: https://atmosphere.meteor.com/package/cron
And if you happen to look into the source of that package, you'll notice they're simply using:
Meteor.setInterval( ... , delay );
So if you save your tasks in a database, then load them into intervals during startup, then you'll probably be on the right track.
Solution 3:
If you are looking for something that is specific to Meteor, I am happy to share there is a new package called Steve Jobs. It makes running background jobs as easy as calling a Method.
It has all the standard features you would expect, such as running a job only once, retrying failed jobs, and so on. You can learn more about it on GitHub:
http://github.com/msavin/stevejobs