Kubernetes or Docker-Compose - how to immediately spawn new docker image from queue

I have a list of scheduled work items.

1. When a work item is "ready", a "Scheduler" should dynamically start a "Worker" Docker image.
2. "Worker" runs 15-240 minutes to completion.
3. If not enough resources, "Scheduler" should wait and promptly try again.
4. I'll implement node scaling later.

What's the simplest way to implement the this in Docker-Compose or Kubernetes?

Solution 1:

1. When a work item is "ready", a "Scheduler" should dynamically start a "Worker" Docker image.

Many Kubernetes functions are now built in using custom resources which makes Kubernetes more modular. Custom resources let you store and retrieve data but when you combine it with custom controller , custom resources provide a true declarative API.

For you task you want to check an Operator pattern which is a combination of custom resources and custom controllers.

You Operator will have to watch for your work item state and once it is ready it triggers the Kubernetes Job that runs yours docker Worker image to completion.

Worth to mention that instead of operator you may use go-client or python-client to access Kubernetes RESTful API interface. client-go is very powerful lib for developing K8s custom controllers.

For more officially supported client lists please visit official documentation.

ClusterAutoscaler feature can used to scale up Kubernetes cluster when you need to run a Job and then scale it down when the job is finished.

If you wishes to read more about operator

Alternatively you may want to look into some kind CI/CD scheduler like Jenkins. Here are some good examples of kubernetes with jenkins:

• Create a CI/CD pipeline with kubernetes and jenkins
• Configuring CI/CD on kubernetes with jenkins

But in case of you are looking for something like message queue for managing task instead Jenkins it might be more convenient to have it all in the same Kubernetes cluster as Deployments and StatefulSets.

Here you have more information about message queue managemet:

• Coarse parallel processing work queue
• Fine parallel processing work queue

Lastly a small comparison between Kubernetes and Docker that can give you some more insight which one is more suitable for you.