How to run containers sequentially as a Kubernetes job?
I'm trying to replace my legacy job scheduler with Kubernetes job and wondering how to write sequential jobs as a Kubernetes job.
First, I wrote the following script to execute job1
and job2
in the written order but it didn't work as I expected.
apiVersion: batch/v1
kind: Job
metadata:
name: sequential
spec:
activeDeadlineSeconds: 100
template:
metadata:
name: sequential_jobs
spec:
containers:
- name: job1
image: image1
- name: job2
image: image2
restartPolicy: Never
The job described above seems to run job1
and job2
in parallel.
Is there any good way to run job1
and job2
in the written order?
Appended.
I recently found https://github.com/argoproj/argo very good for my usecase.
After a few attempts, I did this and that solved the basic problem (similar to what the OP has posted). This configuration ensures that job-1
completes before job-2
begins. If job-1
fails, job-2
container is not started. I still need to work on the retries and failure handling, but the basics works. Hopefully, this will help others:
apiVersion: v1
kind: Pod
metadata:
name: sequential-job
spec:
initContainers:
- name: job-1
image: busybox
# runs for 15 seconds; echoes job name and timestamp
command: ['sh', '-c', 'for i in 1 2 3; do echo "job-1 `date`" && sleep 5s; done;']
- name: job-2
image: busybox
# runs for 15 seconds; echoes job name and timestamp
command: ['sh', '-c', 'for i in 1 2 3; do echo "job-2 `date`" && sleep 5s; done;']
# I don't really need the 'containers', but syntax requires
# it so, I'm using it as a place where I can report the
# completion status
containers:
- name: job-done
image: busybox
command: ['sh', '-c', 'echo "job-1 and job-2 completed"']
restartPolicy: Never
Update
The same configuration as above also works inside a Job spec:
apiVersion: batch/v1
kind: Job
metadata:
name: sequential-jobs
spec:
template:
metadata:
name: sequential-job
spec:
initContainers:
- name: job-1
image: busybox
command: ['sh', '-c', 'for i in 1 2 3; do echo "job-1 `date`" && sleep 5s; done;']
- name: job-2
image: busybox
command: ['sh', '-c', 'for i in 1 2 3; do echo "job-2 `date`" && sleep 5s; done;']
containers:
- name: job-done
image: busybox
command: ['sh', '-c', 'echo "job-1 and job-2 completed"']
restartPolicy: Never
Argo workflow will fit for your usecase. Argo will support sequential, parallel, DAG job processing.
# This template demonstrates a steps template and how to control sequential vs. parallel steps.
# In this example, the hello1 completes before the hello2a, and hello2b steps, which run in parallel.
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: steps-
spec:
entrypoint: hello-hello-hello
templates:
- name: hello-hello-hello
steps:
- - name: hello1
template: whalesay
arguments:
parameters: [{name: message, value: "hello1"}]
- - name: hello2a
template: whalesay
arguments:
parameters: [{name: message, value: "hello2a"}]
- name: hello2b
template: whalesay
arguments:
parameters: [{name: message, value: "hello2b"}]
- name: whalesay
inputs:
parameters:
- name: message
container:
image: docker/whalesay
command: [cowsay]
args: ["{{inputs.parameters.message}}"]
Have you looked at Brigade - https://brigade.sh . Script simple and complex workflows using JavaScript. Chain together containers, running them in parallel or serially. Fire scripts based on times, GitHub events, Docker pushes, or any other trigger. Brigade is the tool for creating pipelines for Kubernetes.