tar gzip slowing down server

I have a backup script that:

1. compress some files
2. generate md5
3. copy the compressed file to another server.
4. the other server finishes comparing MD5 (to find copy errors).

Here it's the core script:

nice -n 15 tar -czvf $BKP $PATH_BKP/*.* \
| xargs -I '{}' sh -c "test -f '{}' && md5sum '{}'" \
| tee $MD5
scp -l 80000 $BKP $SCP_BKP
scp $MD5 $SCP_BKP

This routine got CPU at 90% at gzip routine, slowing down the production server. I tried to add a nice -n 15 but server still hangs.

I've already read 1 but the conversation didn't help me.

What is the best approach to solve this issue ? I am open to new architectures/solutions :)

Solution 1:

If you use nice, you change the priority, but this will have a noticeable impact only if the CPU is close to 100% usage.

The server becomes slow, in your case, not because of the CPU usage, but because of the I/O on the storage. Use ionice to change the I/O priority and keep the nice for CPU priority.

Solution 2:

You could try using chrt to change the scheduling policy of the tar program to SCHED_BATCH.

As per the man page sched_setscheduler(2)

SCHED_BATCH: Scheduling batch processes (Since Linux 2.6.16.) SCHED_BATCH can only be used at static priority 0. This policy is similar to SCHED_OTHER in that it schedules the process according to its dynamic priority (based on the nice value). The difference is that this policy will cause the scheduler to always assume that the process is CPU-intensive. Consequently, the scheduler will apply a small scheduling penalty with respect to wakeup behaviour, so that this process is mildly disfavored in scheduling decisions.

   This policy is useful for workloads that are noninteractive, but do not
   want to lower their nice value, and for workloads that want a determin‐
   istic scheduling policy without interactivity causing extra preemptions
   (between the workload's tasks).

If your still out of luck you could try SCHED_IDLE instead. This would make this program only wake up if there is nothing else to run.

This changes the tar line to this for batch:

nice -n 15 chrt -b tar -czvf $BKP $PATH_BKP/*.* \