Can somebody help me on how to zip these individual log files within while loop

Solution 1:

For compression, you can use an arbitrary compression tool, such as pigz (multithreaded gzip), pbzip2 (multithreaded bzip2) or xz -T0. The old and well-known .gz format is generated by pigz or gzip.

A few more tools and algorithms (some multithreaded, some single-threaded): zstd -T0, lrzip, lzop, lz4, lzip

However, your code snippet has an obvious data loss issue:

    ...
    cp $log_file $tmp_log_file
    # Here. This is a race window.
    # Whatever gets logged here will be lost!
    >$log_file
    ...

Attached is an outline showing how one could possibly address the atomicity (data loss) problem while also adding a standard .gz compression.

If the logger process is long-running and doesn't reopen the log file (on occasions other than a HUP signal), then you won't need the hard link trick (ln -f ...) and a mv followed by a kill -HUP will suffice. This approach would include a time window in which "$log_file" does not exist.

If the logger process can be restarted independently of the script's operation and/or if the log(s) can originate from multiple short-lived processes, then the atomic log file swap procedure shown below will be necessary to prevent data loss. This approach guarantees that (1) "$log_file" always exists and (2) all logs will be stored in exactly one of the per-minute log files.

#!/bin/bash

log_file='/home/tmp/filename.log'
old_log_file="/home/tmp/filename.log.$$.old"
new_log_file="/home/tmp/filename.log.$$.new"

for ((;;)); do
    sleep 60
    logger_PID="$(... find the PID of the logging process ...)"
    # Alternative to the following 3 lines: man renameat2
    ln -f "$log_file" "$old_log_file"  # keep writing the log
    touch "$new_log_file"              # prepare a new, empty log
    mv "$new_log_file" "$log_file"     # break the link atomically
    kill -HUP "$logger_PID"            # switch to the new log
    # At this point:
    # * $log_file is written.
    # * $old_log_file is stable.
    pigz < "$old_log_file" > "$log_file.$(date +%M%D%Y%H%M).gz"
    # This^^^ can be pbzip2 or any other compression tool.
done