Deep Freeze for Linux. Reverting server to a previous state, kind of a "snapshot" being a physical machine (not a VM)

I work in an enviroment with a lot of Linux Servers and a few Windows servers. When we perfom some changes in the Windows ones it is very easy to go back to the original state using "Deep Freeze" software.

In the Linux servers if it is a VM we have the same result as before using Snapshots. It is very easy go back to the previous state of the server by going back with the snapshot.

Here is my problem... with physical machines we use parcial solutions for going back as rsync, yum history undo, yum histoty rollback <Trans.ID>, copy in the original folder the backups done previous to the patching, intallation, deployment activities, etc. Despite of that... We don´t have the 100% garantee of leaving the server exactly as it was before the change was made on it. Not with the same certainty as an snapshot or Deep Freeze.

With Windows Servers (Deep Freeze) or Linux VMs (Snapshot) we are complete calmed since we can always go back to the original position.

With phisical hosts/servers we are always very nervous. Servers are critical.

Is there any software Deep Freeze equivalent for Linux RHEL or any set of good practices to achieve the "go back" without beening scared when it comes to physical machines?

TO SUMMARIZE: Is there a way in Linux Physical Servers (RHEL) to go back to the exact point prior to any attempt of changing something when this attempt ended up being a totally failure and we decide to go back to square one?

Solution 1:

Use file system or volume snapshots, for instance, LVM is the way to go in RHEL. Modern LVM has thin pools which has lightweight snapshots, go for them if you can. Old RHEL could not support it, and "thick" LVM snapshots are believed to be inefficient, but still may be useful; I used them to guarantee fast recovery if large update causes problems. ZFS and, afaik, BTRFS has useful snapshots. Network-defined storage technologies, like CEPH, also support snapshots.

Use hardware features. Many SAN do support snapshots that are invisible by OS, those can be used to capture the state of whole bunch of hosted systems in a single moment of time, which is very good.

Those aren't replacements for backups, but you can take backups from those snapshots (this is actually a proper way to do a consistent backup of complex system).

I think you are too reliant on that Deep Freeze. You need to have an actual detailed plan for a disaster recovery (I mean: knowing in advance what to do, who is doing that, when they must to do that, what they do if something goes wrong like failed backup media or the restored image can't work on replacement hardware, what are dependencies between systems like how MS AD will behave if we restore DC from a month-ago backup and so on). This is the point where you can calm down. It could happen that recovering system from some old image is often not the first and not most important option in this plan. You might even discover that you don't need anything like Deep Freeze for Linux bare metal machines at all.

And go to virtualisation as far as you can. The possibility to easily backup and restore machines and entire virtual networks consisting of bunches of machines is one of its biggest benefits, you seem to already feel that. It enables for very straightforward disaster recovery strategies.