How is it easier to make a persistent live drive with Ubuntu 19.10?

Solution 1:

Persistent live drive with a partition for persistence

Yes, there are some web sites describing new tools and new features in [new versions of] old tools that can create a persistent live drive from an iso file with Ubuntu 19.10 and the corresponding Ubuntu family flavours.

A new feature or fixed bug ('feature request' or 'bug fix' according to this link) makes it possible to use a partition for persistence, when booting directly from a cloned or extracted copy from the iso file.

This makes it much easier to create a live drive with a partition for persistence with Ubuntu 19.10.
In previous versions of Ubuntu it was possible to use a file in a FAT32 file system for persistence. Its size is limited to 4 GiB. It was possible to boot via grub in a separate partition (from the content of the iso file) and use a partition for persistence. But this is more complicated so not suitable for manual methods.
In 19.10 and future versions the size of a partition for persistence is only limited by the size of the drive (USB pendrive, SSD, HDD, memory card).
Debian 10 live iso files have also this feature, and can be used to create persistent live drives with a partition for persistence.
Newest new: In the developing Focal Fossa, to be released as 20.04 LTS, the default label of the partition for persistence is changed from casper-rw to writable. This is implemented in mkusb-plug version 2.5.5, (which is bundled with mkusb version 12.4.3 in the PPA).

Tools

Ubuntu and Debian

mkusb - works with all current versions of Ubuntu

If you run standard Ubuntu live, you need an extra instruction to get the repository Universe. (Kubuntu, Lubuntu ... Xubuntu have the repository Universe activated automatically.)
```
sudo add-apt-repository universe  # only for standard Ubuntu

sudo add-apt-repository ppa:mkusb/ppa  # and press Enter
sudo apt-get update
sudo apt-get install mkusb mkusb-nox usb-pack-efi
```
mkusb-minp - small stand-alone shellscript that can use this new feature in Ubuntu 19.10. It works with Debian 10 too.

Please read the warnings when you try older versions or re-spins (distros based on Ubuntu)

Beginning of dialogue:

End of dialogue:
mkusb-plug - small set of shellscripts that can use this new feature in Ubuntu 19.10. It works with Debian 10 too. I think you will find things easier with this new and very safe tool with a graphical user interface.

Windows

Update: Rufus - a well-known tool that can use this new feature in Ubuntu 19.10. Be sure to use the current version (Rufus 3.9 or newer),
- works well to create live-only drives with all current Ubuntu versions,
- can create persistent live drives with Ubuntu 19.10 and future versions. Please upgrade to version 3.9 (or newer), which creates a good ext3 file system and can make good persistent live drives.
- Please read the changelog at the web site, and the warnings (pop-up windows), they can help you avoid serious mistakes.
- Edit: Rufus 3.9 has arrived :-)
Simple method where the iso file is edited before cloning

This new feature in Ubuntu 19.10 can be used also from Windows,
- Use HxD (a binary editor) to edit the iso file in order to replace two cosmetic boot options 'quiet splash' with 'persistent ' (replace 12 characters with 12 characters)
- Use Win32 Disk Imager to clone the edited iso file
- The first time the cloned drive is booted, the Ubuntu system will create a casper-rw partition with an ext4 file system automatically. Simple and robust!
- There are detailed descriptions at these links:
  - wiki.ubuntu.com/Win32DiskImager/iso2usb/persistent
  - wiki.ubuntu.com/Win32DiskImager/iso2usb

MacOS and Windows and Linux

Unetbootin - you have to create a casper-rw partition for persistence manually and remove or rename the casper-rw file

See this link describing how to add a persistent partition to a UNetbootin live/persistent USB (Ubuntu 19.10+)

Please notice 'Space used to preserve files across reboots ...'
Compressed image file

You can extract and clone a compressed image file directly according to the following link,

Xubuntu Core 20.04 LTS with mkusb 12.4.5 (mkusb-dus and mkusb-plug)

This works in all operating systems where you have
- a tool that can extract a file compressed with xz
- a cloning tool
.

Manual method

You need only a few manual steps to create a persistent live drive with a partition for persistence, when you are running Ubuntu.

Edit the iso file to replace quiet splash with persistent. Yes, you can edit the binary iso file and replace 12 characters with 12 other characters and flash the output to the target device (usually a USB pendrive). sed can do it.
Create a partition 'behind' the flashed copy of the edited iso file. fdisk can do it.
Create an ext2 file system in this partition and put the label casper-rw on this partition. mkfs.ext2 can do it.
Flush the buffers. sync can do it.

This is what mkusb-minp is doing plus a lot of checking, that wrap a safety belt around the process. In other words, you get help to write to the correct target device and get warnings when there might be problems.

You can do it yourself and feel every step, or you can read the code of the shellscript mkusb-minp and understand the details.

Please notice that I use dus and select 'Cloning' in the following example. You can use another cloning tool, but if you use a simple tool like dd, please double-check that you are cloning to the currect device, and that all partitions on that device are unmounted.

strings lubuntu-19.10-desktop-amd64.iso |grep 'quiet splash'  # check that 'quiet splash' is there to be replaced by 'persistent  ' (12 characters)
sed 's/quiet splash/persistent  /' lubuntu-19.10-desktop-amd64.iso > persistent-lubuntu-19.10-desktop-amd64.iso  # yes, sed works with binary files
ls -l *19.10*  # check that the size is the same
strings persistent-lubuntu-19.10-desktop-amd64.iso |grep 'persistent  '  # check that 'persistent' is there now
dus persistent-lubuntu-19.10-desktop-amd64.iso  # I use mkusb-dus, you can use the Ubuntu Startup Disk Creator or another cloning tool
sudo lsblk -fm  # It is important to check the device letter of the target drive (the USB drive, that you want persistent live
sudo fdisk /dev/sdx  # x is the device letter of the target drive, please double-check that you have the correct letter
n           # new partition
p           # primary
<Return>    # default: 3
<Return>    # default: next free sector
<Return>    # default: last addressable sector
w           # write and quit
sudo lsblk -fm  # check that things look good and verify that partition #3 is the correct partition to be used to store the persistent data
sudo mkfs.ext2 -L casper-rw /dev/sdx3  # put label and file system into the partition of persistence
sudo lsblk -fm  # check that things look good
sync  # flush the buffers and wait for prompt

A cloned Ubuntu 19.10+ live drive is not really live-only

Isn't live the default for a cloned drive?

Well, it used to be, and it is still live but not live-only in 19.10.

If you want to make a purely live-only drive, where nothing will be preserved after shutdown and reboot, you can

use mkusb-minp with the option -n or mkusb-plug and select 'No-persistent live drive'.
or manually modify the built-in boot options to replace 'quiet splash' with 'nopersistent'

Link

How can I get a live-only drive with Ubuntu 19.10 and newer versions?

Solution 2:

I wrote a bash script (one liner) to edit live USB disk to change "quiet splash" to "persistent " based on the answer by @sudodus.

Disclaimer: it worked on my system, use at your own risk. It is for bash, may not work on other shells w/out edit.

After boot from USB extra partition writable was created w/out my intervention for overlay for persistence, if not fdisk can be used.

First variant of code is not efficient as it starts search with grep all over after each edit; but it can be stopped via ctrl-c when some changes are made.

device_to_edit=/dev/sdb; string_to_edit="quiet splas\x68"; new_string="persistent  "; block_size=512; ed_count=0; to_end=""; if [ ! ${#string_to_edit} -eq ${#new_string} ]; then echo -e "\n Suspect a danger as strings lengths not same"; elif [ $block_size -lt ${#new_string} ]; then echo -e "\n\ Suspect a danger as block size too small for string length"; elif file_size=$(lsblk -b --output PATH,SIZE | grep "$device_to_edit " | sed -E 's/.* ([0-9]{1,})/\1/'); [ $((file_size/1024/1024/1024)) -gt 20 ]; then echo -e "\n Suspect a danger as device size is large"; else while [ -z $to_end ]; do offset_to_edit=$(sudo grep --only-matching --byte-offset --max-count=1 --text --perl-regexp "$string_to_edit" $device_to_edit | awk --field-separator ":" '{ FS = "0" ; print $1 ; exit }'); if [ -z "$offset_to_edit" ]; then echo -e "\n end of file is reached" ; to_end="yes" ; else block_to_edit=$(($offset_to_edit/$block_size)) ; sudo dd if=/$device_to_edit count=2 skip=$block_to_edit bs=$block_size | sed "s/$string_to_edit/$new_string/" | sudo dd conv=notrunc of=$device_to_edit count=2 seek=$block_to_edit bs=$block_size; echo "blocks $block_to_edit and next edited (blocks start with 0)"; ((ed_count++)) ; fi ; done ; echo -e "\n$ed_count editions made"; fi

In second variant of code grep is done only one time, but grep goes through complete file (e.g. block device, USB) which can take significant time before any editing (on my system it took ~ 15 minutes for 8Gb drive, I added date for the sake of that curiosity):

date; device_to_edit=/dev/sdb; string_to_edit="quiet splash"; new_string="persistent  "; block_size=512; ed_count=0; to_end=""; if [ ! ${#string_to_edit} -eq ${#new_string} ]; then echo -e "\n Suspect a danger as strings lengths not same"; elif [ $block_size -lt ${#new_string} ]; then echo -e "\n\ Suspect a danger as block size too small for string length"; elif file_size=$(lsblk -b --output PATH,SIZE | grep "$device_to_edit " | sed -E 's/.* ([0-9]{1,})/\1/'); [ $((file_size/1024/1024/1024)) -gt 20 ]; then echo -e "\n Suspect a danger as device size is large"; else sudo grep --only-matching --byte-offset --text --perl-regexp "$string_to_edit" $device_to_edit | awk --field-separator ":" '{ FS = ":" ; print $1 }' | { IFS=$'\n' read -d '' -a offsets_to_edit ; for offset in ${offsets_to_edit[@]}; do block_to_edit=$(($offset/$block_size)) ; sudo dd if=/$device_to_edit count=2 skip=$block_to_edit bs=$block_size | sed "s/$string_to_edit/$new_string/" | sudo dd conv=notrunc of=$device_to_edit count=2 seek=$block_to_edit bs=$block_size; echo "blocks $block_to_edit and next edited (blocks start with 0)"; ((ed_count++)); done; echo -e "\n$ed_count editions made"; } ; fi; date

Notes:

Meaning of the scripts parameters and syntax could be understood by reading man bash, man awk, man sed, info '(coreutils) dd invocation', help read. Why (list) / { list; } needed in pipe is explained in https://stackoverflow.com/questions/2746553/read-values-into-a-shell-variable-from-a-pipe ("A pipeline may spawn a subshell, where the environment is inherited by value, not by reference"), working with two blocks is needed in case string starts in one block and ends in next one.

The scrips include three safety checks: lengths of original and replacement strings are same, block size is sufficiently large, device size is not larger then 20Gb (against accidental edit of main drive, need to be edited for larger flash storage, code is $((file_size/1024/1024/1024)) -gt 20).

There are some redundancies in the scripts, i.e. FS = ":" ; in awk, for some reason field separator did not work as I expected and I had to add --field-separator ":", also in grep --perl-regexp is needed to use hex e.g. \xaa values, but this edit does not have them.