In Ubuntu ISO MBR, why does a GPT partition overlap the boot partition, and make MBR uneditable?

Question and context:

In the MBR partition table of distros like for example ubuntu-16.04-desktop-amd64.iso, why does a GPT partition appear to overlap the primary boot partition? This seems to generate errors if I try, in any way, to edit the partition table.

I want to edit the MBR to add USB persistence (as described elsewhere), and must add a partition or enlarge the boot partition.

I believe this used to work, but the overlapped GPT #2 partition in this and similar distros appears to badly confuse fdisk, sfdisk, parted, gparted, and partprobe.

My machine is a MBR, and not a GPT bios.

What am I missing?

Here is the distro's MBR partition table (directly from the ISO file):

cat ubuntu-16.04-desktop-amd64.iso | xxd | head -32 | tail -5

gives:

00001b0: 28db 2b00 0000 0000 708e 0e0e 0000 8000  (.+.....p.......
00001c0: 0100 0058 e0fa 0000 0000 6048 2c00 00fe  ...X......`H,...
00001d0: ffff effe ffff 4411 2c00 8012 0000 0000  ......D.,.......
00001e0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00001f0: 0000 0000 0000 0000 0000 0000 0000 55aa  ..............U.

(To decode this see https://en.wikipedia.org/wiki/Master_boot_record.)

• The table starts at address 01be, where you see "80", for the #1 primary boot partition.

• Then notice the "ef" at address 01d2 which denotes the #2 primary partition as being type=GPT.

Here's a breakdown of the table by partition (little endian):

partition#1 (normal MBR):
  80        = 'boot' partition flag
  00 01 00  = starting HSC (head, sector, cylindar)
  00        =  partition type ("Empty partition entry")
  58 e0 fa  = last     HSC (head, sector, cylindar)

  0000 0000 = LBA (logical block address) of first absolute sector in the 
  6048 2c00 = number of sectors in partition

partition #2 (GPT):
  00        = non-boot partition
  fe ff ff  = starting HSC (head, sector, cylindar)
  ef        = partition type ("EFI system partition")
  fe ff ff  = last HSC

  4411 2c00 = LBA (logical block adr) of first abs sector in part.
  8012 0000 = number of sectors in partition

Attempts to edit partition table:

• fdisk reports that these partitions overlap. Note how the beginning of sdb2 [2927216] is inside of sdb1 [0-2955679] from

  sudo fdisk -l /dev/sdb
  

  gives:

    Disk /dev/sdb: 14.5 GiB, 15527313408 bytes, 30326784 sectors
    Units: sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disklabel type: dos
    Disk identifier: 0x40a863e7
  
    Device     Boot   Start     End Sectors  Size Id Type
    /dev/sdb1  *          0 2955679 2955680  1.4G  0 Empty
    /dev/sdb2       2927216 2931951    4736  2.3M ef EFI (FAT-12/16/32)
  

• From what I've read, it's ok to have a GPT partition, but not for partitions to overlap.

• When I try to dd the iso to a usb stick, the trouble begins. The stick boots Ubuntu live ok, but I can not edit the partition table without errors (that is while not booted and not mounted), no matter which partition editor I try.

• Furthermore, I can't even delete the partition table and return the USB stick to normal without the following lengthy procedure:

  dd if=/dev/zero of=/dev/sdb bs=[something like 512 or 2048; doesn't matter] count=[some large number like 100000^]
  

  ^Smaller numbers don't always seem to fix this. I suspect you need to not only wipe out the primary fs blocks, but the secondary ones also.

  and then use fdisk to make a new ms-dos partition table on the stick.

  Note, if I don't do the dd above to wipe before creating a new partition table, I get the following error message or similar when I try to edit the partition map (from either debian8 or ubuntu-16):

  Libparted Warning The driver descriptor says the physical block size
  is 2048 bytes, but Linux says it is 512 bytes.
  

• Also from ubuntu-12, I get this most enlightening message:

  gparted -l /dev/sdb:  libparted : 2.3 Could not stat device -l -- No
  such file or directory. /dev/sdb contains GPT signatures, indicating
  that it has a GPT table.  However, it does not have a valid fake msdos
  partition table, as it should.  Perhaps it was corrupted -- possibly
  by a program that doesn't understand GPT partition tables.  Or perhaps
  you deleted the GPT table, and are now using an msdos partition table.
  Is this a GPT partition table?  Both the primary and backup GPT tables
  are corrupt.  Try making a fresh table, and using Parted's rescue
  feature to recover partitions.
  

So how do I edit the distro's MBR?

BTW, I'm NOT asking how to fix the overlap, as this is the way the distro actually is, and from the looks of it, has been for some time, (I also looked at Ub-v16, 14 and 12), rather I'm looking to figure out how to edit it if possible.

Ideas:

• Could this be caused by too many applications of isohybrid (used to add a ms-dos MBR to a CDROM 9660 iso so it can boot on a USB)?

• Is there an other, possibly newer partition table editor to use?

• Is there a different, perhaps older, Ubuntu distro which doesn't have this issue?

Solution 1:

Ubuntu's .iso image files use a Frankenstein's Monster format that's designed to support multiple boot methods and devices:

• Boot media
  • Writing the image to a DVD for booting using ISO-9660 and El Torito
  • Copying the image "raw" to a USB flash drive or similar media for booting using conventional disk partitioning systems
• Boot mode
  • BIOS-mode booting
  • EFI-mode booting

To work with so many formats and boot methods, the developers play games with the data structures. These images really should not be treated like "normal" disk images, and in particular, you should most emphatically NOT attempt to modify the images in any way unless you're a top-level expert in these data structures. FWIW, I wrote the GPT fdisk (gdisk, cgdisk, and sgdisk) partitioning tool, and I would not attempt what you say you're attempting!

Instead, if you need to modify the installation medium, you should do one of two things:

• Use a tool like Rufus, Pen Drive Linux, or UNetbootin to write the image file to a USB flash drive, then modify the USB flash drive. The reason this works is that these tools don't do a simple dd-like copy of the image's contents to disk; instead, they take the files from the image and copy them to a conventionally partitioned disk with a pre-existing filesystem (or sometimes one that the tool creates). The result is a more normal disk that can be modified with normal partition table editors and other utilities.
• Prepare your own custom image. I don't have pointers to documentation on how to do this, but the Ubuntu images are of course created by automated tools, not by hand-editing partition tables with hex editors. If you track down the tools and scripts used to create the images, you may be able to create an image that's modified in a way you would want. This approach is used by people who create "spins" of Ubuntu, so you might want to start by looking for documentation on how to do this.

Of these two approaches, the first is probably the better one for what you want. In fact, I'm pretty sure some tools of that sort offer an option to do precisely what you want. (I don't recall which tool(s) offer such a feature, though.) The second approach is more likely to be useful for tasks like adding packages to a custom installation image.

If you were ever able to do this sort of thing before, chances are it's changed because the developers have found a need to make a more monstrous hack of a data structure in order to work around problems with particular systems. For instance, if Brand X's computers can't parse the Frankenstein's Monster partition table, the developers might tweak it so that it works with Brand X computers, even if that means the data structure is even weirder than it had been before. This is just speculation on my part, though, and I certainly can't speak to the specifics you've described.

Solution 2:

Workaround

You can adjust your USB stick's partition sizes, if rather than using a USB maker tool to create the stick, you instead build your USB stick from scratch using only terminal commands. Here are detailed instructions that work if your system uses BIOS:

How to create a live Ubuntu USB drive with persistence for BIOS using only the terminal

Usage Tips for backing up using a USB stick: I keep two identical USB backup-OS sticks that I use to backup my main Debian dual boot W10 system. I want my system to not be running when I'm doing the backup. I also want two identical sticks, because sometimes a USB stick wears out. This allows me to easily make a new one just by a dd copy from the remaining good stick to a new stick.

They have saved me more than once when I got my system so broken that I could not fix it! And the incremental backup and restore procedures I include at this link above are pretty fast too. You should however, do a full backup from time to time, as it's possible for your current incremental to not work when you want it because a system flaw got written to it, corrupting it. I now try to do a full backup weekly.