Debian: creating Windows 10 bootable install USB-drive using terminal (dd)

Solution 1:

I want to understand what is going on

Rufus developer here.

What way too many people fail to understand, because Linux ISOs are applying this method, but this is essentially a MAJOR HACK CALLED 'ISOHYBRID', is that, in most cases, you cannot simply take an ISO image and copy it byte for byte to a USB drive, and expect that to boot.

That is because the ISO format and the underlying file systems it uses (ISO9660 or UDF) are designed for optical boot, which is a completely different beast from regular HDD or USB boot. For one thing optical media, and therefore (regular) ISO images, don't have a partition table, which is (usually) essential for HDD or USB boot, and they also (usually) don't have a Master Boot record, a.k.a. MBR, which is essential for BIOS boot.

This means that, if you do a 1:1 copy of a regular ISO, such as Windows one, onto a disk, and try to boot this is what's going to happen:

• A BIOS system or UEFI system in Legacy/CSM mode will not see any MBR, and especially it will not see the 0x55 0xAA sequence in the very last 2 bytes of the MBR that indicates that a disk is BIOS-bootable. Therefore it won't be able to boot that disk in BIOS mode.
• A UEFI system will (usually) not mount UDF or ISO9660 partitions from a disk or flash drive media, because, even as it has drivers for these files systems, the resulting disk you created will be missing an MBR or GPT partition table. When booting a regular disk, UEFI is designed to first look for a partition, and then look for a bootloader (e.g. /efi/boot/bootx64.efi) on that partition. So if there is no MBR or GPT partition table on the media, which will be the case for a regular ISO, then it doesn't matter if the ISO contains a bootloader file, because the UEFI firmware will not be able to mount the partition it resides on.

So, what utilities like Rufus do when creating a bootable disk media from a Windows ISO, which is a completely standard optical media image, is:

• Create a partition table, either MBR or GPT according to what the user selected, and create at least one partition, that will typically use FAT32 or NTFS as the file system (notice that it uses completely different file systems from what an ISO uses).
• If MBR is used, a bit of code in the MBR that locates the secondary boot loader, on the relevant MBR partition, which is in designed to start the execution of the Windows kernel, in disk mode, from that partition. Oh, and it also ensures that the 0x55 0xAA boot marker is added at the end of the MBR so that BIOS sees the disk as bootable. Then it also copies the content of the ISO onto a FAT32 or NTFS partition.
• If GPT is used, Rufus verifies that there actually exists a UEFI bootloader file, such as /efi/boot/bootx64.efi (well, actually it does that before you it allows you to select GPT, coz there's not much point in trying to create a GPT bootable drive if there is no UEFI bootloader) and then copies it, along with the rest of the ISO files, typically onto a FAT32 partition, since boot from a FAT32 partition is a mandatory requirement of UEFI (but that does not mean UEFI can't boot from NTFS or exFAT if you have the relevant UEFI drivers, which can come handy if you have a Windows ISO with a file that is larger than 4 GB, as FAT32 cannot accommodate such files).

Now, the above only works when the secondary bootloaders (i.e. the ones that comes from Windows and which Rufus doesn't modify) are designed to support both optical and regular boot, which typically mean they need to handle both UDF or ISO9660 and FAT32 or NTFS file systems, as well as the other differences that present themselves when booting from disk vs from optical. But Microsoft did design its bootloader precisely for that, which is the smart thing to do, because, if your target system is UEFI, it means you (usually, as long as the 4 GB max filesize issue of FAT32 doesn't rear its ugly head) don't need a utility to convert an ISO to a bootable USB, but you can just format that USB to FAT32 and copy the ISO files onto it (file copy, not byte copy), and you have a bootable media.

And now that we have gone through all of the above, I can get into a rant and explain why I believe that the Linux distro maintainers, who usually are smarter than that, are actually doing some disservice to their users, even as they are trying to help them:

Almost all recent Linux distros use a MAJOR HACK called "IsoHybrid", where someone managed to figure out a way to make an ISO9660 optical image masquerade as a regular disk image, with a partition table, an MBR and everything... In other words, most Linux ISOs you find these days are abusing the ISO9660 file system to make it look like something it was never designed to look like: a dual disk and optical image.

Obviously, the goal is to create an ISO that can also be used with the dd command, even as an ISO should never be able to work that way. And I agree that, in theory, this sounds awesome, because being able to use a single image for completely different uses should be great for users, but in practice, this leads to issues that are often overlooked:

• A lot of Linux distro maintainers don't want to bother using a secondary file system that Windows can mount (e.g. they will use ext as the "secondary" file system on top of ISO9660), which means that a lot of Windows people, who are creating a bootable drive to use Linux for the first time, are super confused as to why they can no longer access the content of their flash drive. It's even worse if the "IsoHybrid" also includes an EFI System Partition (ESP) because then these users get the impression that their drive has completely shrunk in size. If you go on reddit or elsewhere, you will many posts from users who are utterly confused as to what happened to their USB media, which doesn't make for a great Linux first impression...
• As lot of Linux distro maintainers focus so much on making ISOHybrid work that they completely disregard the option of creating a UEFI bootable media by simply copying the content onto a FAT32 formatted partition, which, really, should always be the preferred method of creating UEFI bootable drives (because it's usually a lot less risky to format a partition and then copy files than it is to use the dd command). Because of this, we've seen several issues that make for a subpar user-experience with Manjaro, Ubuntu... This is actually my main point of contention with "ISOHybrid": It should not serve as an excuse to ditch established means of creating bootable media!
• GPT and "ISOHybrid" can be problematic on account that the secondary GPT table will be seen as corrupted when using dd... which actually leads to a BSOD on Windows 7 (but that's really a Windows bug rather than an ISOHybrid issue). Still, not the best experience for Windows folks creating bootable drives...
• And finally, because "ISOhybrids" are presented as if they were the most natural media in the world (which they certainly aren't), people like yourself are led to believe that every ISO image can be applied using dd, when it's the exception rather than the rule. This is very unfortunate, because it creates TONS of user confusion, with some Linux users telling people who want to create Windows bootable media that they should just be able to use dd when that most certainly will never work! Also, if you pick any Linux ISO from 10 years ago, I'm pretty confident that you'll find that almost none of them can actually be used to create a bootable media using dd because this "IsoHybrid` thing is actually a recent development.

As far as I know, Microsoft have no plans to switch to the "hack" that is ISOHybrid for their Windows ISOs, which means that you're unlikely to ever be able to use dd to create a bootable USB media from it, and therefore, if you want to create Windows bootable media from an ISO you either:

• (UEFI) Need to format a drive with a file system that Windows can boot from (NTFS, FAT32 and more recently exFAT) and extract the ISO files onto it. Now, if using NTFS or exFAT, you may have to do a little extra work as well...
• (BIOS/Legacy) Need to format a drive with a file system that Windows can boot from (NTFS or FAT32 -- exFAT will not work because Microsoft never published BIOS bootloaders for it), and then create the relevant bootloader chain, from MBR boot code to volume boot records.

It's actually not that complicated to achieve, but it does take a bit more work than a 1:1 copy from an ISO file.

Hope that answers your question.

Solution 2:

dd isn't the right tool to create a windows bootable USB. The easy way is to use woeusb.

Install:

sudo apt-get install devscripts equivs gdebi-core
cd WoeUSB
./setup-development-environment.bash
mk-build-deps
sudo gdebi woeusb-build-deps_3.3.1_all.deb
dpkg-buildpackage -uc -b
sudo gdebi ../woeusb_3.3.1_amd64.deb

Now, the package version is 3.3.1 , in case of package update ,the command ./setup-development-environment.bash will print the current version, you should replace it in the above commands.

Usage:

You can use the GUI , run woeusbgui from the terminal. Or you can use the CLI:

Unmount the USB device (important). Then run:

sudo woeusb -v --device /path/to/windows.iso /dev/sdc