(Ab)using Microsoft's symbol servers, for fun and profit

Since I find myself doing this on regular basis (Hail Ghidra!), and can never quite remember the commands.

Say you have a little Microsoft executable, such as the latest ARM64 version of usbxhci.sys, that you want to investigate using Ghidra.

Of course, one thing that can make the whole difference between hours of "Where the heck is the function call for the code I am after?" and a few of seconds of "Judging by its name, this call is the most likely candidate" is the availability of the .pdb debug symbols for the Windows executable you are analysing.

You may also know that, because of the huge corporate ecosystem they have where such information might be critical (as well as some government pressure to make it public), it so happens that Microsoft does make available a lot of the debug information that was generated during the compilation of Windows components. Now, since it can amount to a large volume of data (one can usually expect a .pdb to be 3 to 5 times larger than the resulting code) this debug information is not usually provided with Windows, unless you are running a Debug/Checked build.

But it can "easily" be retrieved from Microsoft's servers. Here's how.

First of all, you need to ensure that you have the Windows SDK or Windows Driver Kit installed. If you have Visual Studio 2019 (remember, the Community Edition of VS2019 is free) with the C++ development environment, these should already have been installed for you. But really it's up to you to sort that out and alter the paths below as needed.

With this prerequisite taken care of, you should find a commandline executable called symchk.exe somewhere in C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\. This is the utility that can connect to the Microsoft's servers to fetch the symbol files, i.e. the compilation .pdb's that Microsoft has made public.

So, let's say we have copied our ARM64 xHCI driver (USBXHCI.SYS - Why Microsoft suddenly decided to YELL ITS NAME is unknown) to some directory. All you need to do to retrieve its associated .pdb then is issue the command:

"C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\symchk.exe" /s srv*https://msdl.microsoft.com/download/symbols /ocx .\ USBXHCI.SYS

The /s flag indicates where the symbols should be retrieved from (here the Microsoft's remote server) and the /ocx flag, followed by a folder, indicates where the .pdb should be copied (here, the same directory as the one where we have our driver).

If everything goes well, the output of the command should be:

SYMCHK: FAILED files = 0
SYMCHK: PASSED + IGNORED files = 1

with the important part being that the number of PASSED files is not zero, and you should find a newly created usbxhci.pdb in your directory. Neat!

"Hello, my name is Mr Snrub"

So, what do you do with that?

Well, I did mention Ghidra, and as a comprehensive disassembly/decompiler utility, Ghidra does of course have the ability to work with debug symbols if they happen to be available (sadly, it doesn't seem to have the ability to look them up automatically like IDA, or if it does, I haven't found where this can be configured), which helps turn an obtuse FUN_1c003ac90() function name, into a much more indicative XilRegister_ReadUlong64()...

For instance, let's say you happen to have been made aware that the reason why you currently can't use the rear USB-A ports for Windows 10 on the Raspberry Pi 4 is because Broadcom/VIA (most likely Broadcom, because they've already done everyone a number with implementing a DMA controller that chokes past 3 GB on the Bcm2711) have screwed up 64-bit PCIe accesses, and they end up returning garbage in the high 32-bit DWORD unless you only ever attempt to read 64-bit QWORDs as two sequential DWORDs instead of a single QWORD.

As a result of this, you may be exceedingly interested to find out if there exists something in the function calls used by Microsoft's usbxhci.sys driver, that can set 64-bit xHCI register accesses to be enacted as two 32-bit ones.

Obviously then, if, after using the .pdb we've just retrieved above, Ghidra helpfully tells you that there does exist a function call at address 1c003ac90 called XilRegister_ReadUlong64, you are going to be exceedingly interested in having a look at that call:

undefined8 XilRegister_ReadUlong64(longlong param_1,undefined8 *param_2)
{
  undefined8 local_30 [6];
  
  local_30[0] = 0;
  if (*(char *)(*(longlong *)(param_1 + 8) + 0x219) == '\0') {
    DataSynchronizationBarrier(3,3);
    if ((*(ulonglong *)(*(longlong *)(param_1 + 8) + 0x150) & 1) == 0) {
      // 64-bit qword access
      local_30[0] = *param_2;
    } else {
      DataSynchronizationBarrier(3,3);
      // 2x32-bit dword access
      local_30[0] = CONCAT44(*(undefined4 *)((longlong)param_2 + 4),*(undefined4 *)param_2);
    }
  } else {
    Register_ReadSecureMmio(param_1,param_2,3,1,local_30);
  }
  return local_30[0];
}

NB: The comments were not added by Ghidra. Ghidra may be good at what it does, but it's not that good...

Guess what? It so happens that there exists an attribute somewhere, that Microsoft uses the bit 0 of, to decide whether 64-bit xHCI registers should be read using two 32-bit access. Awesome, this looks exactly like what we're after.

The corresponding disassembly also tells us that this if condition is ultimately encoded as a tbnz ARM64 instruction. So if we revert that logic, by using a tbz instead of tbnz, we should be able to force the failing 64-bit reads to be enacted as 2x32-bit, which may fix our xHCI driver woes...

Let's do just that then, by editing USBXHCI.SYS and changing the EA 00 00 37 sequence at address 0x03a0d0 to EA 00 00 36 (tbnz → tbz) and, for good measure, do the same for XilRegister_WriteUlong64 at address 0x005b34, by also changing 0A 01 00 37 into 0A 01 00 36 to reverse the logic. "Yes that'll do".

"I like the way Snrub thinks!"

Well, we may have patched our driver and tried to fool the system by reversing some stuff, but, as the Simpsons have long attested, it's not going to do much unless you have a trusted sidekick to help you out.

Obviously, since we broke the signature of that driver the minute we changed even a single bit, we're going to have to tell Windows to disable signature enforcement for the boot on our target ARM64 platform, which can be done by setting nointegritychecks on in the BCD. And while we're at it we may want to enable test signing as well. Now, most of the commands you'll see are for the local BCD, but that's not what we are after here, since we want to modify a USB installed version of Windows, where, in our case, the BCD is located at S:\EFI\Microsoft\Boot\BCD. So the trick to achieving that (from a command prompt running elevated) is:

bcdedit /store S:\EFI\Microsoft\Boot\BCD /set {default} testsigning on
bcdedit /store S:\EFI\Microsoft\Boot\BCD /set {default} nointegritychecks on

However, if you only do that and (after taking ownership and granting yourself full permissions so that you can replace the existing driver) copy the altered USBXHCI.SYS to Windows\System32\drivers\ you will still be greeted by an obnoxious

Recovery

Your PC/Device needs to be repaired

The operating system couldn't be loaded because a critical system driver is missing or contains errors.

File: \Windows\System32\drivers\USBXHCI.SYS
Error code: 0xc0000221

Oh noes!

The problem, which is what generates the 0xc0000221 (STATUS_IMAGE_CHECKSUM_MISMATCH) error code, is that the optional PE checksum field, used by Windows to validate critical boot executables, has not been updated after we altered USBXHCI.SYS. Therefore checksum validation fails, and this is precisely what the Windows boot process is complaining about.

Fixing this is very simple: Just download PEChecksum64.exe (e.g. from here) and issue the command:

D:\Dis\>PEChecksum64.exe USBXHCI.SYS
USBXHCI.SYS: Checksum updated from 0x0008D39B to 0x0008D19B

For good measure, you will also need to self-sign that driver, so that you can avoid Windows booting into recovery mode with an obnoxious 0xc000000f from winload.exe (though you can still proceed to full boot from there).

Now we finally have all the pieces we need.

For instance, we can replace USBXHCI.SYS on a fast USB 3.0 flash drive containing an Raspberry Pi Windows 10 ARM64 installation created using WOR (and if you happen to have the latest EEPROM flashed as well as a version of the Raspberry Pi 4 UEFI firmware that includes this patch, you can actually boot the whole thing straight from USB), and, while we are at it, remove the 1 GB RAM limit that the Pi 4 had to have when booting from USB-C port (since we're not going to use that USB controller), by issuing, from an elevated prompt:

bcdedit /store Y:\EFI\Microsoft\Boot\BCD /deletevalue {default} truncatememory

Do all of the above and, who knows, you might actually end up with a usable Windows 10 ARM64 OS, running from one of the rear panel's fast USB 3.0 ports with a whooping 3 GB of RAM, on your Raspberry Pi 4.

Now, isn't that something?

But this is just a post about using Microsoft's symbol servers.

It's not a post about running full blown Windows 10 on the Raspberry Pi 4, right?

Addendum: In case you don't want to have to go through the taking of ownership, patching, updating of the PE checksum and digitally re-signing of the file yourself, may I also interest you in winpatch?

Compiling and installing Grub2 for standalone USB boot

The goal here, is to produce the necessary set of files, to be written to an USB Flash Drive using dd (rather than using the Grub installer), so that it will boot through Grub 2.x and be able to process an existing grub.cfg that sits there.

As usual, we start from nothing. I'll also assume that you know nothing about the intricacies of Grub 2 with regards to the creation of a bootable USB, so let me start with a couple of primers:

1. For a BIOS/USB boot, Grub 2 basically works on the principle of a standard MBR (boot.img), that calls a custom second stage (core.img), which usually sits right after the MBR (sector 1, or 0x200 on the UFD) and which is a flat compressed image containing the Grub 2 kernel plus a user hand-picked set of modules (.mod).
   These modules, which get added to the base kernel, should usually limit themselves to the ones required to access the set of file systems you want Grub to be able to read a config file from and load more individual modules (some of which need to be loaded to parse the config, such as normal.mod or terminal.mod).
   As you may expect, the modules you embed with the Grub kernel and the modules you load from the target filesystem are exactly the same, so you have some choice on whether to add them to the core image or load them from the filesystem.
    
   
2. You most certainly do NOT want to use the automated Grub installer in order to boot an UFD. This is because the Grub installer is designed to try to boot the OS it is running from, rather than try to boot a random target in generic fashion. Thus, if you try to follow the myriad of quick Grub 2 guides you'll find floating around, you'll end up nowhere in terms of booting a FAT or NTFS USB Flash Drive, that should be isolated of everything else.

With the above in mind, it's time to get our hands dirty. Today, I'm going to use Linux, because my attempts to try to build the latest Grub 2 using either MinGW32 or cygwin failed miserably (crypto compilation issue for MinGW, Python issue for cygwin on top of the usual CRLF annoyances for shell scripts due to the lack of a .gitattributes). I sure wish I had the time to produce a set of fixes for Grub guys, but right now, that ain't gonna happen ⇒ Linux is is.

First step is to pick up the latest source, and, since we like living on the edge, we'll be using git rather than a release tarball:

git clone git://git.savannah.gnu.org/grub.git

Then, we bootstrap and attempt to configure for the smallest image size possible, by disabling NLS (which I had hoped would remove anything gettext but turns out not to be the case - see below).

cd grub
./autogen.sh
./configure --disable-nls
make -j2

After a few minutes, your compilation should succeed, and you should find that in the grub-core/ directory, you have a boot.img, kernel.img as well as a bunch of modules (.mod).

As explained above, boot.img is really our MBR, so that's good, but we're still missing the bunch of sectors we need to write right after that, that are meant to come from a core.img file.

The reason we don't have a core.img yet is because it is generated dynamically, and we need to tell Grub exactly what modules we want in there, as well as the disk location we want the kernel to look for additional modules and config files. To do just that, we need to use the Grub utility grub-mkimage.

Now that last part (telling grub that it should look at the USB generically and in isolation, and not give a damn about our current OS or disk setup) is what nobody on the Internet seems to have the foggiest clue about, so here goes: We'll want to tell Grub to use BIOS/MBR mode (not UEFI/GPT) and that we'll have one MBR partition on our UFD containing the boot data that's not included in boot.img/core.img and that it may need to proceed. And with BIOS setting our bootable UFD as the first disk (whatever gets booted is usually the first disk BIOS will list), we should tell Grub that our disk target is hd0. Furthermore, the first MBR partition on this drive will be identified as msdos1 (Grub calls MBR-like partitions msdos#, and GPT partitions gpt#, with the index starting at 1, rather than 0 as is the case for disks).

Thus, if we want to tell Grub that it needs to look for the first MBR partition on our bootable UFD device, we must specify (hd0,msdos1) as the root for our target.
With this being sorted, the only hard part remaining is figure out the basic modules we need, so that Grub has the ability to actually identify and read stuff on a partition that may be FAT, NTFS or exFAT. To cut a long story short, you'll need at least biosdisk and part_msdos, and then a module for each type of filesystem you want to be able to access. Hence the complete command:

cd grub-core/
../grub-mkimage -v -O i386-pc -d. -p\(hd0,msdos1\)/boot/grub biosdisk part_msdos fat ntfs exfat -o core.img

NB: If you want to know what the other options are for, just run ../grub-mkimage --help
Obviously, you could go crazy adding more file systems, but the one thing you want to pay attention is the size of core.img. That's because if you want to keep it safe and stay compatible with the largest choice of disk partitioning tools, you sure want to have core.img below 32KB - 512 bytes. The reason is there still exists a bunch of partitioning utilities out there that default to creating their first partition on the second "track" of the disk. And for most modern disks, including flash drives, a track will be exactly 64 sectors. What this all means is, if you don't want to harbour the possibility of overflowing core.img onto your partition data, you really don't want it to be larger than 32256 or 0x7E00 bytes.
OK, so now that we have core.img, it's probably a good idea to create a single partition on our UFD (May I suggest using Rufus to do just that? ;)) and format it to either FAT/FAT32, NTFS or exFAT.

Once this is done, we can flat-copy both the MBR, a.k.a. boot.img, and core.img onto those first sectors. The one thing you want to pay attention to here is, while copying core.img is no sweat, because we can just use a regular 512 byte sector size, for the MBR, you need to make sure that only the first 440 bytes of boot.img are copied, so as not to overwrite the partition data and the disk signature that also resides in the MBR and that has already been filled. So please pay close attention to the bs values below:

dd if=boot.img of=/dev/sdb bs=440 count=1
dd if=core.img of=/dev/sdb bs=512 seek=1 # seek=1 skips the first block (MBR)

Side note: Of course, instead of using plain old dd, one could have used Grub's custom grub-bios-setup like this:

../grub-bios-setup -d. -b ./boot.img -c ./core.img /dev/sdb

However, the whole point of this little post is to figure out a way to add Grub 2 support to Rufus, in which we'll have to do the copying of the img files without being able to rely on external tools. Thus I'd rather demonstrate that a dd copy works just as good as the Grub tool for this.
After having run the above, you may think that all that's left is copying a grub.cfg to /boot/grub/ onto your USB device, and watch the magic happen... but you'll be wrong.

Before you can even think about loading a grub.cfg, and at the very least, Grub MUST have loaded the following modules (which you'll find in your grub-core/ directory and that need to be copied on the target into a /boot/grub/i386-pc/ folder):

• boot.mod
• bufio.mod
• crypto.mod
• extcmd.mod
• gettext.mod
• normal.mod
• terminal.mod

As to why the heck we still need gettext.mod, when we made sure we disabled NLS, and also why we must have crypto, when most usages of Grub don't care about it, your guess is as good as mine...

Finally, to confirm that everything works, you can add echo.mod to the list above, and create a /boot/grub/grub.cfg on your target with the following:

insmod echo
set timeout=5

menuentry "test" {
    echo "hello"
}

Try it, and you should find that your Grub 2 config is executing at long last, whether your target filesystem in FAT, NTFS or exFAT, and you can now build custom bootable Grub 2 USBs on top of that. Isn't that nice?

FINAL NOTE: In case you're using this to try boot an existing Grub 2 based ISO from USB (say Aros), be mindful that, since we are using the very latest Grub code, there is a chance that the modules from the ISO and the kernel we use in core may have some incompatibility. Especially, you may run into the obnoxious:

error: symbol 'grub_isprint' not found.

What this basically means is that there is a mismatch between your Grub 2 kernel version and Grub 2 module. To fix that you will need to use kernel and modules from the same source.

Nexus 7 (2013) - The good, the bad and the ugly

Since I have been using it for a few weeks, here's my short pictureless yet picturesque review of the Nexus 7 2013.

The good

• The screen resolution of course. 1920x1200 is the minimum resolution I consider reasonable for a 7" tablet, period. Anything less is crappy DPI. It's 2013, we're not using pixelated DOS any more.
• Plays any 720p mkv you throw at it without trouble, including Hi10 (at least with the excellent MX Player). Also seems to play a fair share of (non Hi10) 1080p mkv's alright, but more about this below.
• This auto-brightness feature is great. It's probably been a feature of Android for some time, and most likely didn't originate from Android in the first place, but with the Nexus 7 being my first mobile Android device, I can only go "Of course, you'd want to use the front camera sensor to automatically adjust the screen brightness according to the environment!". 
• Likewise, the independent new message notification light, is a another "Of course!"-type of feature.
• Screen is actually readable while standing outside on a bright sunny day. Not that I care that much about using a tablet outside, but it's nice to know you should be able to do so if you need to.
• Passed a few minor hurdles (unlocking and rooting), the device is very developer-friendly. At least, it's open enough to let run your own customized kernel in a reasonable amount of time, as well as configure the OS exactly as you see fit. Bite me, Apple!
• Supports > 54 Mbps WiFi (a.k.a. 802.11n). This is likely to be useful if, unlike everybody else out there, you actually plan to keep using your Nexus in 5 years time or more. Yes, there exist consumers who don't go around and change their mobile device every other year!
• Standard USB port for charging and data. None of this proprietary connector crap.
• Supports wireless charging. Being able to rely on two options for charging your device could be a life saver in the long run, if the other option breaks down.
• Can actually store and play mkv's that are larger than 8 GB. If it isn't obvious by now, I am quite interested in HD movie playout.
  I mean, it would be a shame to have a high resolution 7" screen, and not use it as an HD player wouldn't it?
• Sleek design. This is my first tablet after all, and I do consider it as a direct "upgrade" from my last portable device, which was a PSP-1000 (bought in 2006), so I'll reserve the right to be pleased by the form factor. Oh, and I sure hope that tablet will last as long in terms of use as my old sturdy PSP.
• Automatically recognizes USB keyboards and/or mice. Though you have to use an OTG adapter, this too could come handy.
• Quad core CPU. Unless recompilation of a full Linux kernel on your device doesn't sound like a ridiculous prospect, your device is underpowered.
• Decent battery life. Could probably watch a couple movies over a transatlantic flight, and have some power left. I like that idea. Of course, the real killer would be to have enough power to play a full game of Civ 5, should Civ 5 ever been ported (or portable) to Android. Oh well, a man can dream...
• Storage can (awkwardly) be extended through the use of a cheap USB OTG adapter. Doesn't compensate for an SD card expansion, but makes the pill somewhat easier to swallow.
• The Play Store is furnished with more than enough good free apps to find something that will appeal to you when it comes to performing common tasks.
• From having used some of their hardware, I have at least some confidence that Asus do actually know a thing or two about designing devices that can last.
• Awesome pick as a first Android device. It does feel polished and mature enough to be considered as something a consumer truly deserves, i.e. a device where a sufficient amount of thought and effort seems to have gone into designing the various core functionalities.

 The bad

• Uses a micro-USB connector #1. I cannot seriously see this connector last. Just like with mini-USB (and boy have I seen mini-USB connectors fail...), there's nothing sturdy about it, and the more you use the connector, the more you feel like it's gonna break like a twig and render your device close to unusable the day you inadvertently yank a cable or an OTG extension, or insert the plug in an odd position... which is apparently what all USB plugs are really designed for.
  Having seen the allegedly more sturdy USB 3.0 micro connector rendered worthless from a cable mishap, as well as having probably performed more than a hundred plugs/unplugs in the few weeks I got the device, I can only dread the day the connector will break. A meagre consolation will be that, when that happens, you're not going to be stranded with a complete paperweight, as you'll be able to use a wireless charger. But then you can kiss goodbye to USB debugging, device reset and any form or development...
  As an aside, were they actually listening to end-users' woes, the USB committee would have devised a sturdy small factor connector, that could be inserted either way, long before Apple did. Then again, that would take letting representatives of the general public have a say in the committee's agenda, which, as we all know, is pure unadulterated heresy!
• 32 GB max internal flash max is way too small!! 32GB was already too small in 2010 for crying out loud! In 2013, 128GB should come as standard, with 256GB for high end. I can only imagine how ridiculously limited a tablet with "only" 32GB will be in 5 years time... This is all the more incomprehensible as anybody can purchase 64GB micro SD cards off Amazon for less than $50, ergo this is neither a question of technological limitation nor price.
• No SD slot for you! Google and its minions can lie as much as they want, but everybody knows there's just one reason not to have an SD card slot on a portable device, and that is to force users to get their multimedia content from the restrictive cloud. :(
• Uses a micro-USB connector #2: The lack of an SD slot above means you pretty much have to use a micro to regular USB OTG adapter, for anything that involves expanding the ridiculous amount of internal flash, which is both awkward and uncomfortable. That is unless you are using a micro SD card with meenova Mini MicroSD Card Reader (which, while a bit pricey, is well designed and well worth the satisfaction of sending a big fat "screw you and your stupid restrictions" message to Google)
• I've seen a bunch of artefacts (banding in gradients) when displaying pictures, which I haven't seen when the same picture is shown on a regular monitor. That makes me think that the screen is not actually able to display 16M nuances of colours. I have waited 8 years to upgrade from the lousy screen that Sony decided to ship with original PSP (Oh, the remanence!), and this is this only as far we've been able to make the technology evolve? That is a (small) letdown... 
• Uses USB 2.0 rather than USB 3.0. I don't care if the read and write speed of your flash can't reach USB 3.0 speeds, the RAM can, and one might want to use the device in the future for applications connecting to an USB device where high bandwidth is a must (software oscilloscope, software radio).
• The default picture and movie viewers from Android are severely lacking functionality. Thankfully, the free QPic and MX Player can be installed, that do a much better job. Still, having a default picture viewer that you can't set to display pictures in alphabetical order is unacceptable.
• Can't play Hi10 (a.k.a. 10 bits) 1080p mkv's (unless you consider a 2 FPS rate as acceptable). I don't think even a fully optimized video player will ever be able to so on this device, because the GPU and CPU are unlikely to be that powerful. Sure hope the future will prove me wrong though...
• Video playout of some 8 bits 1080p mkv can be very jerky too, if MX Player decides that the HW+ decoder cannot apply, and, besides Hi10, I have no idea what decides whether HW+ can be used for decoding or not. Thus, depending on how the video file was produced, 1080p playout can be a bit of a hit and miss...
• Touch screens suck. Having to wipe finger smudges all the time, and wonder where the OS will have decided the tip of your finger went is annoying. Oh and don't even get me started about trying to play arcade games using touch controls. At least for gaming, it shouldn't be that hard to add two 4+1 positions knobs (think microswitch joystick) in a very shallow depressed speaker-like shape, at the rear of the device, to at least allow some acceptable [game|arrow-keys]-like input.
• As with any tablet, there's no "good way" to get a firm yet relaxed grip when using it.
• When you try to raise the volume past a completely arbitrary point when using headphones, you get a MOST ANNOYING POPUP about listening at high volume for a long time being potentially damaging, that you have to accept EVERY FRIGGING TIME. The problem is, if you are using a high quality headphone with relatively large speakers, the volume at which this message appears is way too low! I think I'll have to figure out a way to disable this little sucker, because this chronic nagging is really degrading the whole media player experience.
• No pouch? My old PSP came with a pouch! A portable device whose screen isn't expected to survive a 1.5m drop should come with a pouch for transport and (minimal) protection. What's more, if it has a touch screen, it should also come with a microfibre cloth, that's at least as big as the width of the device. This may sound like a trifle, but it shows that you might actually value your customers.

The Ugly

• The Android community.
  Trying to find information that is actually of interest for development is such a massive ordeal! You have to go through page upon page of ill-informed posts, as well as scores of people jumping on the Android bandwagon to try to make a quick adversing buck, before reaching anything of actual substance... And don't get me started on Google's "Welp, that 5 line overview of the development process'll do". I'd like to remind Google that, as opposed to most Open Source projects, they do have paid employees that could take care of providing comprehensive documentation...

I guess that will do. This should be different enough from any other review of the Nexus 7 2013 you've seen. And I know some who have made a small jump from a 2012 Nexus 7 to a 2013 Nexus 7, and have been all ecstatic in their review about it, whereas here I am, having jumped leaps and bounds from an outdated portable device, yet with loads to complain about. Still, by my standards, this is a fairly positive review, and I would recommend the device (even over the new more powerful Amazon Kindle for instance), especially as I can't really fault Asus and Google for some items I have a gripe with, such as the USB committee being led by a bunch of monkeys...

Rooting your Nexus 7 (2013) for the Android virgin...

...but Linux experienced.

Having recently purchased a Nexus 7 32GB (2013 model) as my very first Android mobile device, I am finding that the information on how to gain complete control is both all over the place and supposes you're already familiar with the whole rooting process, which I'm not. Therefore, I'm going to write yet another step by step guide on what you need to do, to be able to remotely ssh into it and run a shell as root, which, IMO, is the one stage where we can pretend that we are actually running a GNU/Linux system under the hood and get things done.

Incidentally, if you don't care about commandline Linux, this process will also grant you full control and let you do whatever you please with the device.

Being a complete Android noob, I will also make sure not to skim over all the "Oh, but you're supposed to know that already!" details, so that you will hopefully be as successful as I was, in rooting your device. On the other hand, I will assume that you are somewhat familiar with Linux. But don't let that worry you as the Linux specific items will come at the very end, at which stage your device will already be fully rooted.

Finally, since it's somewhat convenient, I will be using Windows as the platform we'll use to communicate with the Nexus 7 during this process, and I'll assume that you are familiar enough with it to find your way around the Device Manager, manual driver installation and the Windows command line.

Why *I* want to root my device

I'm not going to venture a guess as to what your reasons are for rooting your device. But one of the first reasons I have, in a very long list, is to be able to edit the device's /etc/hosts and get rid of most of the ads and unwanted trackers, which every other device and computer that I have doesn't have to put up with, through a skilful use of the ever so awesome WinHelp 2002 host file. And yeah, I know there are other ways to block ads, but the /etc/hosts method is still the simplest, most universal and effective way, since it'll block ads for all apps running on your system, rather than the ones in a single browser application.

Now, the ironic part is: the reason I purchased a Nexus 7 in the first place is through revenue that I got from placing ads on a successful little web site (Hurray, for unexpected disposable income!).
Am I really going to bite the hand that feeds me? Watch me!

Besides eliminating ads, there are of course plenty of other reasons to want to root your device, such as recompiling and running your own custom kernel (we'll come to that in a next post), but we might as well start small, by improving our mobile surfing experience.

What the other guides omit to tell you...

For the most part, I am going to follow the video guides at xda-developers and most specifically, the third and second to last from this link. Oh, and if you want a less verbose, yet noob friendly and up to the point guide, I can only recommend the "How to build Cyanogen for Nexus 7 2013" one, as it has a very easy to follow section on rooting. Still, part of these guides suppose some familiarity with the process and omit some important details that you may be interested in knowing beforehand, such as:

The internal name of the Nexus 7 2013 platform is 'flo'

This might be relevant if you are flashing an Android image file that came from a website that isn't dedicated exclusively to the Nexus 7 2013. If you do that, you may want to check that 'flo' is mentioned somewhere in what you're looking at. Alternatively, 'msm', which is the development branch for the 'flo' platform might be a good indicator. Remembering these two can be a good way of avoiding looking at data that is only relevant for the "old" Nexus 7, or other incompatible Android hardware.

You are going to lose all your data!

More specifically, as soon as you unlock the device, which is pretty much the first step invoked in rooting, your device will be completely reset to factory settings.
In the notice they display when you do that (see screenshots below) Google say that this is to prevent someone who got unwarranted access to your phone from accessing your personal data (which is probably only a half truth, but hey...).
My advice then is: if your device is new, you want to root it as soon as possible, so that you don't have to go through the pain of restoring your apps, settings and personal data.

Then again, if you have let some time pass and installed a few things, provided you did set the device to use your google account for backup, a lot of it will be restored automatically as you re-indentify yourself after unlock. Especially, all the apps you have installed, whether free or not, will be re-downloaded and re-installed. Also, if you're lucky, some of the customization and personal data files associated with them may be restored... though you shouldn't really count on that and take your precautions.

But really, the less customized your device is before you root it, the better. Before attempting this procedure then, you probably want to back up as much as you can... though of course, it takes a rooted device to be able to perform a comprehensive backup of your data! As if we needed more reasons to want to root our device early.

As I assume that you are as new to the process as I was, here is a quick overview of the steps we are going to follow:

1. Enable Developer access on your device - this will allow to connect your Nexus 7, through USB, to the Android debug console as well as the boot/flash utility. We'll need this to unlock the device and flash the custom (non Google) recovery image, that'll help with installing root access
2. Install the Developer tools on your PC, so that you can communicate with your Nexus in developer mode.
3. Unlock the stock bootloader, so that it will let you flash a non-stock recovery image. This is also the process that resets your device and data to factory settings.
4. Install (flash) a custom recovery image - This will allow the installation of the binary that will supervise the granting of root access to applications (SuperSU).
5. Boot into that recovery image and install (flash) the SuperSU system binaries and deamon.
6. Reboot the device into standard operation mode, and install the SuperSU app. (SuperSU comes in two parts: A regular "client" app, that can be downloaded from the App Store, and a system executable part that needs to be installed separately into the flash filesystem).
7. Install Busybox
8. Install an SSH server
9. Customize our GNU/Linux system, including /etc/hosts.
   

Enabling Developer access

So you've backed up whatever you could and accepted the idea that your Nexus will be restored to the clean slate state you found it right after you purchased it. Good.

The first thing we need then is enable developer access, which is disabled by default. To do that, you need to navigate to "Settings" → "About Tablet" (at the very end) and tap the "Build number" section 7 times. Why 7 times? Who knows...

Once you do that, you will get a small prompt indicating "You are now a developer" and after that has occurred, if you go back into the Settings menu, you will see a new "{ } Developers options" category. There, you will find a bunch of additional parameters that you can modify, that may come handy if you do development, but the only one we need right now, under the "DEBUGGING" section, is "USB debugging". So please go ahead and tap that one.

As soon as you do that, you will receive a warning about "USB debugging being intended for development purposes only", which you should accept. This should lead to the following checkmark being displayed:

Installing the USB driver and developers tools

Unfortunately, even as it is WinUSB based, the current version of Android is not WCID enabled (but I'm working on that!), so that means you need to manually install some drivers, to be able to communicate with the device in debug mode.

You should therefore head to http://developer.android.com/sdk/win-usb.html click the "Download Google USB Driver" button, then pretend you accept whatever agreement they want you to agree by not reading it (so that you can feign complete ignorance of the terms if you break them, which I hear is how EULAs work these days) and unzip the files into an usb_driver\ directory somewhere.

Now, you should connect your device to an USB port on your Windows machine. Windows will try, and fail, to locate a driver for the new debugging USB interface, at which stage you should select the "Nexus 7" unrecognised device in Device Manager, select install driver, and point to the usb_driver\ directory you previously extracted.

Once you have done that, you should see something very similar to this:

All that is left, with regards to the developers tool installation, is to go to http://developer.android.com/sdk/index.html and Download the SDK ADT Bundle for Windows (about 450 MB in size). Depending on whether you picked the 32 or 64 bit version, you should end up with a zip file such as adt-bundle-windows-x86_64-20130917.zip which you can then extract somewhere like D:\Android\adt-bundle-windows-x86_64-20130917\.

Unlocking the bootloader

The only thing we are really interested in package above are the adb.exe (Android Debug Bridge - The application that communicates with the Android OS in debug mode) and fastboot.exe (The application that communicates with the bootloader) and that are located in the sdk\platform-tools\ directory. That's right; we downloaded more than 450 MB worth of junk just to access a couple of executables that don't even occupy 1 MB worth of space... talk about bandwidth wastage! (if you are concerned about this, you may want to look for smaller unofficial packages containing only adb and fastboot, or recompiling these from git)

Next, you should open a command prompt into your sdk\platform-tools\ directory and issue:

adb start-server

The command window will display the following:

D:\Android\adt-bundle-windows-x86_64-20130917\sdk\platform-tools>adb start-server
* daemon not running. starting it now on port 5037 *
* daemon started successfully *

D:\Android\adt-bundle-windows-x86_64-20130917\sdk\platform-tools> 

And you will also receive a request similar to this one on your device, which you should accept:

Once you have accepted the request, you will get a short message on the top status bar, indicating "USB debugging connected". Then a persistent USB debug icon, similar to the one displayed on the left, will appear in the top left corner.

The next command we want to enter, in our command prompt, is:

adb reboot bootloader

Your Nexus will now reboot in bootloader mode and once that is done, you should see a screen similar to this one, indicating "LOCK STATE - locked":

Note 1: If needed, you can use Volume Up/Volume Down to select the option highlighted at the top, and press  Power to execute that option.

Note 2: You can also enter the bootloader without using the adb command, by powering your table off, and then pressing Volume Down while powering it up.

Of course, we're going to change that LOCK STATE. To do that, and since we are in bootloader mode, we can use the fastboot command:

fastboot oem unlock

You will get a warning similar to this:

Use Volume Up/Down to select "Yes" and press the Power button. It should return to bootloader menu, but this time you should see "LOCK STATE - unlocked":

Now, you can now reboot your device by issuing the following:

fastboot reboot

Note: Because you unlocked the bootloader, the device will now reset to its factory settings. This is the part where you lose ALL YOUR DATA AND CONFIGURATION.

Installing the TWRP custom recovery image

Make sure your device boots properly, and recreate your account.

You probably also want to take some time to let the backup process reinstall your apps, as well as manually download and reconfigure whatever personal settings and files you did backup, before you unlocked the bootloader. The rest of our operations here should not affect this data.

While your Nexus is being restored, you also want to download the following 2 files:

• The latest TeamWin Open Recovery image from http://techerrata.com/browse/twrp2/flo, which you should save into your sdk\platform-tools folder\
• The latest SuperSU zip, from http://download.chainfire.eu/supersu, which you should save, on your Nexus device, in the "Downloads" folder, without extracting it (i.e. as a zip file). This file will be named something like "UPDATE-SuperSU-vX.YZ.zip".

For safety, once you have restored apps, settings and files, you should, issue a complete poweroff. Then boot it back again, confirm that you have USB debugging running (The small Android icon at the top left corner) and, in the command prompt Window issue:

adb reboot bootloader

Now that we're back in the bootloader, we are going to replace the stock recovery image, by flashing the TWRP we just downloaded over it. Provided the TWRP file you downloaded above is named "recovery openrecovery-twrp-2.6.3.0-flo.img", the command you will need to issue:

fastboot flash recovery openrecovery-twrp-2.6.3.0-flo.img

Now you need to reboot in recovery mode. You can do that by using Volume Up/Down until "Recovery Mode" is displayed at the top, and then pressing Power.

You should be greeted with the following screens:

Once you see the above, tap "Install" and select the "UPDATE-SuperSU-vX.YZ.zip" you should have in your "Download" folder:

You should then see the following:

At this stage, you should proceed to swipe, to install the files contained in the zip into the OS partition. What this process actually does is copy two executables su and chattr (Change Attribute) in your /system/xbin/ directory, as well as install a daemon that, will be launched at boottime.

You can then proceed to reboot the device, by selecting "Reboot" on the initial screen and then "System".

Installing BusyBox

Now that we have the system part of SuperSU installed, we can go to the Play Store and install the SuperSU client app, so just go ahead and do that.

Congratulations! At this stage, you have supposedly completed the hardest part, and your device is effectively rooted.

First thing you probably want to do then, is confirm that this is actually the case by installing "Root Checker" from the Play Store. If you launch it and click on the "Verify Root Access" you should get a prompt such as this one:

This is a SuperSU prompt, one that you should become exceedingly familiar with as you install more apps that request root access. When you see it, you should tap on "Grant", and after that, Root Checker should confirm that you have root access indeed.

Alrighty, moving on to our next target. We're of course not going to be sitting around installing rooted apps all day. What we want is an actual Linux shell, that we can access remotely. Thus, the first application we want to install is Busybox, that provides a basic set of base Linux utilities, as well as a basic shell.

On to the Play Store it is. If you search for BusyBox, you'll find a bunch of 'em. The one I installed was the free version from Stephen Ericson (Stericson) that will probably be the first result you'll see.

After it's installed, you should run the app, let it scan your system for a bit, and then go for the Smart Install. In the process, you will be greeted by another "Superuser request" prompt from SuperSU, which you should of course grant.

Once we've done that, and to confirm that BusyBox is properly installed, we're can to go back to our Windows prompt and issue:

adb shell

This opens an interactive shell on our Linux box (with a terminal that sucks, but we're not going to worry about that). You can then confirm that BusyBox was properly installed by issuing the command busybox. The result you get should be similar to this:

shell@flo:/ $ busybox
busybox
BusyBox v1.21.1-Stericson (2013-07-08 15:58:11 BST) multi-call binary.
BusyBox is copyrighted by many authors between 1998-2012.
Licensed under GPLv2. See source distribution for detailed
copyright notices.

Usage: busybox [function [arguments]...]
   or: busybox --list[-full]
   or: busybox --install [-s] [DIR]
   or: function [arguments]...

        BusyBox is a multi-call binary that combines many common Unix
        utilities into a single executable.  Most people will create a
        link to busybox for each function they wish to use and BusyBox
        will act like whatever it was invoked as.

Currently defined functions:
        [, [[, ash, awk, base64, basename, blkid, bunzip2, bzcat, bzip2, cal,
        cat, chat, chattr, chgrp, chmod, chown, chroot, chrt, cksum, clear,
        comm, cp, crond, crontab, cut, date, dd, depmod, devmem, df, diff,
        dirname, dmesg, dnsd, dnsdomainname, dos2unix, du, echo, egrep, env,
        ether-wake, expand, expr, fakeidentd, fdflush, fdformat, fdisk, fgrep,
        find, fold, free, freeramdisk, fsck, fsync, ftpd, ftpget, ftpput,
        fuser, getopt, grep, groups, gunzip, gzip, hd, head, hexdump, hostid,
        hostname, httpd, hwclock, id, ifconfig, ifenslave, inetd, inotifyd,
        insmod, install, ionice, iostat, ip, ipaddr, ipcalc, iplink, iproute,
        iprule, iptunnel, kill, killall, killall5, less, ln, logname, losetup,
        ls, lsattr, lsmod, lsof, lsusb, lzop, lzopcat, md5sum, microcom, mkdir,
        mkdosfs, mke2fs, mkfifo, mkfs.ext2, mkfs.vfat, mknod, mkswap, modinfo,
        modprobe, more, mount, mountpoint, mt, mv, nameif, nanddump, nandwrite,
        nc, netstat, nice, nmeter, nslookup, ntpd, od, patch, pgrep, pidof,
        ping, ping6, pkill, pmap, powertop, printenv, printf, ps, pscan, pwd,
        rdate, rdev, readahead, readlink, realpath, renice, reset, rev, rfkill,
        rm, rmdir, rmmod, route, run-parts, script, scriptreplay, sed, seq,
        setkeycodes, setlogcons, setsid, sha1sum, sha256sum, sha3sum,
        sha512sum, showkey, sleep, smemcap, sort, split, start-stop-daemon,
        stat, strings, stty, sum, swapoff, swapon, sync, sysctl, tac, tail,
        tar, tee, telnet, telnetd, test, tftp, tftpd, time, timeout, top,
        touch, tr, traceroute, traceroute6, tty, ttysize, tunctl, umount,
        uname, uncompress, unexpand, uniq, unix2dos, unlzop, unzip, uptime,
        usleep, uudecode, uuencode, vconfig, vi, watch, wc, wget, which, who,
        whoami, whois, xargs, zcat

shell@flo:/ $

Looks good enough. Now you can type exit to exit the shell and go back to the Windows command prompt.

And since we're done with the Windows command, you may want to issue:

adb kill-server

So that you can later disconnect your device without Windows complaining about it.

Installing an SSH server

At last, we've come to the final piece of our puzzle! Surprisingly, if you visit the Play Store, you'll find that there aren't that many SSH servers on offer.

The one I picked then is "Rooted SSH/SFTP Daemon", which looks a bit basic but does its job very well.

Once you launch it, tap the gear icon in the middle of the screen, and change at least the Login ID (I used 'root') and Password. You probably also want to enable "Wifi Lock" so that the SSH server will keep running in the background as you switch apps.

Now you want to press the large ON/OFF button so that it is highlighted red and so that our server is running. You should also get a small grey round ON/OFF icon in the top left area (see previous or next screenshot). Oh, and you'll probably have gotten another SuperSU grant request at this stage.

"Well, I seem to have an SSH server running alright, you say, but how do I find the name or IP address I should use to connect to it?"

The simplest method, without having to install another app or check your Wifi router, is to swipe down the Wifi icon in the top right corner and tap the Wifi properties.
Oh yeah, did you know that, if you swipe down any of the small status icons that appear at the top, you can quickly access the application and settings behind them? That's a good trick to know, and I sure didn't.

In this case, you'll be able to tap your current Wifi connection and get the IP address you need to use for SSH:

Give a go to that IP address in an ssh client, such as putty, and lo and behold, you should be connected to your Nexus 2013 Android device under a nice shiny shell. That's more like it!

Customizing our GNU/Linux Android system

Let's now push things to their logical conclusion, by performing 2 last items:

1. As you may have noticed if you scan your network, and if you let the DHCP client provide its desired hostname, your Nexus 7 will be identified as 'android_12345678' or something. Ugh, let's change that!
2. Edit that /etc/hosts to filter ads, as we said we'd do.

With regards to item 1, a quick search tells us that the file we want to edit is /system/build.prop, where you want to add a net.hostname=my_name line. So you go ahead and try that, only to find that the file is read-only?!?
What gives? Weren't we supposed to be root here?

Well, that's Android security for ya (which may or may not be that bad a thing). By default, only the directories under /sdcard (such as "Download" and stuff) are read/write. The rest of the system is read-only.

Fixing this is easy though. If you want to modify system files then, you just need to issue the following to be granted full read/write access:

mount -o rw,remount -t rootfs /system

Much better!

Now you can edit your /system/build.prop to your hearts content and change the DHCP hostname.

Let's finish the job by going ads busting then. Still from the ssh shell:

cd /sdcard/Download
wget http://winhelp2002.mvps.org/hosts.zip
unzip hosts.zip
mv HOSTS hosts
dos2unix hosts
cp hosts /etc/

The last line will overwrite the existing /etc/hosts, which is fine since it only contains one entry to alias 127.0.0.1 to localhost (which is also present in the new file). Oh, and the dos2unix is necessary as the original HOSTS file was saved in DOS mode.

Do that, try to surf, and watch in awe as the ads pollution has now gone away...

I think my work here is done...

Crafting an MBR from scratch

If you follow this site, you'll remember that we previoulsy crafted a BIOS from scratch. Of course we may as well follow that up with writing an MBR while we're at it!

This time, the problem that was put to us was as follows:

As part of XP/2003 installation support in Rufus, which, if needed to be reminded, is your friendly bootable USB stick creation tool, we thought it'd be nice if, rather than having to fiddle with boot.ini options like other XP ISO → USB apps do for the second part of the XP setup process, we did something similar to what the original optical installation medium provides, with a "Press any key to boot from CD/DVD..." prompt.

More technically, the issue is that XP was not exactly designed by Microsoft to be installable from and USB drive. Therefore Windows expects to see the BIOS disk ID of the HDD it boots from, during the later stage of the installation process, as the first bootable device = 0x80, whereas 0x80 is the disk ID the BIOS assigns to the USB whenever it boots from it.
Thus, scripted methods of installing XP from USB would install an ntldr + boot.ini on the USB and prompt the user to select the second drive (first bootable HDD) to continue the process. Else the other workaround is to unplug the USB drive during reboot after the first part of the installation process is complete, and plug it back later, as Windows still needs to read files from it. Since these methods deviate from what users would see from a regular installation from CD/DVD, we tried to see if we couldn't come up with something better.

Our solution, then, is to craft an MBR that does the following:

1. If a bootable HDD is detected as second BIOS bootable device (0x81), the MBR prompts the user whether they want to boot from USB and, if no input is given, will fall back to booting from the HDD (0x81) instead of USB (0x80)
2. According to the bootable disk ID provided in the USB partition table, the MBR will swap the 0x80 device with the ID provided.
   This means that for instance, if the first partition on the USB drive has disk ID 0x81, then the USB disk is remapped to this ID, whereas the original 0x81, which would typically be the first bootable HDD, is remapped as 0x80 (first bootable device). Because this approach falls between swap and remapping, we call it masquerading, as each bootable drive is now being masqueraded as a different one.

Once the above is properly set, then the setup process on the HD can be led to think it always boots from 0x80, even if it was the USB that actually booted the system, and operate as if that was the case, leaving the installation process as close as possible to what users would experience when installing from CD/DVD.

For a more technical breakdown, of our process is as follows, knowing that the BIOS would have copied over the 512 bytes MBR at address 0x00007C00 when we start to run it.

1. Because later stages need to copy boot records at address 0x00007C00, the first thing we do is move our 512 bytes code out of the way, by allocating 1 KB of RAM, duplicating our code there and then jumping to it.
2. With our code now safely out of the way, we attempt to read the MBR of the second bootable device (0x81, as 0x80 would be the USB) into the 0x00007C00 address we just moved out from using INT_13h (disk), with either function 02h or 42h depending on the extensions found (Some BIOSes, such as older DELL, can only use 42h, so we must handle both 02h and 42h). If the read is unsuccessful, we just boot the USB.
3. If the read is successful, we check the partition entries of the HDD MBR we just read, to see if there exists one that is bootable/active. If none is found, we give up and boot the USB.
4. If an active partition is found,  we prompt the user to hit a key if they want to boot from USB by installing an override for INT_08h (timer) to provide us with both a timeout and the ability to print a dot every second.
5. If the user presses a key, we install an override for INT_13h that does the following:
   - masquerades boot device 0x81 (first bootable HDD, second bootable device after the USB) as 0x80 (first bootable device)
   - masquerades the USB boot device (0x80) with the disk ID provided for its first partition (typically 0x81, but could be higher)
6. We then handle the rest of the boot process to the relevant boot record (after removing the INT_08h override if needed). If the HDD is booted, we simply jump to the MBR we read at address 0x00007C00. If booting from USB, we read the partition boot record (eg. NTFS boot record) into address 0x00007C00 and jump there.

All of the above (and more!) is done in 440 bytes and with 2 bytes to spare (or 9 if you count the ones spent on convenience instructions). Not bad...

On a side note, the process of overriding INT_13h is used by various MBR viruses (eg. Michelangelo), and we actually had to take some measures to prevent anti virus applications from detecting our MBR as one. Our override of INT_08h for "Please press any key to boot from USB..." is also very close to what Microsoft does in the bootfix.bin it provides on its XP/2003 installation media.

The x86 assembly source of the MBR can be accessed here. If you're interested in writing your own MBR, feel free to have a look at it. Or, if you just want to see the MBR in action, feel free to download the latest version of Rufus, and use it to install Windows XP or Windows 2003 from USB.

Rufus - The bootable USB Formatting Utility

It is my very great pleasure to introduce Rufus (direct downloads here), my own GPL'd version of a bootable USB formatting utility (DOS + ISO).

If you have been using the old HP Utility to create DOS bootable USB, you can throw that old thing away! In a small executable, and with no requirement for an installer, Rufus offers you a much better and up to date interface, with better features, and a DOS creation that doesn't rely on external files. It can also create bootable USB from ISO images. Plus you will find welcome additional features, such as the ability to check your USB stick for bad blocks. Best of all, and as you have come to expect from this site, it is 100% Free Software.

Please make sure you check the official Rufus page, and stop looking further when you need a DOS bootable USB stick. Of course, Rufus is compatible with all versions of Windows starting with Windows XP and lets you use the always awesome FreeDOS alongside the rather old and not up to date Windows Millenium DOS.

Why are you doing this?

Well, the truth of the matter is that, after having used the HPUSBFW utility for some time, it turns out that I really can't stand proprietary software utilities (as well as Windows' glaring shortcomings), so I decided to create my own Open Source version of an equivalent tool. Also, the fact that the many people, who have taken a stab at creating their own DOS bootable USB formatting utility, decided to go closed source doesn't really help. Simple utilities should only be Open Source, period.

Some interesting technical details (or yet another annoying technical rant)

You'd think there wouldn't be much to formatting an USB flash drive for DOS on Windows, but you would be wrong. As I already explained, there's some reason why Windows doesn't do it natively. Also, you may be surprised to hear that Windows doesn't actually provide a public API to format a drive, and instead you have to hijack an undocumented one called FormatEx, and which can be found in fmifs.dll. Then you will find that FormatEx kind of destroys your partition table when you want to use LBA, so you need to fix it manually. There's also this whole business of allowed cluster sizes. And then there's all the usual traps, such as having the partition boot records needing to be patched on XP, because unlike Vista or later, it equates an USB Flash drive to a floppy, as well as Windows' somewhat mysterious handling of Physical Drives vs. Logical Volumes. One will let you access the MBR and the other one the Partition Boot Record, yet, you still need to hold a lock to the latter to be able to access its underlying sectors with the former. Straightforward, it is not.

If you are so inclined, you'll find Rufus' FormatThread() in format.c as a good starting point. Oh, and would be quite ungrateful if I didn't acknowledge other OSS projects, such as ms-sys (boot records handling) or e2fsprogs (bad blocks check) for providing some of the building blocks used by the utility, as well as the talented designers from PC-Unleashed for the Rufus icon.

Finally, for those interested, the acronym stands for "The Reliable USB Formatting Utility, with Source".

Enjoy!

IMPORTANT NOTE: If you want support, please use the Github issue tracker. Any requests for support in the comments will be left unanswered.

Why Windows doesn't let you create an USB MS-DOS startup disk

Today, I'll provide an answer to the following question, which, if you're like me, must have irked you some when formatting an USB flash drive:

Yes why? Why is is that Windows has no problems creating an MS-DOS startup disk for the now useless floppy technology, but does not allow you to do so for the ubiquitous USB memory stick one?
It's not really like this is an impossible feat on Windows, as there exist quite a few tools that allow you to create one already (and I have now added my MUCH BETTER ONE to the mix). So why doesn't Microsoft do it?

If it ain't broke, don't fix it... But don't upgrade it either.

Ah, conservatism. I'm not talking about the political kind here, but rather its no less sinister computer equivalent: "Hey, we worked hard enough to get there. We don't see why we should take the risk of changing existing code, just so newcomers can reap the benefits."

You see, the way Windows creates an MS-DOS startup disk is not by doing anything smart, but by doing something lazy —which some also say is the second smartest thing to do after "smart"— such as including a 1:1 image of a 1.4MB floppy disk in a DLL (diskcopy.dll) and just splashing it wholesale onto a floppy when you request a startup disk.
Oh and yes, that's L-A-Z-Y-lazy since, if you actually have a look at the image file, you'll see that Microsoft didn't even spend much time creating a well thought-out one, but instead picked up the first Windows Millenium Edition (Me) startup disk they had lying around and just proceeded to "delete" stuff until it looked OK enough. The proof of it is that the image still contains much of the "deleted" content from the original disk and for a good read on how exactly the Windows Me Startup disk was "emasculated", you are very much invited to read Daniel Sedory's entry on the subject.

Of course, this whole technique of imaging a whole floppy does fall apart as soon as you're trying to create an unknown size USB bootdisk and this, my little children, is why Windows won't let you create DOS bootable USB flash drives to this day. End of story. Good night.

But wait, there's more!

Alright, alright, I hear you: that was a pretty underwhelming bedtime story. So of course there's more. And to answer your follow up question on why Microsoft didn't just compensate for the issue above by dropping the floppy image from diskcopy.dll and simply include the files themselves, we'll press on with a little bit of history.

As stated previously, the "DOS" files included in the image file that Windows uses when creating an "MS-DOS" startup disk actually come from a Windows Me boot disk, which, amongst its many disappointments, is also well known for crippling the DOS version it came with. As Wikipedia indicates, "One of the most publicized changes from Windows 98 is that Windows Me does not include real mode MS-DOS". Furthermore, departing from any MS-DOS that came prior, "the Autoexec.bat and Config.sys files are used only to set global environment variables". Thus DOS 8.0, which is the one included with Windows Millenium, is indeed seen by many as crippled, whereas DOS 7.1, a.k.a. the one from Windows 98 SE, is often labelled the "last good DOS".

But much more relevant to the USB boot issue is that the Windows Me DOS is not actually able to produce a command prompt on anything else but a floppy media. That is to say, even if you were to create all the required boot records on an USB drive, and then copy the unmodified COMMAND.COM and IO.SYS from the diskcopy.dll floppy image, your drive will never boot. You can try this for yourself, if you have an existing bootable stick, as 7-zip will happily extract the files from diskcopy.dll if you want to copy them over.

That still doesn't explain anything...

"OK fine," you say, "Windows Me is unusable but Microsoft should have the rights to their own DOS stuff anyway, so why don't they either just use the DOS files from Windows 98, or produce an uncrippled new version of the WinMe DOS, and go with that?". That's actually a very valid question, which people have only been able to speculate onto this day. Technically and most likely legally, there doesn't exist a reason why Microsoft shouldn't be able to provide a proper set of DOS files to boot an USB stick with. Furthermore, and this is the trick people use when they want to legally provide MS-DOS files for USB boot on Windows, uncrippling the WinMe DOS files from the diskcopy.dll to get them working on USB is actually trivial and documented here. For the record, even the well known HPUSBFW tool, from HP will patch the COMMAND.COM and IO.SYS files according to the above, when provided with WinMe DOS files. Yet, Microsoft went as far as explicitly discouraging people from doing so...

Thus, the moral of our story is that, while there is no real technical limitation to it, there is no satisfying explanation as to why Microsoft decided to inconvenience millions of their users by not allowing to create a DOS bootable USB drive. However, if you keep watching this blog, you will soon be provided with a 100% Open Source tool that does just that (and more!), so stay tuned.

Bonus

By the way, if you get your hands on an older version of the HPUSBFW tool --the one that is about 400 KB in size, you should be advised that it contains, as a resource, a self extracting UPX compressed executable that will extract the Windows 98 DOS files. The utility itself is designed to only enable extraction of these files if specific HP USB hardware (VID:PID 03F0:0023) is detected, however that can easily be patched to enable any device (F0 03 75 15 → F0 03 90 90 & 23 00 75 0C → 23 90 90 0C) and one could use HPUSBFW as a standalone USB DOS boot utility.

On Windows 8 and WCID devices

With the 1.2.0 release of libwdi, which, for those who don't know, is a library aimed at making the installation of Windows USB drivers as simple and as non intrusive for end-users as possible, one of the most exciting features brought forward is WCID device driver support.

WCID, which was brought to the limelight with Windows 8, and which stands for Windows Compatible ID, is a set of additional USB Descriptors one can add to a device firmware, to make the OS automatically identify which type of driver should be used, and potentially automatically install it. In other words, what WCID does is bring the Plug-and-Play functionality that exists for HID or Mass Storage, to any USB device.

Thus, as upcoming Windows 8 comes with a WinUSB WCID driver, if you currently produce USB devices that require driver installation, you should seriously consider converting them to WCID as it'll make life a lot easier for everybody.

This is also good news for libusb-1.0 users, as libusb-1.0 relies on WinUSB for generic access on Windows, which means that you can actually produce libusb-1.0 applications for Windows 8, that do not require you or your users to take care of any driver installation. And for earlier versions of Windows, up to Windows XP SP2, libwdi and its friendly installer application Zadig can make it very easy to proceed to the one-off installation of the WinUSB (or libusb-win32 or libusbK) WCID driver, to make those platforms behave in the same way as Windows 8.

More information:

• WCID Devices
• Microsoft WinUSB Device

Which controllers support USB 3.0 Debug?

As defined per section 7.6 of the xHCI specifications?

Unfortunately, none of the current ones from Renesas, VIA Labs or Fresco Logic do, and neither of these manufacturers are planning to add the capability on their existing controllers, through a firmware update for instance... Instead, they only have plans to provide it on their next generation controllers, which won't be available for some time (yes, I am aware that the new Renesas chips have been announced since 2011.03, but I have yet to see a PCI-E expansion card with one of those -- If you actually managed to get your hands on one, I'd like to hear from you!  
UPDATE 2011.08.29: Apparently those guys did... but as a world exclusive, since these cards won't be available till later on this year. Darn!).
UPDATE 2012.01.10: At long friggin' last, uPD720201 based PCIE cards, with debug support, are starting to become available, one such being the Buffalo IFC-PCIE4U3S. Only seems to be available in Asia and Australia at the moment however, but even then, orders may not be satisfied before February because of lack of availability...

So this means, if you have a NEC/Renesas uPD720200/uPD720200A, a VIA Labs VL800/801 or a Fresco Logic FL1000/FL1009 based USB 3.0 controller, you're out of luck with regards to USB 3.0 debug.
This doesn't bode too well for USB 3.0 as replacement for legacy RS232 ports...
Oh and the other disappointing part of the current crop of PCIE-USB3 controllers is they don't support USB-3.0 boot either. Quite the letdown...

Hint: If you want to display the Extended Capabilities of your controller, you can do so by modifying the xhci_setup_port_arrays() of drivers/usb/host/xhci-mem.c in () on a recent Linux kernel, as it already has some code to scan the extended caps. Using this method, I was able to find that, as of firmware 4015, the only Extended Capabilities provided by an uPD720200 USB 3.0 controller was "USB Legacy Support" (xHCI specs chpater 7.1) and "xHCI Supported Protocol" (7.2). The same applies for a Fresco Logic FL1000 based controller that I just got my hands on, though this one also sports vendor specific Extended Capabilities, with IDs 192 and 193.

Below are the answers from various manufacturers regarding their planned support of USB 3.0 debug. Being mostly interested in addon cards for existing systems, I haven't asked Intel or AMD.

NEC/Renesas:

"The uPD720200 does not support the debug port capability, and there is no plan to add it. But we have already added debug port support to our newer USB3 host controllers, uPD720201 and uPD720202."

VIA Labs:

"VL800/801 doesn't support debug port capability. As I know, the only one support is Renesas 3rd gen. host controller UPD720201_202, we plan to add this feature with VL805 that will MP next year."

Fresco Logic:

"FL1000/FL1009 didn't support USB debug extended capability.
We are doing it (for the next generation of controllers) and the schedule will be 2012/Q1."

USB_DOS: The (once) easiest way to create a DOS bootable USB stick on Windows

(Updated 2011.12.14: this guide is now obsolete and has been superseded. If you want to create an MS-DOS or FreeDOS bootable USB Flash Drive, please visit the Rufus page).

If you ever need to run a DOS utility (eg. to flash a BIOS or a firmware), and want to painlessly create a bootable DOS USB key from Windows, this is for you.

Courtesy of FreeDOS and the HP Bootable USB utility (HPUSBFW), the following archive contains everything you need to create a fully functional DOS bootable USB stick. Just extract the file below (using 7-zip if needed) and follow the instructions:

USB_DOS v1.0

Now, for some additional information, in case this may be of interest to you:

• The HPUSBFW executable has been modified from the original to request elevation on Vista and later. If you remove the manifest you should find that it matches the official HP one.
• The FreeDOS command.com and kernel.sys files come from the current version of FreeDOS (1.0). command.com has been patched to prevent it from requesting date and time input from the user at boottime. If you unpack command.com with upx, you should find that the only difference with official is a set of 4 NOPs (0x90).
  Oh, and yes I tried to recompile the latest FreeCOM from SVN, in a FreeDOS VMWare image, but it's been quite a struggle (hint: don't waste your time with OpenWatcom, use Borland Turbo C++ v2.01 instead) and the resulting command.com freezes after a few commands when used bare with kernel.sys, as is the case with HPUSBFW.

Creating a bootable UEFI DUET USB stick

Now that one of my patches has made it into the UEFI/EDK2 SVN repository, I'm going to provide a quite guide on the easiest way to create a bootable UEFI USB stick for legacy platforms, on Windows.
In case you are not familiar with EFI/UEFI, it is very much possible to run EFI even on legacy (i.e. non EFI) hardware, through a process called DUET, which provides a complete EFI emulation layer using the underlying BIOS. This is great for testing, without having to modify your hardware in any way.

It is all provided by the DuetPkg of the EDK2. However, the main issue with EDK2 is that it may be a difficult to get working, unless you used the same development tools as the EDK2 developers, which, on Windows would default to Visual Studio 2008, which is not free.

Isn't there simple way to use freely available tools on Windows, to easily compile a DUET USB bootstick?
Glad you asked. There is now, and here is the complete process that goes with it:

• Download and install the Windows Server 2003 WinDDK, v3790.1830, using this direct link (230 MB ISO download). This is of course not the latest DDK but unless you want to waste time in extra configuration, you might as well go with the supported version. If I find the time to fix EDK2 for the latest, I may send another patch to the EDK2 team, but for now, this will have to do.
• (Optional) If you plan to use ASL to compile ACPI, which isn't needed for DUET, create a C:\ASL directory, download the Microsoft ASL Assembler/Compiler archive from here, open the .exe with 7-zip. Then open the .msi from the .exe (Open Inside), find the file _C6C6461BF87C49D66033009609D53FA5, extract it to your C:\ASL directory and rename it asl.exe (count on Microsoft to provide an installer... for a mere 55 KB exe). Also note that the ASL compiler is for 32 bit only. If you're running 64 bit, tough luck.
• Mount or burn the ISO and install the WinDDK, to the default C:\WINDDK\3790.1830 directory.
• If needed, download and install TortoiseSVN, or any other SVN client.
• Fetch the latest EDK2 from SVN, using https://edk2.svn.sourceforge.net/svnroot/edk2/trunk/edk2 as the SVN repository.
• Open a DDK console ("Development Kits" → "Win2k3 Free x64 Build Environment" or "Win2k3 Free x86 Build Environment") and navigate to your edk2 direcvtory
• Run the command:
  edksetup.bat
  This will initialize your environment by creating the required files in the Conf\ directory. You can safely ignore the warning about cygwin.
• Open the file Conf\target.txt and edit the TOOL_CHAIN_TAG line to have:
  TOOL_CHAIN_TAG = DDK3790xASL
  You also may want to change MAX_CONCURRENT_THREAD_NUMBER and make sure TARGET_ARCH is properly set to either IA32 or X64 according to your needs, as it avoids having to provide extra arch parameter when invoking the commands below).
• In the console, run (32 bit):
  build -p DuetPkg\DuetPkgIA32.dsc
  or (64 bit)
  build -p DuetPkg\DuetPkgX64.dsc
  It may take a while but it should complete successfully.
• cd to DuetPkg\ and run:
  PostBuild.bat
• Plug in your USB key and check its drive letter. In the following I will assume it is F:, and run:
  CreateBootDisk.bat usb F: FAT16
• Unplug and replug the key as requested, and run:
  CreateBootDisk.bat usb F: FAT16 step2

You should now have a bootable UEFI USB key. Plug that into a PC, boot it, and you will be welcomed into the wonderful world of EFI.
Note that if you rebuild any of the packages, you only have to perform the last step to update the key.