Using rss2email and github feeds to only notify of issues

Long story short:

• We want any github issue notification related to the libusbx github project being pushed as an e-mail to libusbx-devel mailing list.
• The default notification mechanism of github is next to useless to do anything like that, especially if you're working with a project that is associated with a github organization.
• As a result, we have created a libusbx-devel user that, monitors the libusbx project as a pull-only participant, and that receives NewsFeed about issues among other things. This NewsFeed can then be used as RSS/Atom (https://github.com/libusbx-devel.private.atom), and fed through rss2email to send e-mail notifications to the mailing list.
• However, besides report about issues, the NewsFeed also contains reports about wiki updates and whatnot, i.e. loads of polluting events that we don't want the mailing list to get notified about. And neither github or rss2email offer the possibility to filter events on their own.
• A quick look at the feed with curl indicates that RSS/Atom entries are tagged, with something like: <id>tag:github.com,2008:IssuesEvent/1570141350</id> or <id>tag:github.com,2008:IssueCommentEvent/1570141345</id> for the ones that are of interest to us, and <id>tag:github.com,2008:GollumEvent/1570245314</id> for the ones that aren't
• Thus, if you use a dedicated copy of rss2email for the github feed, it is possible to modify the rss2email.py code, and add a filter that ensures only entries that have an issue related tag are processed, with:

  for entry in r.entries:
          id = getID(entry)
          if id[:25] != "tag:github.com,2008:Issue": continue
          
          # If TRUST_GUID isn't set, we get back hashes of the content.

  so that anything that isn't issue related is filtered out.
  

Of course, the proper way would be to add an INCLUDE_TAGS/EXCLUDE_TAGS section in rss2email's config.py, that can take wildcards, and then cross reference these in the code above, but since we don't have a all day...

Bonus 1: If you need authentication with rss2email, assuming that libusbx-devel is your user, the following is an example of the URI you should use for the RSS: https://libusbx-devel:PASSWORD@github.com/libusbx-devel.private.atom
Bonus 2: In case you want to play with the github JSON API, that pertains to issues, rather than the RSS, and provide libusbx/libusbx is your project, you can issue something like

curl -i -u libusbx-devel:PASSWORD https://api.github.com/repos/libusbx/libusbx/issues/events

Crafting an MBR from scratch

If you follow this site, you'll remember that we previoulsy crafted a BIOS from scratch. Of course we may as well follow that up with writing an MBR while we're at it!

This time, the problem that was put to us was as follows:

As part of XP/2003 installation support in Rufus, which, if needed to be reminded, is your friendly bootable USB stick creation tool, we thought it'd be nice if, rather than having to fiddle with boot.ini options like other XP ISO → USB apps do for the second part of the XP setup process, we did something similar to what the original optical installation medium provides, with a "Press any key to boot from CD/DVD..." prompt.

More technically, the issue is that XP was not exactly designed by Microsoft to be installable from and USB drive. Therefore Windows expects to see the BIOS disk ID of the HDD it boots from, during the later stage of the installation process, as the first bootable device = 0x80, whereas 0x80 is the disk ID the BIOS assigns to the USB whenever it boots from it.
Thus, scripted methods of installing XP from USB would install an ntldr + boot.ini on the USB and prompt the user to select the second drive (first bootable HDD) to continue the process. Else the other workaround is to unplug the USB drive during reboot after the first part of the installation process is complete, and plug it back later, as Windows still needs to read files from it. Since these methods deviate from what users would see from a regular installation from CD/DVD, we tried to see if we couldn't come up with something better.

Our solution, then, is to craft an MBR that does the following:

1. If a bootable HDD is detected as second BIOS bootable device (0x81), the MBR prompts the user whether they want to boot from USB and, if no input is given, will fall back to booting from the HDD (0x81) instead of USB (0x80)
2. According to the bootable disk ID provided in the USB partition table, the MBR will swap the 0x80 device with the ID provided.
   This means that for instance, if the first partition on the USB drive has disk ID 0x81, then the USB disk is remapped to this ID, whereas the original 0x81, which would typically be the first bootable HDD, is remapped as 0x80 (first bootable device). Because this approach falls between swap and remapping, we call it masquerading, as each bootable drive is now being masqueraded as a different one.

Once the above is properly set, then the setup process on the HD can be led to think it always boots from 0x80, even if it was the USB that actually booted the system, and operate as if that was the case, leaving the installation process as close as possible to what users would experience when installing from CD/DVD.

For a more technical breakdown, of our process is as follows, knowing that the BIOS would have copied over the 512 bytes MBR at address 0x00007C00 when we start to run it.

1. Because later stages need to copy boot records at address 0x00007C00, the first thing we do is move our 512 bytes code out of the way, by allocating 1 KB of RAM, duplicating our code there and then jumping to it.
2. With our code now safely out of the way, we attempt to read the MBR of the second bootable device (0x81, as 0x80 would be the USB) into the 0x00007C00 address we just moved out from using INT_13h (disk), with either function 02h or 42h depending on the extensions found (Some BIOSes, such as older DELL, can only use 42h, so we must handle both 02h and 42h). If the read is unsuccessful, we just boot the USB.
3. If the read is successful, we check the partition entries of the HDD MBR we just read, to see if there exists one that is bootable/active. If none is found, we give up and boot the USB.
4. If an active partition is found,  we prompt the user to hit a key if they want to boot from USB by installing an override for INT_08h (timer) to provide us with both a timeout and the ability to print a dot every second.
5. If the user presses a key, we install an override for INT_13h that does the following:
   - masquerades boot device 0x81 (first bootable HDD, second bootable device after the USB) as 0x80 (first bootable device)
   - masquerades the USB boot device (0x80) with the disk ID provided for its first partition (typically 0x81, but could be higher)
6. We then handle the rest of the boot process to the relevant boot record (after removing the INT_08h override if needed). If the HDD is booted, we simply jump to the MBR we read at address 0x00007C00. If booting from USB, we read the partition boot record (eg. NTFS boot record) into address 0x00007C00 and jump there.

All of the above (and more!) is done in 440 bytes and with 2 bytes to spare (or 9 if you count the ones spent on convenience instructions). Not bad...

On a side note, the process of overriding INT_13h is used by various MBR viruses (eg. Michelangelo), and we actually had to take some measures to prevent anti virus applications from detecting our MBR as one. Our override of INT_08h for "Please press any key to boot from USB..." is also very close to what Microsoft does in the bootfix.bin it provides on its XP/2003 installation media.

The x86 assembly source of the MBR can be accessed here. If you're interested in writing your own MBR, feel free to have a look at it. Or, if you just want to see the MBR in action, feel free to download the latest version of Rufus, and use it to install Windows XP or Windows 2003 from USB.

Chkdskx and Formatx by Mark Russinovich

There used to be a time where, before they got purchased by Microsoft, Mark Russinovich's SysInternals provided a slew of very useful Windows utilities, with source. Of these, one of the most interesting had to be Chkdskx and Formatx, 2 utilities that leverage the fmifs.dll to duplicate as much as the Microsoft chkdsk and format functionality. Mark had the following to say about these utilities:

Have you ever wondered how exactly NT's two file system management utilities, chkdsk and format, work? Maybe you've had an application that would have been perfect if you could have incorporated chkdsk or format functionality into it. I present Chkdskx and Formatx, two utilities that very precisely clone the command-line chkdsk and format utilities that come with NT. In fact, the clones support the same switches as the standard chkdsk and format and produce almost exactly the same output. So while they don't do anything special, their source code demonstrates how you can write your own interfaces to chkdsk and format functionality.

The truth is, fmifs usage has barely changed since the days of Windows NT and what applied then to formatting and checking volumes still applies today if you want to format or check a volume in Windows 7 or Windows 8. Unfortunately however, the original SysInternals site went down around 2006, along with its utilities and source... Fortunately Mick's blog provided a link to a torrent of a full SysInternals site rip issued in 2006 (which I also duplicate here as well), and I will now make the original FMIFS.zip archive available here for your convenience, as it can be quite useful.

Or, if you want to have an overview of how fmifs.dll is used in a modern application, I can only invite you to have a look at the CheckDisk() and FormatExCallback()/FormatThread() calls of Rufus's format.c/format.h.