2019-07-24

Installing Debian ARM64 on a Raspberry Pi 3 in UEFI mode


That's right baby, we're talking vanilla Debian ARM64 (not Raspbian, not Armbian) in pure UEFI mode, where you'll get a GRUB UEFI prompt allowing you to change options at boot and everything.

At long last, the Raspberry Pi can be used to install vanilla GNU/Linux distributions in the same manner as you can do on a UEFI PC. Isn't that nice?


Not that I don't like Raspbian or Armbian (as a matter of fact I am impressed by the very fine job the Armbian maintainers are doing with their distro), but I have now spent enough time helping with the UEFI Raspberry Pi 3 effort not to push this whole endeavour to its logical conclusion: Install vanilla ARM64 GNU/Linux distros. That's because, in terms of long term support and features, nothing beats a vanilla distro. I mean, what's the point of having an 64-bit CPU if the distro you're going to install forces you to use 32-bit?

Prerequisites

Hardware:

  • A micro SD card with sufficient space (16 GB or more recommended). You may also manage with a USB Flash Drive, but this guide is geared primarily towards SD card installation
  • A Raspberry Pi 3 (Model B or Model B+) with a proper power source. If you're ever seeing a lightning bolt on the top left of your display during install, please invest into a power supply that can deliver more wattage.
Note that our goal here is to install the system on a SD card, through netinstall, using a single media for the whole installation process.

In other words, there is no need to use an additional USB Flash Drive, as we could do, to boot the Debian installer and then install from USB to SD. This is mostly because it's inconvenient to have to use two drives when one can most certainly do, and also because while USB to SD may look easier on paper (no need to fiddle with the "CD-ROM" device for instance) it's actually more difficult to complete properly.

Thus, while I'll give you some pointers on how to perform a USB based installation in Appendix D, I can also tell you, from experience, that you are better off not trying to use a separate USB as your installation media and instead performing the installation from a single SD, as described in this guide.

Software:

  • The latest Raspberry Pi 3 UEFI firmware binary, along with the relevant Broadcom bootloader support files (i.e. bootcode.bin, config.txt, fixup.dat, start.elf).

    You can find a ready-to-use archive with all of the above at https://github.com/pbatard/RPi3/releases
    (RPi3_UEFI_Firmware_v#.##.zip, 3 MB).

    Note that this firmware archive works for both the Raspberry Pi 3 Model B and the Raspberry Pi 3 Model B+ (as the relevant Device Tree is automatically selected during boot).
  • (Optional) The non-free WLAN firmware binaries that are needed if you want to use Wifi for the installation.
    Note that, if you picked up the archive above then you don't need to do anything as the WLAN firmware binaries are included in it too.

Preparation


Note: a complete example of how to achieve the first 3 steps below using DISKPART on Windows or fdisk + mkfs on Linux is provided in Appendix A at the end of this post.
  • Partition your SD media as MBR and create a single partition of 300 MB of type 0x0e (FAT16 with LBA).
    Do not be tempted to use GPT as the partition scheme or 0xef (ESP) for the partition type, as the ondie Broadcom bootloader does not support any of those. It must be MBR and type 0x0e. You can use the command line utilities fdisk on Linux or DISKPART on Windows to do that.
  • Set the partition as active/bootable. This is very important as, otherwise, the Debian partition manager will not automatically detect it as ESP (EFI System Partition) which will create problems that you have to manually resolve (See Appendix C).
    If using fdisk on Linux, you can use a to set the partition as active.
    If using Windows, you can use DISKPART and then type the command active after selecting the relevant disk and partition.
  • Format the partition as FAT16. It MUST be FAT16 and not FAT32, as the Debian partition manager will not detect it as ESP otherwise and, again, you will have to perform extra steps to salvage your system befor reboot (Appendix C).
    The Linux and Windows base utilities should be smart enough to use FAT16 and not FAT32 for a 300 MB partition, so you should simply be able to use mkfs.vfat /dev/<yourdevice> (Linux) or format fs=fat quick in Windows' DISKPART. The Windows Disk Manager should also be smart enough to use FAT16 instead of FAT32 if you decide to use it to format the partition.
  • Extract the UEFI bootloader support files mentioned above to the newly formatted FAT partition. If you downloaded the Raspberry Pi 3 UEFI firmware binary from the link above, you just have to uncompress the zip file onto the root of your media, and everything will be set as it should be.
  • Extract the content of the Debian ISO you downloaded to the root of the FAT partition. On Windows you can use a utility such as 7-zip to do just that (or you can mount the ISO in File Explorer then copy the files).
Once you have completed the steps above, eject your SD card, insert it in your Pi 3 and power it up. Make sure no other media is plugged in besides the SD card. Especially, make sure that there aren't any USB Flash Drives or USB HDDs connected.

Initial Boot


Unless you did something wrong, you should see the multicoloured boot screen, which indicates that the Raspberry Pi properly detected your SD media and is loading the low level CPU bootloader from it.

Then you should see the black and white Raspberry logo, which indicates that the Raspberry Pi UEFI firmware is running.



(Optional) Provided that you have proper cooling, it is strongly recommended that you go to the UEFI firmware setup by pressing the Esc key once you see the Raspberry logo and then go to Device ManagerRaspberry Pi ConfigurationChipset Configuration to set CPU Clock to Max, as the default for the firmware is to limit the CPU clock to 600 MHz which is slow. Or you can also set your own CPU Frequency in the same menu.



Once you have done that, choose Reset and then let the boot process proceed until you see the GNU GRUB menu for the Debian installer.

On the GNU GRUB menu choose *Install (which should already be the default) and let the Debian installer process start.

Debian Installer


Note: In case anything goes wrong during install, remember that you can use Alt-F4 to check the current installation log for details about the error.
  • Select your Language, Country and Keyboard and let the installer proceed until it reports that No Common CD-ROM drive was detected.
  • At this stage, on Load CD-ROM drivers from removable media select No.
  • On Manually select a CD-ROM module and device select Yes.
  • On Module needed for accessing the CD-ROM select none.
  • On Device file for accessing the CD-ROM type exactly the following:

    -t vfat -o rw /dev/mmcblk0p1

    For the reasons why you need to type this, see Appendix B below.
  • With the "CD-ROM" device set, let the installation process proceed and retrieve the base packages from the media until it asks you for the non-free firmware files on the network hardware detection. If you plan to use the wired connection, you can skip the (Optional) step below.
  • (Optional) If you plan to use WLAN for the installation, choose Yes for Load missing firmware from removable media. If you created the media from that Raspberry Pi 3 firmware archive linked above, the relevant firmware files will be detected under the firmware/ directory.

    Note 1: Because there are multiple files to load, you will be prompted multiple times for different firmware files (look closely at their names, you will see that they are actually different). This is normal. Just select Yes for each new file.

    Note 2: Though they are included in the UEFI firmware zip archive we linked above, it is most likely okay not to provide the .clm_blob if you don't have it (the Wifi drivers should work without that file), so don't be afraid to select No here if needed.
  • Set up your network as requested by the installer by (optionally) choosing the network interface you want to use for installation and (also optionally) setting up your access point and credentials if you use Wifi.
  • Go through the hostname, user/password set up and customize those as you see fit.
  • Let the installer continue until you get to the Partition disks screen. There, for Partitioning method select Manual. You should see something like this:

    MMC/SD card #1 (mmcblk0) - 16.0 GB SD 2WCGO
         #1  primary  314.6 MB  B  K  ESP
             pri/log                  FREE SPACE

    If, instead, you see something like this:

    MMC/SD card #1 (mmcblk0) - 16.0 GB SD 2WCGO
         #1  primary  314.6 MB  B   fat16
             pri/log                FREE SPACE

    In other words, if you don't see B K ESP for the first partition, then it means that you didn't partition or format your drive as explained above and you will need to reference Appendix C (Help, I screwed up my partitioning!) to sort you out.
  • From there select the FREE SPACE partition and use the partition manager's menu to create two new primary partitions (one for swap and one for the root file system), until you have something like this:

    MMC/SD card #1 (mmcblk0) - 16.0 GB SD 2WCGO
         #1  primary  314.6 MB  B  K  ESP
         #2  primary    1.0 GB     f  swap    swap
         #3  primary   14.7 GB     f  ext4    /
    
  • Select Finish partitioning and write changes to disk and then Yes on Write the changes to disks? and let the installer continue with the base system installation.
  • After a while, the installer will produce a big red ominous message that says:

    [!!] Configure the package manager
      
    apt-configuration problem
    An attempt to configure apt to install additional packages from the CD failed.

    This, however, is actually a completely benign message and you can safely ignore it by selecting Continue . That's because, since we are conducting a net install, so we couldn't care less about no longer being to access the "CD-ROM" files after install...
  • Once you have dimissed the message above, pick the mirror closest to your geographical location and let the installer proceed with some more software installation (this time, the software will be picked from that network mirror rather than the media).
    When prompted for the "package usage survey" pick whichever option you like.
  • Finally, at the Software selection screen, select any additional software package you wish to install. Note that the "Debian desktop environment" should work out of the box if you decide to install it (though I have only tested Xfce so far). It's probably a good idea to install at least "SSH server".
  • Let the process finalize the software and GRUB bootloader installation and, provided you didn't screw up your partitioning (i.e. you saw B K ESP when you entered the partition manager, otherwise see Appendix C) select Continue to reboot your machine on the Installation complete prompt.

If everything worked properly, your system will now boot into your brand new vanilla Debian ARM64 system. Enjoy!

Post install fixes


Here are a few things that you might want to fix post install:
  1. You may find a cdrom0 drive on your desktop, which can't seem to be accessible. This is a leftover from the installer process not knowing how to handle the installation media device. You should edit /etc/fstab to remove it.
     
  2. If you installed the cups package, you may get an error while loading modules (systemctl --failed will report that systemd-modules-load.service is in failed state). This is all due to the current cups package trying to load IBM PC kernel modules... on a non PC device. To fix this, simply delete /etc/modules-load.d/cups-filters.conf and reboot.
 

Appendix A: How to create and format the SD partition for installation


IMPORTANT NOTE 1: Please make sure to select the disk that matches your SD media before issuing any of these commands. Using the wrong disk will irremediably destroy your data!

IMPORTANT NOTE 2: Do not be tempted to "force" FAT32 in DISKPART or mkfs and do not forget to set the bootable/active flag, else you will afoul of into the issue described in Appendix C. 

Windows

C:>diskpart

Microsoft DiskPart version 10.0.18362.1

Copyright (C) Microsoft Corporation.
On computer: ########

DISKPART> list disk

  Disk ###  Status         Size     Free     Dyn  Gpt
  --------  -------------  -------  -------  ---  ---
  Disk 0    Online          238 GB      0 B        *
  Disk 1    Online          465 GB  1024 KB        *
  Disk 4    Online         4657 GB  1024 KB        *
  Disk 5    Online         4649 GB      0 B        *
  Disk 6    Online           14 GB    14 GB

DISKPART> select disk 6

Disk 6 is now the selected disk.

DISKPART> clean

DiskPart succeeded in cleaning the disk.

DISKPART> convert mbr

DiskPart successfully converted the selected disk to MBR format.

DISKPART> create partition primary size=300

DiskPart succeeded in creating the specified partition.

DISKPART> active

DiskPart marked the current partition as active.

DISKPART> format fs=fat quick

  100 percent completed

DiskPart successfully formatted the volume.

DISKPART> exit

Leaving DiskPart...

C:>

Note, if needed you can also force a specific partition type (e.g. set id=0e to force FAT16 LBA), but that shouldn't be needed as DISKPART should set the appropriate type accordingly.

Linux


The following assumes /dev/sdf is your SD/MMC device. Change it in all the commands below to use your actual device.

(Optional) If your drive was partitioned as GPT, or if you're not sure, you may want to issue the two following commands first. If it's MBR you can skip this step:

# Delete the primary GPT:
dd if=/dev/zero of=/dev/sdf bs=512 count=34
# Delete the backup GPT.:
dd if=/dev/zero of=/dev/sdf bs=512 count=34 seek=$((`blockdev --getsz /dev/sdf` - 34))

Now use fdisk and mkfs to partition the drive:

root@debian:~# fdisk /dev/sdf

Welcome to fdisk (util-linux 2.34).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0x7d188929.

Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-31291391, default 2048):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-31291391, default 31291391): +300M

Created a new partition 1 of type 'Linux' and of size 300 MiB.

Command (m for help): t
Selected partition 1
Hex code (type L to list all codes): e
Changed type of partition 'Linux' to 'W95 FAT16 (LBA)'.

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

root@debian:~# mkfs.vfat -F 16 /dev/sdf1
mkfs.fat 4.1 (2017-01-24)
root@debian:~#

Appendix B: Why do we need to use -t vfat -o rw /dev/mmcblk0p1 as the CD-ROM device.

  • Why this weird device name with options? Because these are actually mount command line parameters and the Debian installer actually calls mount behind the scenes and feeds it exactly what we write here. This means we can hijack the device name field to invoke the additional mount parameters we need.
  • Why /dev/mmcblk0p1? That's simply name of the device for the first partition (p1) on the SD/MMC media (mmcblk0) as seen by the Linux kernel on a Raspberry Pi.
  • Why -t vfat? Because the Debian installer appends fstype=iso9660 to the mount option which prevents automount and forces us to override the file system type.
  • Why -o rw? Because the Debian installer won't be able to use the first partition for /boot/efi otherwise or load the WLAN firmware from the media (you get a device or resource busy when trying to remount the media).

Appendix C: Help I screwed up my partitioning!


Of course you did. You thought you knew better, and now you are paying the price...

The problem in  a nutshell is that:
  1. You can't use a regular ESP on a Raspberry Pi, on account that GPT or an MBR partition with type 0xef are not handled by the Broadcom CPU bootloader. And there is nothing you can do about this, because this is a behaviour that's hardcoded in the CPU silicon itself.
     
  2. The Debian installer's partition manager is very temperamental about what it will recognize as an ESP. In other words, if you don't use the perfect combination of boot flag, partition type and file system, it will fail to see it as an ESP.
Now the good news is that this is recoverable, but you need to know what you're doing.
  • The first thing you should do in the Debian partition manager is set the first partition to be used as ESP. In other words, you will need to edit the first partition until you get this:
    MMC/SD card #1 (mmcblk0) - 16.0 GB SD 2WCGO
         #1  primary  314.6 MB  B  K  ESP
             pri/log                  FREE SPACE
  • Then you can proceed as the guide describe, but you need to bear in mind that, as soon as you choose to write the partition changes, the partition manager will have changed your first partition type to 0xef, which, as we have seen is ubootable by the CPU. Therefore, DO NOT PROCEED WITH THE SYSTEM REBOOT AT THE END UNTIL YOU HAVE CHANGED THE PARTITION TYPE BACK.
  • To do that, once you get to the Installation complete prompt that asks you to select Continue to reboot, you need to press Alt-F2 then Enter to activate a console.
  • Then type exactly the following command:
    chroot /target fdisk /dev/mmcblk0
    Then press the keys t, 1, e, w
  • No you can go back to the installer console (Alt-F1) and select Continue to reboot

Appendix D: Installing to a SD from an USB Flash Drive


As I explained above, and though it may seem simpler, I would discourage to use this method to install Debian on a Raspberry Pi. But I can understand that, if you don't have a card reader, you may be constrained to using this method.

For the most part, this should work fine out of the box. As a matter of fact, if you do it this way, you won't have to fiddle with the "CD-ROM" media detection. However, I will now list some of the caveat you will face if you proceed this way:

Caveat 1: If you use guided partitioning your SD/MMC media will be formatted as GPT (because this is a UEFI system after all) which the Broadcom CPU used in the Raspberry Pi can not boot. It has to be MBR. How you are supposed to force MBR over GPT in the Debian partition manager, I'll let you figure out.

Caveat 2: Similarly, you need to go through the 0xef to 0x0e conversion of your ESP, as the Pi won't boot from that partition otherwise.

Caveat 3: Of course you will also need to duplicate all the bootcode.bin, fixup.bat and so on from your USB boot media onto the SD ESP partition if you want it to boot (which is the reason why is is much more convenient to just set the ESP and Debian installer on the SD right of the bat, so you don't risk forgetting to copy a file).

Caveat 4: When I tried USB to SD install, I found that the GRUB installer somehow didn't seem to create an efi/boot/bootaa64.efi, which, if left uncorrected, will prevent the system from booting automatically.

Addon: Fixing 

2018-10-31

GitHub verified commits with GPG, TortoiseGit and MSYS/MinGW

If you've been browsing git repositories in GitHub, you may have seen that some of them have Verified commits, which is a nice way to indicate that the person who actually committed the code is indeed who they say they are, and not an impersonator who just happened to reuse an e-mail address that is not theirs, for dubious reasons.

Typical display of "Verified" GPG commits in GitHub


Obviously, if you are the only person who has write access to your github repositories (which is how I tend to operate, for obvious security reasons) verified commits are not that much of a big deal. Still, having the badge show in github does help with ensuring that people who are browsing the repo know that you are taking security and trust seriously. So we might as well add commit signing, since it's pretty straightforward to do.

Now, since these are my main development tools, I will hereafter demonstrate how you can do that using TortoiseGit and MSYS/MinGW GPG on Windows. If you use something else, then you will have to look for post entries by other people, that match the tools you use. Also, to give credit where credit is due, I will point out that I am mostly copying Julian's dev.to entry titled "Sign your git commits with tortoise git on windows".

So, without further ado, here's how you should proceed:
  1. Create a new GPG key by firing up a MinGW prompt and issuing the following:

    $ gpg --full-generate-key --allow-freeform-uid
    gpg (GnuPG) 2.2.10-unknown; Copyright (C) 2018 Free Software Foundation, Inc.
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.
    
    gpg: keybox '/home/nil/.gnupg/pubring.kbx' created
    Please select what kind of key you want:
       (1) RSA and RSA (default)
       (2) DSA and Elgamal
       (3) DSA (sign only)
       (4) RSA (sign only)
    Your selection? 1
    RSA keys may be between 1024 and 4096 bits long.
    What keysize do you want? (2048) 4096
    Requested keysize is 4096 bits
    Please specify how long the key should be valid.
             0 = key does not expire
          <n>  = key expires in n days
          <n>w = key expires in n weeks
          <n>m = key expires in n months
          <n>y = key expires in n years
    Key is valid for? (0) 0
    Key does not expire at all
    Is this correct? (y/N) y
    
    GnuPG needs to construct a user ID to identify your key.
    
    Real name: Pete Batard
    Email address: pete@akeo.ie
    Comment:
    You selected this USER-ID:
        "Pete Batard <pete@akeo.ie>"
    
    Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
    We need to generate a lot of random bytes. It is a good idea to perform
    some other action (type on the keyboard, move the mouse, utilize the
    disks) during the prime generation; this gives the random number
    generator a better chance to gain enough entropy.
    We need to generate a lot of random bytes. It is a good idea to perform
    some other action (type on the keyboard, move the mouse, utilize the
    disks) during the prime generation; this gives the random number
    generator a better chance to gain enough entropy.
    gpg: /home/nil/.gnupg/trustdb.gpg: trustdb created
    gpg: key F3E83EBB603AF846 marked as ultimately trusted
    gpg: directory '/home/nil/.gnupg/openpgp-revocs.d' created
    gpg: revocation certificate stored as '/home/nil/.gnupg/openpgp-revocs.d/236D8595DE48618C26293122F3E83EBB603AF846.rev'
    public and secret key created and signed.
    
    pub   rsa4096 2018-10-31 [SC]
          236D8595DE48618C26293122F3E83EBB603AF846
    uid                      Pete Batard <pete@akeo.ie>
    sub   rsa4096 2018-10-31 [E]
    

    You'll notice that, when prompted, we chose to create a 4096 RSA and RSA key that never expires.

    During that process, you will also be prompted to enter the password that safeguards your key. This is the password you will have to enter each time you sign a new commit, so choose it wisely.

    Note that, when using MSYS2 + MinGW, your GPG keys will be stored under C:\msys2\home\<your_user_name>\.gnupg\.
     
  2.  Generate the public key in a format that GitHub can accept:

    $ gpg --armor --export pete@akeo.ie
    -----BEGIN PGP PUBLIC KEY BLOCK-----
    
    mQINBFvZ0+gBEAC7Jkdt3aW5iURti+36suQN9dmhGfVJMEV/Y9giby78wYcq51rj
    IvJ2AuYEhVgiFwT2hrlKuems0Jsln6wGUULAQXpLMU4XxlyKHwBE3ETXCXWQbzxH
    rNqerDKNu54M/r3XNCW7r38vwNdYrh656eLccZ/jOH8aSSZ9KkBjJ1wa78tx7YZy
    +FXXjDbamP3Pu3CPp7Nx3y69FCFm2uYrDkLWqcOvweME9imIqdsLfd5bM+wYclbN
    QQuZArV7uoQ2xYFlVweaob5U3iUsGUQYuY7x3Mlbz/73wYxuOGUt5n6de3tdefrN
    V5csD3aJVQKjFWOW2oNzI8Qik9pDie+3XQEfbIVHhgCx9kLVe2MzBaWrnPgk2Epj
    bIhRheqzvV15iC70QchMrtDzXOcbNhaytggYWPRx1YtEN3G4pPnsVfq0oSdNhwlw
    VLYm6eK+kjr0PykIANiiDDe/4WiFTIS1mobp++QCFXm41jtfXP6PM3NJdf1Hx5VX
    CcRQKXmukeyW4DfYtr9GoKeu9G1vGQev1U+qjtOk+9SRrofsqfCqzJP4drjbSyk9
    43q9HBYSBjnslisQnrhhcl5/5Yb99+sS2EnpW7am/sarCHGiPkLi6eHfYpbxX7Lg
    nLXjmXYlpyCkJnkgwzsTUs3+7w2KHaBZ7yme70x2edBD9f1Ar3zm+ryW5QARAQAB
    tBpQZXRlIEJhdGFyZCA8cGV0ZUBha2VvLmllPokCTgQTAQgAOBYhBCNthZXeSGGM
    JikxIvPoPrtgOvhGBQJb2dPoAhsDBQsJCAcCBhUKCQgLAgQWAgMBAh4BAheAAAoJ
    EPPoPrtgOvhGUpkQAIHSu7BNo4/jUhtHjBMiiYVE6eJh1J8+lWkXuATCxo3BXrMb
    AAAdNsrPca09NVdSli3xameKSnWt3hXRpkNM2cAC/Sus8UjYGDaCP1pWNyfmd70y
    /uAZGf1FIeWL4yIiFcDROobLqlCE+qViWu8sG2Ris8hGA8sjR0cn5891Q/ncHFtE
    YYHzh0mn+A9I/gGSvArqYJdNNBptGplo2fnQQIODwHNYSPMCzBawFoll6jocjg6q
    FqlawC5f9zPs5HP9k0k0pp37f8i+ANftCfdwOEWurfBDGqrxKiJIyIaS9kLzwCQX
    poJGZO/rVbCDGvexfVkqoKMJRK0jO2Rh3p0vifZ2cwKPSFjWfSjUiPAUpcz0nuV5
    BSkrMNc1VHgP1FM4v2Vpi7lnaoWMLpVz3VJ8yRyRD/7c7oVEl0NL8lHMZaHiPprf
    LmeLIgM5ndh9wkvD9j2EH5JR72lACQtg5n9qmbDro2uJbtGqrhqrVQdPrPtv1XoM
    0JAIL+1RvdTuPPBclmTLwdXaztlnEjJOA9loWpkyMIlZVcb/6TWamGAzxu4wMv8o
    aQpaVqNIO9kq79lZMHFGDE4VRHAjrJh3nXKpi+/JOIf7xKAnwrZAquAC+bfqYYUm
    W9jg35aB+jASlI7+TvQHgal2dFSYebCeWpwPlJr7XeXWJab+UNajeKxRQ2wMuQIN
    BFvZ0+gBEAC6nJAWbF6YAnPDaHTTBAAYEHlbiPTt8gYUgoxkUJxV2fcj0g2ye0+x
    gFh7Z3eTw5zq3iojah8EWBj5WOHeI1R1q244qaje467onbgowcxsFOH/TgBs1aew
    DWNDIMJl/vkSEY5xdmtJIGIUJ/+BH9U7kSX3lB5IFz37WH6hcgQZUjD0fx+Hv5ZX
    7Fz8YGXnBnJRwblCJbvkq2BD/1fSI5REddILkQAKd9mzRoXFvKRYwV5Oq78NU4cd
    5e20+ALHCPC7fQQ3jFzUo2WMLywWDAi42DOn7E6/tIZT7BwKF08ozNDPpWTj5OOO
    OAqjesgsXI410kdayv25LopHnnPCcIcjm35AtA8TDSEfPFlbm59tBo7q5VWi15yb
    X1+vkSZfcUoe9lXIr/Ea+RYgayI8xFkBiOlWn8NaWjWrZEr6OG4EOk97bAgey4M4
    KEJJkQsQYsVSQ8yVkt1wETkH6GHQFoyoFJUJkxeWDXoG9LyBYr7n+NSbjOAujy/c
    XyemCFkJXSeTcn4KAIboBvEV0nQOMjfaEr+hkfXbESfm92MSlL54arrgyY7vcOSI
    iztc4ZiTmkQPeeG4PsqUaHYB1lj+qapVQlZ9O+OFH280YWylLBZJMWOKM1lMqgz3
    Z2avF2FVax+xBeE8pMnWAUbKTHB7BQAhATjxGGlWy6QtJRxpOrTcGwARAQABiQI2
    BBgBCAAgFiEEI22Fld5IYYwmKTEi8+g+u2A6+EYFAlvZ0+gCGwwACgkQ8+g+u2A6
    +EbNAQ//WL261oYfKskEmBzz88M7Tt6aj8NyQmXyrIY6RoEYK4+rnS2zFwQfIF6p
    3e4avUZYF5xTOSuuiJv4IImnjlilHjA+r6LcmqIGKilIeFQwyNLVr+H/FvZSzKYY
    Psr6v0CCBn/6UICmrLoDgr1IiWmlwVDKVNXDZLGHprB00WBrso0pBVWEmbkKzlP9
    lYlC11yXo/wsLLnQNbz3DzcUgtyFExyL37EGr1zw2xfmwmRZRQmpILpuiBE/VGI0
    pH4JReeGjcqh0TkK+70whQnM9VX6eZbV4cwtBXg1CixY+cwyQcCreRTneGPQT9jj
    5dmD9duQOiDw5QGAoQ4tc6AxQcf62KsZmXQ715IMVrbn3leeoVR5PaFQ/PR3MQn+
    eS0f+wIDLBgD1tjUeOvjWs79sB7LAvinndZUA/6+nfxR29753gpssFW5tFEK5Kit
    OwCnNG4P3SjqfYAN+IIBTUUUPjGPHTKEd85XUBUlCJg7i1iLaeZqamp9oga4gv6d
    lLQ50J84i4yk02Afhlic5CNw1l9TfCgdFWF/9+WO7qzHmdJsZl/9Gs05J3hbPzqh
    uji6ujyI7v9vDTDC2tR1l3zHTomFJ6Vs42MdpaBWtnePAIohnhtLKCjG3/Z04idj
    jjGTV+5EASM2h3WV7vfmxem2HyxEM0lwa5zj8AtaWugqmiO6Rik=
    =aMFF
    -----END PGP PUBLIC KEY BLOCK-----
    
     
  3. Go to https://github.com/settings/gpg/new and copy/paste the public key data from above.
     
  4. Because we are going to call GPG independently of MinGW, now you must copy your C:\msys2\home\<your_user_name>\.gnupg directory to C:\Users\<your_user_name>\.

    This is needed because this is the default location gpg.exe looks for key when not invoked from msys/MinGW and it doesn't seem possible to alter it without modifying the registry or creating environment variables, which is cumbersome. Besides, this is important data and you are a lot more likely to backup the content of C:\Users\<your_user_name>\ than C:\msys2\home\, so it's probably not a bad idea to duplicate this valuable content there.
     
  5. Get the key id that you'll need to use in your config file with:
    $ gpg --list-keys --keyid-format LONG pete@akeo.ie
    gpg: checking the trustdb
    gpg: marginals needed: 3  completes needed: 1  trust model: pgp
    gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
    pub   rsa4096/F3E83EBB603AF846 2018-10-31 [SC]
          236D8595DE48618C26293122F3E83EBB603AF846
    uid                 [ultimate] Pete Batard <pete@akeo.ie>
    sub   rsa4096/308A9C6106D2FCE4 2018-10-31 [E]

    The 40 characters hex string under pub is the value you are after.
     
  6. In each project where you want to have signed commits, edit your .git/config so that it contains the following options:
    [user]
        signingkey = 236D8595DE48618C26293122F3E83EBB603AF846
    [commit]
        gpgsign = true
    [gpg]
        program = "C:/msys2/usr/bin/gpg.exe"
If you do the above correctly, then next time you commit into the git repo you modified, you should be prompted for your GPG key password, and, after you push to GitHub, you should find that the commit has the Verified badge.

Note that you can also validate whether your commit was properly signed, before pushing, by issuing:
$ git log --show-signature

2017-05-17

Using a YubiKey to store code signing certificates

Preamble (skip this if you only want the How To)

If you are a Windows software developer and/or distributor, then, by all means, you are well aware that you should always digitally sign your software, so that a minimum level of accountability and trust can be established between yourself and your users.

As you should also know, this process is usually accomplished by acquiring a Windows Authenticode credential (a credential is a certificate + its associated private key) which can then be used to digitally sign binary executables.

However, one must also consider the security aspect of the signing process and realize that, given the faintest opportunity, ill-intentioned people will try to grab your code signing credentials, if they can. For instance, perhaps you are already aware that the NSA's stuxnet virus was signed using credentials that were stealthily duplicated from JMicron and Realtek, and that, outside of state sponsored endeavours, malware authors are also exceedingly interested in acquiring data they could use to steal the identify of a trustworthy person, even more so if that person or entity is producing popular software.

(Image credits: Yubico.com)
This means that, should malware find its way on your development machine (as part of an infected development tool for instance which malware authors are likely to target if they can, as it can mean a huge payoff), it'll most likely be able to steal BOTH your credential and its private key password, since one can only expect semi-competent malware to implement both a disk scanning and a keylogging facility.

Therefore, as a code signing developer, you're only ever one dodgy software installation away from finding that your credential(s) and protectivve password(s) have been exfiltrated into very wrong hands indeed...

As a result, it doesn't take a paranoid person to realize that storing credentials on disk, even if it's a removable USB flash drive, that you only plug when signing binaries, is a very very bad idea. Similarly, if your alternative is to store your signing credentials into the Windows certificate store, and expect that it'll be enough, you should probably realize that the level of a software-only security solution on Windows goes about as far as the distance you can throw a chair... with Steve Ballmer sitting on it.

Thus, what you really want, is have your credentials stored on a dedicated removable device, that's designed precisely to protect that kind of stuff. And this is where FIPS 201 Personal Identity Verification (PIV) devices, and especially the convenient and relatively affordable YubiKeys from Yubico come into play.


For the record, most of the process described below can likely be applied to any FIPS 201 PIV device, but since YubiKeys are what I use, I'll focus only on YubiKey usage.

Prerequisites


First of all, it is important to note that not all YubiKeys are created equal. Especially, the cheapest YubiKey model does NOT have PIV support. Thus, if you plan to use a YubiKey for the purpose of signing code, you should steer away from the FIDO U2F Security Key model, as it is incompatible with this procedure. With this being said, the prerequisites are as follow:
  • Any YubiKey model EXCEPT the FIDO U2F Security Key. My preference goes for the YubiKey 4, but anything that has the PIV feature will do.
  • Your code signing credentials, which you have obtained from your Certification Authority, temporarily saved as a .p12 file (Note: You may have to use the Windows certificate store export feature to get to that file, and follow the procedure highlighted here, if your CA only delivers signing credentials into the certificate store)
  • The latest version of YubiKey PIV Manager, which you should download and install from here.

Storing your code signing credential into a YubiKey

  1. Open PIV Manager (pivman.exe). You may have to go fetch it from its installation directory if it did not create a Start menu entry, as was the case on my machine:


  2. Plug your YubiKey. If this is the first time you use it, you will be greeted by the following screen asking you to set a PIN:

    Under "Management Key" you should keep the "Use PIN as key" option checked.

    On the other hand, since you're going to use that key for code signing on Windows, you can disregard the cross-platform compatibility recommendation, as I haven't seen any issues with using a PIN with extended alphanumeric characters on Windows, and, with a length of 8 characters, the PIN is already short enough as it is.

    One thing I should point out is that, just like with a credit card, the device only gives you 3 attempts at entering the right PIN before locking itself (which is exactly what you want from a device that stores valuable data) so keep that in mind when you use it. Of course, a YubiKey can always be reset if locked, but you will lose access to the credentials stored on it.

  3. Once you have set the PIN, you should see the following screen, where you need to click the "Certificates" button:

  4. On the Certificates screen, select the "Digital Signature" tab:

  5.  Click "Import from file" and select your .p12 code signing credential. You will be prompted by a password, which of course is the password for the private key of your .p12 (and not the key's PIN).

  6. If everything goes well, you will see the following notice, which you should follow by unplugging your YubiKey:

  7. After re-plugging your YubiKey, and going back to the "Digital Signature" certificate, you should see details about the installed credential, which is ready to be used for code signing:

Bonus: Storing more than one code signing credential onto your YubiKey

If you are producing Windows software that still needs to target platforms like Vista or XP, you might be saying: "That's all very well, but what if I need to sign my software with both an SHA-1 and SHA-256 Authenticode credential? There's only one Digital Signature slot on the YubiKey after all..."

Well, the thing is, this is one of the exact issues I have been faced with for Rufus, and I can tell you that, as far as code signing is concerned, the labels assigned to the certificate/credential storage slots are pretty much irrelevant. You can use any of these 4 slots to store any code signing credential you want (since they are referenced by their fingerprint), and we only used the "Digital Signature" PIV slot because that's the one that makes most sense for storing a code signing signature. However, if you also want to store an SHA-1 credential, you can use any of the remaining slots to do that.

My preference is to use the optional "Card Authentication" slot to store your extra SHA-1 credential (so that you can use the "Authentication" and "Key Management" for actual authentication or key management if you ever need to). At least, this is what I have been doing for double-signing my Rufus application, and neither SignTool or the YubiKey seem to have any trouble with that.

Using the stored credentials with SignTool


Okay, so you have your code signing credential(s) safely stored on a secure YubiKey. Now what?
Clearly you can't use SignTool in the usual fashion, where you reference a local .p12 or .pfx file.

Instead, and especially if you have multiple code signing credentials residing on it, because the YubiKey is automatically detected as a credentials storage device by Windows, what you want to do is reference your credentials by their unique SHA-1 fingerprint in SignTool, and let Windows/YubiKey handle the rest. This is exactly what the /sha1 flag of SignTool is for.

However, before we can do that, we need to figure out the SHA-1 fingerprint of your certificate.
The simplest way to do that, while ensuring that you are really going to be accessing the credentials that you want to access, is:

  1. Go back to PIV Manager, and open the slot where the credential you are after resides:

  2. Click "Export Certificate" and save the file as a .crt (you will need to type the extension as part of the file name)

  3. Double click on the .crt you just saved and go to the "Details" tab

  4. Scroll down to the "Thumbprint" field (should be the very last) and copy its content. This is the SHA-1 fingerprint you are after:
Now you can use SignTool with /sha1 instead of /f and when you do so, you will be prompted to plug your YubiKey (if it isn't plugged in) as well as your PIN, which, if you enter successfully, will enable the signature operation.

I'll conclude with a real life example, using a YubiKey 4 where I store both an SHA-256 code signing credential (fingerprint 5759b23dc8f45e9120a7317f306e5b6890b612f0) and an SHA-1 credential (fingerprint 655f6413a8f721e3286ace95025c9e0ea132a984), that I use to sign and timestamp the dual SHA-1+SHA-256 Rufus binary:

SignTool sign /v /sha1 655f6413a8f721e3286ace95025c9e0ea132a984 /fd SHA1 /tr http://sha256timestamp.ws.symantec.com/sha256/timestamp rufus.exe
SignTool sign /as /v /sha1 5759b23dc8f45e9120a7317f306e5b6890b612f0 /fd SHA256 /tr http://sha256timestamp.ws.symantec.com/sha256/timestamp rufus.exe

IMPORTANT NOTE: Do *NOT* let Windows install the Yubikey Minidriver as part of Windows Update!


It looks like the latest versions of Windows insist on installing a Yubikey Minidriver, which ends up wrecking havoc on your ability to actually use a Yubikey as a signing device. If you let Windows have its way, you may end up getting the a message stating The smart card cannot perform the requested operation or the operation requires a different smart card when attempting to sign your binary:



If you get this issue, just go to your installed software and delete the Yubikey Smart Card Minidriver:


Once you have done that, you should find that you can use your Yubikey for signing applications again.

Or, if you want to use the MiniDriver, you can follow the steps highlighted here.

Final words

Now that you've seen how to do it, I would strongly urge you to go and purchase a YubiKey (or any other FIPS 201 PIV device) and NEVER, EVER again store code signing credentials on anything else than a secure password protected device that was designed precisely for this.

This means that, once you have done all of the above and validated that it works, you should DELETE your .p12/.pfx and remove any trace of your credential(s) from your computer.

Of course, if you are really worried, you may still choose to store a copy of said credential(s), on a backup CD-ROM (preferably in a password protected archive), that you'll only store in a locked place. But by all means, if you have a working YubiKey, you should not let your code signing credential(s) anywhere near any of the computers that you own!

2017-05-15

Compiling desktop ARM or ARM64 applications with Visual Studio 2017

Unlike what I was led to think, and despite the relative failure of Windows RT (which was kind of a given, considering Microsoft's utterly idiotic choice for its branding), it doesn't look like Microsoft has abandoned the idea of Windows on ARM/ARM64, including allowing developers to produce native desktop Windows applications for that platform.

However, there are a couple caveats to work through, before you can get Visual Studio 2017 to churn out Windows native ARM/ARM64 applications.

Caveat #1 - Getting the ARM/ARM64 compiler and libraries installed


In Visual Studio 2017 setup, Microsoft seem to have done their darnedest to prevent people from installing the MSVC ARM compilers. This is because, they are not allowing the ARM development components to be listed in the default "Workloads" view, and if you want them, you will need to go fetch them in the "Individual components" view as per the screenshot below:


It is important to note that you will need Visual Studio 2017 Update 4 (version 15.4) or later for the ARM64 compiler to be available for installation. ARM64 was silently added by Microsoft late in the Visual Studio 2017 update cycle, so if it's ARM64 you are after, you will need an updated version of the Visual Studio installer.

Caveat #2 - Error MSB8022: Compiling Desktop applications for the ARM/ARM64 platform is not supported.


Okay, now that you have the ARM/ARM64 compiler set, you created your nice project, set the target to ARM/ARM64 (while cursing Microsoft for even making that step painful), hit "Build Solution", and BAM!, you are only getting a string of:

Toolset.targets(53,5): error MSB8022: Compiling Desktop applications for the ARM platform is not supported.


What gives?

Short answer is, Microsoft doesn't really want YOU to develop native desktop ARM/ARM64 applications. Instead, they have this grand vision where you should only develop boring, limited UWP interpreted crap (yeah, I know the intermediary CIL/bytecode gets compiled to a reusable binary on first run, but it is still interpreted crap - I mean, if it is so great, then why isn't the Visual Studio dev env using it?), that they'll then be able to provide from the App Store. God forbid, in this day and age, you would still want to produce a native win32 executable!

So, they added an extra hurdle to produce native ARM/ARM64 windows binaries, which you need to overcome by doing the following:

  1. Open every single .vcxproj project file that is part of your solution
  2. Locate all the PropertyGroup parts that are relevant for ARM/ARM64. There should usually be two for each, one for Release|ARM[64] and one for Debug|ARM[64] and they should look something like this:

      <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|ARM[64]'" Label="Configuration">
        <ConfigurationType>Application</ConfigurationType>
        <CharacterSet>Unicode</CharacterSet>
        <WholeProgramOptimization>true</WholeProgramOptimization>
        <PlatformToolset>v141</PlatformToolset>
      </PropertyGroup> 

  3. Insert a new <WindowsSDKDesktopARMSupport>true</WindowsSDKDesktopARMSupport> or <WindowsSDKDesktopARM64Support>true</WindowsSDKDesktopARM64Support> property, so that you have (ARM64):

      <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|ARM64'" Label="Configuration">
        <ConfigurationType>Application</ConfigurationType>
        <CharacterSet>Unicode</CharacterSet>
        <WholeProgramOptimization>true</WholeProgramOptimization>
        <PlatformToolset>v141</PlatformToolset>
        <WindowsSDKDesktopARM64Support>true</WindowsSDKDesktopARM64Support>
      </PropertyGroup> 

If you do just that, then Visual Studio should now allow you to compile your ARM application without complaining. However, this may not be the end of it because...

Caveat #3 - You may still be missing some libraries unless you use a recent SDK


Depending on the type of desktop application you are creating, you may find that the linker will complain about missing libraries. Some of which are fairly easy to sort out, and are due to the Win32 and x64 default configs being more forgiving about not explicitly specifying libraries such as gdi32, advapi32, comdlg32.lib or shell32 for linking. However, if you are using the default 8.1 SDK in your project, you may also find that some very important libraries are missing for ARM, such as setupapi.lib and other ones. These libraries seem to only have been added by Microsoft recently, so it might be a good idea to switch to using a recent Windows SDK in your project, such as 10.0.15063.0:


Be mindful however that the default SDK that Visual Studio automatically installs if you select the C/C++ components is not full-featured, and will be missing important libraries for ARM/ARM64 compilation, so there again, you must go to the individual components and explicitly select the SDK you want, so that the full version gets installed.

With all the above complete, you should now be able to join the select club of developers who are able to produce actual native Windows applications for ARM/ARM64, and this just in time for the release of Windows 10 on ARM64.

Of course, and as usual, if you want a real-life example of how it's done, you can look at how Rufus does it...

2016-05-18

Adding a new driver to an existing UEFI firmware

Following up on efifs and VMWare firmware extraction, you might be interested to find out how, for instance, you should proceed to add an NTFS EFI driver to an existing UEFI firmware, so that you can access/boot NTFS volumes from UEFI.
Reading NTFS volumes natively from UEFI

If that's the case, then look no further than this guide.

It provides a step by step breakdown, using VMWare, of how you can generate an UEFI firmware module from an EFI driver executable (through FFS, which is a convenient repackaging of the EDK2's GenSec and GenFfs), and insert it into an existing UEFI firmware to make the driver natively available:
A VMWare UEFI firmware with a native NTFS driver

2016-05-16

Help, I lost all networking on my Raspberry Pi!

This happened to me the other day, as I was upgrading a Pi system from Debian Jessie to Sid.

After reboot, I suddenly got the following warning in the boot log:
[FAILED] Failed to start Raise network interfaces.
See 'systemctl status networking.service' for details.
And of course, issuing ifconfig returned the dreaded output with only loopback:
root@pi ~ # ifconfig
lo: flags=73  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10
        loop  txqueuelen 1  (Local Loopback)
        RX packets 4  bytes 240 (240.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4  bytes 240 (240.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
A look at the suggested systemctl status networking.service yielded no better results:
root@pi ~ # systemctl status networking.service
• networking.service - Raise network interfaces
   Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor preset: enabled)
  Drop-In: /run/systemd/generator/networking.service.d
           └─50-insserv.conf-$network.conf
   Active: failed (Result: exit-code) since Mon 2016-05-16 22:05:36 IST; 1min 2s ago
     Docs: man:interfaces(5)
  Process: 296 ExecStart=/sbin/ifup -a --read-environment (code=exited, status=1/FAILURE)
  Process: 288 ExecStartPre=/bin/sh -c [ "$CONFIGURE_INTERFACES" != "no" ] && [ -n "$(ifquery --read-environment --list --exclude=lo)" ] && udevadm settle (code=exited, status=0/SUCCESS)
 Main PID: 296 (code=exited, status=1/FAILURE)

May 16 22:05:36 pi systemd[1]: Starting Raise network interfaces...
May 16 22:05:36 pi ifup[296]: Cannot find device "eth0"
May 16 22:05:36 pi ifup[296]: Failed to bring up eth0.
May 16 22:05:36 pi systemd[1]: networking.service: Main process exited, code=exited, status=1/FAILURE
May 16 22:05:36 pi systemd[1]: Failed to start Raise network interfaces.
May 16 22:05:36 pi systemd[1]: networking.service: Unit entered failed state.
May 16 22:05:36 pi systemd[1]: networking.service: Failed with result 'exit-code'.
Drats! What on earth am I gonna do if I no longer have networking?!?

Well, below is what you can to to get out of this precarious situation:

  1. Issue a networkctl to confirm that your Ethernet interface is still present. At this stage, it will probably only be listed as enxa1b2c3..., where A1B2C3... is your Pi's MAC address:
    root@pi ~ # networkctl
    IDX LINK             TYPE               OPERATIONAL SETUP
      1 lo               loopback           carrier     unmanaged
      2 enxa1b2c3d4e5f6  ether              off         unmanaged
    
    2 links listed.
    
    
    This is actually where the problem lies: The network interface isn't mapped to it's usual eth0, which in turn makes the networking boot scripts go "Huh?"...
  2. Check that you can bring the interface up and down, to confirm that it isn't a hardware or kernel issue with the following:
    root@pi ~ # ifconfig enxa1b2c3d4e5f6 up
    [  190.272495] smsc95xx 1-1.1:1.0 enxa1b2c3d4e5f6: hardware isn't capable of remote wakeup
    [  190.285729] IPv6: ADDRCONF(NETDEV_UP): enxa1b2c3d4e5f6: link is not ready
    [  191.851700] IPv6: ADDRCONF(NETDEV_CHANGE): enxa1b2c3d4e5f6: link becomes ready
    [  191.864838] smsc95xx 1-1.1:1.0 enxa1b2c3d4e5f6: link up, 100Mbps, full-duplex, lpa 0xCDE1
    root@pi ~ # networkctl
    IDX LINK             TYPE               OPERATIONAL SETUP
      1 lo               loopback           carrier     unmanaged
      2 enxa1b2c3d4e5f6  ether              routable    unmanaged
    
    2 links listed.
    root@pi ~ # ifconfig enxa1b2c3d4e5f6 down
    [  199.3354400] smsc95xx 1-1.1:1.0 enxa1b2c3d4e5f6: hardware isn't capable of remote wakeup
    NB: Make sure you leave the interface down for the next steps.
  3. Now, we should be able to get eth0 going again by issuing this:
    root@pi ~ # ip link set enxa1b2c3d4e5f6 name eth0
    [  277.211063] smsc95xx 1-1.1:1.0 eth0: renamed from enxa1b2c3d4e5f6
    root@pi ~ # systemctl restart networking
    [  300.952068] smsc95xx 1-1.1:1.0 eth0: hardware isn't capable of remote wakeup
    [  300.959844] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
    [  302.475405] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
    [  302.484821] smsc95xx 1-1.1:1.0 eth0: link up, 100Mbps, full-duplex, lpa 0xCDE1
  4. A quick check with ifconfig should confirm that we're rolling again. However this is just a temporary solution, which won't persist after reboot. So we need something a bit more permanent, which is to create a /etc/udev/rules.d/70-persistent-net.rules (which is probably the one file that got screwed when you lost your network) and which should contain something like:
    SUBSYSTEM=="net", ACTION=="add", DRIVERS=="smsc95xx", ATTR{address}=="*", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

    If you add this file and reboot, you should find that everything's back in order again. Pfew, another crisis averted!

2016-01-07

Windows 10 N edition, MTP and EVR

If you have Windows 10 installed, you may have come into a stream of unexpected annoyances, such as being unable to access your Android device as an MTP device to copy files or the Enhanced Video Renderer (EVR) options not being offered as options in K-Lite Codec Pack's awesome Media Player.



What gives? Wasn't Windows 10 supposed to make things easier?!?

Well, as it turns out, if you happen to have the N version of Windows installed (which you can find by going to SettingsSystemAbout), you are effectively using a version of Windows that is crippled and has quite a lot more functionality removed, than simply the front facing Windows Video Player.

So off you head to the internet, where they tell you to install KB3010081 (the Media feature pack for Windows 10 N and Windows 10 KN editions)... except this doesn't work if you have the 1511 (Nov. 2015 update)!! Oh and you may also find out that, if you have the feature working before, the Nov. update broke it altogether.

That's because each update of Windows 10 requires its own specific Media Feature pack, which means that if you're using 1508, 1511, 1607, 1703, 1709 or 1803 (or upgraded to one of these versions) then you must install the corresponding pack from this list!

IMPORTANT NOTE: It looks like Microsoft is taking its sweet time to update the list when they release a new version of Windows. However, the direct download page may have the latest Media Feature Pack available before the list is updated.

Sure there are a small notices on some of these to indicate that they might have been superseded, but one really has to wonder why Microsoft can't provide a proper update for the Media Feature pack..

2015-01-08

Easily create UEFI applications using Visual Studio 2015

As pointed out before, Visual Studio is now essentially free for all development, and its solid IDE of course makes is very desirable as the environment to use to develop UEFI applications on Windows.

Now, you might have read that, short of using the oh-so-daunting EDK2, and the intricate voodoo magic you'll have to spend days on, to make it play nice with the Visual Studio IDE, there is no salvation in the UEFI world. However, this couldn't be further from the truth.

Enters UEFI:SIMPLE.

The thing is, Visual Studio can already compile EFI applications without having to rely on any external tools, and even if you want an EDK2 like environment, with the common EFI API calls that it provides, you can totally do away with the super heavy installation and setup of the EDK, and instead use the lightweight and straightforward GNU-EFI library, that provides about the same level of functionality (as far as building standalone EFI applications or drivers are concerned, which is what we are interested in).

So really, if you want to craft an EFI application in no time at all, all you need to do is:

  1. Install Visual Studio 2015, which is totally free and which, no matter who you work for or what restrictions your corporate IT department tries to impose, you are 100% legally entitled to when it comes to trying to compile and test UEFI:SIMPLE.
  2. As suggested by the Visual Studio installer, install a git client such as msys-git (or TortoiseGit + msys-git). Now, you're going to wonder why, with git support being an integral part of Visual Studio 2015, we actually need an external client, but one problem is that Microsoft decided to strip their embedded git client of critical functionality, such as git submodule support, which we'll need.
  3. Because you'd be a fool not to want to test your EFI application or driver in a virtual environment, and, thanks to QEMU, this is so exceedingly simple to achieve that UEFI:SIMPLE will do it for you, you should download and install QEMU, preferably the 64 bit version (you can find a 64 bit qemu installer here), and preferably to its default of C:\Program Files\qemu.
  4. Clone the UEFI:SIMPLE git project, using the URI https://github.com/pbatard/uefi-simple.git. For this part, you can either use the embedded git client from Visual Studio or your external client.
  5. Now, using your external git client, navigate to your uefi-simple directory and issue the following commands:
    git submodule init
    git submodule update
    This will fetch the gnu-efi library source, which we rely on to build our application.
  6. Open the solution file in Visual Studio and just click the "Local Windows Debugger" button to both compile and run our "Hello, World"-type application in QEMU.
    Through its debug.vbs script, which can be found under the "Resource File" category, the UEFI:SIMPLE solution will take of setting everything up for you, including downloading the OVMF UEFI firmware for QEMU.
    Note that if you didn't install QEMU into C:\Program Files\qemu\ you will need to edit debug.vbs to modify the path.
  7. Finally, because the UEFI:SIMPLE source is public domain, you can now use it as a starting point to build your own UEFI application, whilst relying on the standard EFI API calls that one expects, and, more importantly, with an easy way to test your module at your fingertips.
Oh and I should point out that UEFI:SIMPLE also has ARM support, and can also be compiled on Linux, or using MinGW if you don't want to use Visual Studio on Windows. Also, if you want real-life examples of fully fledged UEFI applications, that were built using UEFI:SIMPLE as their starting point, you should look no further than efifs, a project that builds whole slew of EFI file system drivers, or UEFI:NTFS, which allows seamless EFI boot of NTFS partition.

2014-11-18

Applying a series of Debian patches to an original source

Say you have a nice original source package, as well as bunch of extra Debian patches, which you want to apply to that source (for instance, you may want to compile Debian's grub 2.00-22 using the tarballs you picked here.

However, since Debian uses quilt, or whatever it's called, to automate the application of a series of patches, and you either don't have it on your system or don't want to bother with it (since you're only interested in the patches), you end up wanting to apply all the files from the patches directory of the .debian addon, and there's of course no way you'll want to do that manually.

The solution: Copy the patches/ directory from the Debian addon to the root of your orig source, and run the following shell script.

#!/bin/bash
while read p; do
  patch -p1 < ./patches/$p
done < ./patches/series

By Grabthar's hammer, what a timesaver!

2014-11-16

Compiling Grub4DOS on MinGW

Since chenall committed a handful of patches that I submitted to make the compilation of Grub4DOS on MinGW easier, I'm just going to jolt down some quick notes on how you can produce a working Grub4DOS on Windows.
Note that part of this guide is shamelessly copied from the RMPrepUSB Grub4DOS compilation guide.
  • If you don't already have a git client, download and install msys-git (a.k.a. "Git for Windows") from here.
  • Download the latest MinGW32 installer (mingw-get-setup.exe) by clicking the "Download Installer" button on the top right corner of the main MinGW site.
  • Keep the default options on the first screen (but you can change the destination directory if you want)
  • On the package selection screen, select
    • mingw-developer-toolkit
    • mingw-base
    • msys-base
  • Select menu InstallationApply Changes and click Apply
  • Now navigate to your msys directory, e.g.. C:\MinGW\msys\1.0\, and open the file etc\profile in a text editor.
  • Assuming that you installed msys-git in C:\Program Files (x86)\Git, change the following:
    if [ $MSYSTEM == MINGW32 ]; then
      export PATH=".:/usr/local/bin:/mingw/bin:/bin:$PATH"
    else
      export PATH=".:/usr/local/bin:/bin:/mingw/bin:$PATH"
    fi
    to
    if [ $MSYSTEM == MINGW32 ]; then
      export PATH=".:/usr/local/bin:/mingw/bin:/bin:/c/Program Files (x86)/Git/bin:$PATH"
    else
      export PATH=".:/usr/local/bin:/bin:/mingw/bin:/c/Program Files (x86)/Git/bin:$PATH"
    fi
    This is to ensure that your system will be able to invoke git. Of course, if you use a different git client, you can ignore this step.
  • Download nasm (current build is: http://www.nasm.us/pub/nasm/releasebuilds/2.11.06/win32/nasm-2.11.06-win32.zip) extract and copy nasm.exe to C:\MinGW\msys\1.0\bin (the other files in the zip archive can be discarded).
  • Download upx (current build is: ftp://ftp.heanet.ie/mirrors/sourceforge/u/up/upx/upx/3.91/upx391w.zip) extract and copy upx.exe to C:\MinGW\msys\1.0\bin (the other files in the zip archive can be discarded).
  • In C:\MinGW\msys\1.0\ launch msys.bat
  • In the shell that appears, issue the following command (this may be necessary to locate mingw-get):
    /postintall/pi.sh
    You should accept all the default options.
  • Now issue the following commands:
    mingw-get upgrade gcc=4.6.2-1
    mingw-get install mpc=0.8.1-1
    This will effectively downgrade your compiler to gcc 4.6.2, which is necessary as gcc 4.7 or later doesn't seem to produce a working grldr for the time being.
  • Download the latest Grub4DOS source from github by issuing the following command
    git clone https://github.com/chenall/grub4dos.git
    Note: By default this will download the source into C:\MinGW\msys\1.0\home\<your_user_name>\grub4dos\, but you can of course navigate to a different directory before issuing the git clone command if you want it elsewhere.
  • Run the following commands:
    cd grub4dos
    ./autogen.sh
    make
At the end of all this, you should end up with a grldr and grldr.mbr in the C:\MinGW\msys\1.0\home\<your_user_name>\grub4dos\stage2\ directory, which is what you want

IMPORTANT: Do not try to invoke ./configure directly on MinGW, as compilation will fail. Instead should ensure that you call autotools to re-generate configure and Makefiles that MinGW will be happy with. Note that you can run ./bootstrap.sh instead of ./autogen.sh, if you don't want configure to be invoked with the default options.

What's the deal with gcc 4.7 or later on MinGW?

I haven't really investigated the issue, but the end result is that grldr is 303 KB, vs 307 KB for gcc 4.6.2, and freezes at boot after displaying:
A20 Debug: C806 Done! ...

I'm getting an error about objcopy during the configure test...

That's because you're not listening to what I say and try to compile a version of Grub4DOS that doesn't contain the necessary updates for MinGW. You must use a version of the source that's more recent than 2014.11.14 and right now, that source is only available if you clone from git.

Dude, could you, like, also provide the steps to compile from Linux?

Sigh... Alright, since I'm a nice guy, and it's a lot simpler, I'll give you the steps for a bare Debian 7.7.0 x64 Linux setup:
aptitude install gcc glibc-devel.i686 gcc-multilib make autotools autoconf git nasm upx
git clone https://github.com/chenall/grub4dos.git
cd grub4dos
./autogen.sh
make
Happy now? Note that the Linux compiled version is usually a lot smaller than the MinGW32 compiled one.

2014-11-13

Visual Studio 2013 has now become essentially free...

See http://www.visualstudio.com/products/visual-studio-community-vs.

I'm just going to point out to the first 2 paragraph of the license terms:
1.   INSTALLATION AND USE RIGHTS.
a.   Individual license. If you are an individual working on your own applications to sell or for any other purpose, you may use the software to develop and test those applications.
b.   Organization licenses. If you are an organization, your users may use the software as follows:
  • Any number of your users may use the software to develop and test your applications released under Open Source Institute (OSI)-approved open source software licenses.
  • Any number of your users may use the software to develop and test your applications as part of online or in person classroom training and education, or for performing academic research.
  • If none of the above apply, and you are also not an enterprise (defined below), then up to 5 of your individual users can use the software concurrently to develop and test your applications.
  • If you are an enterprise, your employees and contractors may not use the software to develop or test your applications, except for open source and education purposes as permitted above. An “enterprise” is any organization and its affiliates who collectively have either (a) more than 250 PCs or users or (b) more than one million US dollars (or the equivalent in other currencies) in annual revenues, and “affiliates” means those entities that control (via majority ownership), are controlled by, or are under common control with an organization.
Basically, this means that even if you're a corporate user, you can legally install and use Visual Studio Community Edition, on any computer you want, to compile and/or contribute to Open Source projects, and this regardless of your company's internal policies regarding the installation of Software (otherwise any company could enact an internal policy such as "Microsoft software licenses don't apply here" to be entitled to install as many unlicensed copies of Windows as they like).
So I have to stress this out very vehemently: If a company or IT department tries to take your right to download and install Visual Studio 2013 Community Edition to compile or test Open Source projects, THEY ARE IN BREACH OF THE LAW!
The only case where you are not entitled to use Visual Studio Community Edition is if you're developing a closed source application for a company. But who in their right mind would ever want to do something like that anyway?... ;)

So all of a sudden, you no longer have to jump through hoops if you want to recompile, debug and contribute to rufus, libusb or libwdi/Zadig - simply install Visual Studio 2013, as you are fully entitled to (because all these projects use an OSI approved Open Source license), and get going!

Oh, and for the record, if you want to keep a copy of Visual Studio 2013 Community Edition, for offline installation, you should run the installer as:
vs_community.exe /layout
Note however that this will send you back 8 GB in terms of download size and disk space.

2014-10-08

Free SSL certificate for Open Source projects

Just going to point out that GlobalSign are currently offering a 1 year SSL certificate for Open Source projects for free.

Alas, this is only for a specific domain name, such as app.project.org, rather than for a wildcard domain, such as *.project.org, and at this stage, I'm not entirely sure if the certificate is also renewable for free after one year. But at least, this now allows me to offer access to Rufus from https://rufus.akeo.ie.

Oh, and once your site is set for SSL, you probably want to ensure that it is properly configured by running it through Qualys SSL Labs' excellent SSL analysis tool.

And I'm just going to jolt down that, to get a proper grade with Apache, you may have to edit your /etc/apache2/mods-enabled/ssl.conf and set the following:
SSLProtocol all -SSLv2 -SSLv3

SSLHonorCipherOrder on
SSLCipherSuite EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH:EDH+aRSA:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4

2014-09-29

Getting proper coloured directory listing with Debian and Putty

Since I keep having to do that:

  1. In putty, in the Colours setting tab for your connection, make sure that "Indicate bolded text by changing" is set to "The colour" and not "The font"
  2. In Debian's bashrc, uncomment the line:
    force_color_prompt=yes

2014-06-30

So I built an NTFS EFI driver...

It's Free Software of course, and it only took me about two weeks to do so.

Since I've been doing it in my limited spare time, I might as well brag about it and say that, had I been able to work on this full time (which I sure wouldn't mind), it probably wouldn't have taken more than 7 days... Can't help but wonder how much a proprietary/non-free software development workflow would have had to budget, or outsource, to try to achieve the same thing, within the same amount of time.

At the very least, this demonstrates that, if you start with the the right resource set and, more importantly, if you stop being irrational about how "using the GPLv3 means the death knell of your software revenue stream", a project such as this one can easily and cheaply be completed in a matter of days.


Anyway, the driver itself is read-only (which is all I need for Rufus, as my intent is to use it there) and it could probably use some more polishing/cleanup, but it is stable enough to be used right now.

So, if you are interested in a redistributable and 100% Free Software read-only NTFS EFI driver, you should visit:
https://efi.akeo.ie (the link includes pre-built binaries).

Alternatively, you can also visit the github project page at:
https://github.com/pbatard/efifs

Now, I'd be ungrateful if I didn't mention that the main reason I was able to get something off the ground this quickly is thanks to the awesome developers behind the GRUB 2.0 project, who abstracted their file system framework enough, to make reusing their code in an EFI implementation fairly straightforward.
And I also have to thank the iPXE developers, who did most of the back-breaking work in figuring out a GPL friendly version of an EFI FS driver, that I could build on.
Finally, I was also able to reuse some of the good work from the rEFInd people (the GPLv3 compatible parts), which was big help!

But the lesson is: Don't waste your time with proprietary/non-free software. If you are both interested in being productive and budget-conscious, Free Software is where it's at!

2014-05-31

Restoring EFI NVRAM boot entries with rEFInd and Rufus

So, you reinstalled Windows, and it somehow screwed the nice EFI entry you had that booted your meticulously crafted EFI system partition? You know, the one you use with rEFInd or ELILO or whatever, to multiboot Linux, Windows, etc., and that has other goodies such as the EFI shell...

Well, here's how you can sort yourself out (shamelessly adapted from the always awesome and extremely comprehensive Arch Linux documentation):
  • Download the latest rEFInd CD-R image from here.
  • Extract the ISO and use Rufus to create a bootable USB drive. Make sure that, when you create the USB, you have "GPT partition scheme for UEFI computer" selected, under "Partition scheme and target system type".
  • Boot your computer in UEFI mode, and enter the EFI BIOS to select the USB as your boot device
  • On the rEFInd screen select "Start EFI shell".
  • At the 2.0 Shell > prompt type:
    bcfg boot dump

    This should confirm that some of your old entries have been unceremoniously wiped out by Windows.
  • Find the disk on which your old EFI partition resides by issuing something like:
    dir fs0:\efi

    NB: you can use the map command to get a list of all the disks and partitions detected during boot.
  • Once you have the proper fs# information (and provided you want to add an entry that boots into a rEFInd installed on your EFI system partition under EFI\refind\refind_x64.efi), issue something like:
    bcfg boot add 0 fs0:\EFI\refind\refind_x64.efi rEFInd
    Note: If needed you can also use bcfg boot rm # to remove existing entries.
  • Confirm that your entry has been properly installed as the first option, by re-issuing bcfg boot dump. Then remove the USB, reset your machine, and you should find that everything is back to normal.
NOTE: Make sure you use the latest rEFInd if you want an EFI shell that includes bcfg. Not all EFI shells will contain that command!