Showing posts with label linux. Show all posts
Showing posts with label linux. Show all posts

2020-05-15

Why is my Samba connection failing?

Or how nice it is to have a problem that has long eluded you finally explained.

Part 1: The horror

You see, I've been using a FriendlyArm/FriendlyElec RK3399-based NanoPC-T4 to run Linux services, such as a staging web server for Rufus, network print host, various other things as well as a Samba File Server...

However, this Samba functionality seemed to be plagued like there was no tomorrow: Almost every time I tried to fetch a large file from it, Windows would freeze during transfer, with no recovery and not even the possibility of cancelling unless the Samba service was restarted manually on the server.

But what was more vexing is that these problems with Samba did not manifest themselves until I switched from using an old Lubuntu distribution, that was provided by the manufacturer of that device, to a more up to date Armbian. With Lubuntu, Samba seemed rock-solid, but with Armbian, it was hopeless.


This became so infuriating that I had to giveup on using Samba on that machine altogether and, considering that things usually seemed to be okay-ish after the service had restarted, I dismissed it as a pure Samba/arch64 bug, that newer versions of Samba or Debian had triggered, and that would eventually get fixed. But of course, that long awaited fix never seemed to manifest itself and I had better things to do than invest time I didn't have trying to troubleshoot a functionality that wasn't that critical to my workflow.


Besides, the Samba logs were all but useless. Nothing in there seemed to provide any indication that Samba was even remotely unhappy. And of course, you can forget about Windows giving you any clue about why the heck your Samba file transfers are freezing...

Part 2: The light at the end of the tunnel

Recently however, in the course of the Raspberry Pi 4 UEFI firmware experiments, it turns out that I was using that same server to test UEFI HTTP boot of a large (900 MB) ISO, that was being served from the Apache server running on that NanoPC machine, and had no joy with getting the full transfer complete either. Except, there, it wasn't freezing. It just seemed to produce a bunch of TcpInput: received a checksum error packet before giving up on the transfer altogether...

URI: http://10.0.0.7/~efi/ubuntu.iso
File Size: 916357120 Bytes
Downloading...1%
TcpInput: received a checksum error packet TcpInput: Discard a packet TcpInput: received a checksum error packet TcpInput: Discard a packet TcpInput: received a checksum error packet TcpInput: Discard a packet TcpInput: received a checksum error packet TcpInput: Discard a packet TcpInput: received a checksum error packet TcpInput: Discard a packet TcpInput: received a checksum error packet TcpInput: Discard a packet HttpTcpReceiveNotifyDpc: Aborted! Error: Server response timeout.

Yet, serving the same content from the native python3 HTTP server (python3 -m http.server 80, which is a super convenient command to know as it acts as an HTTP server and serves any content from the current directory through the specified port) appeared to be okay, albeit with the occasional checksum errors. This is suddenly starting to look like a lot of compounded network errors... Could this be related to that Samba issue?


Now, the first thing you do when you get reports of TCP checksum errors, is try a different cable, a different switch and so on, to make sure that this is not a pure hardware problem. But I had of course tried that during the process of trying to troubleshoot the failing Samba server, and, once again, the results of switching equipment and cabling around were all negative.

But at least a bunch of checksum errors does give you something to start to work with.

For one thing, you can monitor these errors with tcpdump (tcpdump -i eth0 -vvv tcp | grep incorrect) and, more importantly, you may find some very relevant articles that point you to the very root of the problem.

Long story short, if tcpdump -i eth0 -vvv tcp | grep incorrect produces loads of checksum errors on the platform you serve content from, you may want to look into disabling offloading from the network adapter with something like:

ethtool -K eth0 rx off tx off

Or you may continue to hope that the makers of your distro will take action, but that might just turn out to be wishful thinking...

2019-07-24

Installing Debian ARM64 on a Raspberry Pi 3 in UEFI mode


That's right baby, we're talking vanilla Debian ARM64 (not Raspbian, not Armbian) in pure UEFI mode, where you'll get a GRUB UEFI prompt allowing you to change options at boot and everything.

At long last, the Raspberry Pi can be used to install vanilla GNU/Linux distributions in the same manner as you can do on a UEFI PC. Isn't that nice?


Not that I don't like Raspbian or Armbian (as a matter of fact I am impressed by the very fine job the Armbian maintainers are doing with their distro), but I have now spent enough time helping with the UEFI Raspberry Pi 3 effort not to push this whole endeavour to its logical conclusion: Install vanilla ARM64 GNU/Linux distros. That's because, in terms of long term support and features, nothing beats a vanilla distro. I mean, what's the point of having an 64-bit CPU if the distro you're going to install forces you to use 32-bit?

Prerequisites

Hardware:

  • A micro SD card with sufficient space (16 GB or more recommended). You may also manage with a USB Flash Drive, but this guide is geared primarily towards SD card installation
  • A Raspberry Pi 3 (Model B or Model B+) with a proper power source. If you're ever seeing a lightning bolt on the top left of your display during install, please invest into a power supply that can deliver more wattage.
Note that our goal here is to install the system on a SD card, through netinstall, using a single media for the whole installation process.

In other words, there is no need to use an additional USB Flash Drive, as we could do, to boot the Debian installer and then install from USB to SD. This is mostly because it's inconvenient to have to use two drives when one can most certainly do, and also because while USB to SD may look easier on paper (no need to fiddle with the "CD-ROM" device for instance) it's actually more difficult to complete properly.

Thus, while I'll give you some pointers on how to perform a USB based installation in Appendix D, I can also tell you, from experience, that you are better off not trying to use a separate USB as your installation media and instead performing the installation from a single SD, as described in this guide.

Software:

  • The latest Raspberry Pi 3 UEFI firmware binary, along with the relevant Broadcom bootloader support files (i.e. bootcode.bin, config.txt, fixup.dat, start.elf).

    You can find a ready-to-use archive with all of the above at https://github.com/pftf/RPi3/releases
    (RPi3_UEFI_Firmware_v#.##.zip, 3 MB).

    Note that this firmware archive works for both the Raspberry Pi 3 Model B and the Raspberry Pi 3 Model B+ (as the relevant Device Tree is automatically selected during boot).
  • (Optional) The non-free WLAN firmware binaries that are needed if you want to use Wifi for the installation.
    Note that, if you picked up the archive above then you don't need to do anything as the WLAN firmware binaries are included in it too.

Preparation


Note: a complete example of how to achieve the first 3 steps below using DISKPART on Windows or fdisk + mkfs on Linux is provided in Appendix A at the end of this post.
  • Partition your SD media as MBR and create a single partition of 300 MB of type 0x0e (FAT16 with LBA).
    Do not be tempted to use GPT as the partition scheme or 0xef (ESP) for the partition type, as the ondie Broadcom bootloader does not support any of those. It must be MBR and type 0x0e. You can use the command line utilities fdisk on Linux or DISKPART on Windows to do that.
  • Set the partition as active/bootable. This is very important as, otherwise, the Debian partition manager will not automatically detect it as ESP (EFI System Partition) which will create problems that you have to manually resolve (See Appendix C).
    If using fdisk on Linux, you can use a to set the partition as active.
    If using Windows, you can use DISKPART and then type the command active after selecting the relevant disk and partition.
  • Format the partition as FAT16. It MUST be FAT16 and not FAT32, as the Debian partition manager will not detect it as ESP otherwise and, again, you will have to perform extra steps to salvage your system before reboot (Appendix C).
    The Linux and Windows base utilities should be smart enough to use FAT16 and not FAT32 for a 300 MB partition, so you should simply be able to use mkfs.vfat /dev/<yourdevice> (Linux) or format fs=fat quick in Windows' DISKPART. The Windows Disk Manager should also be smart enough to use FAT16 instead of FAT32 if you decide to use it to format the partition.
  • Extract the UEFI bootloader support files mentioned above to the newly formatted FAT partition. If you downloaded the Raspberry Pi 3 UEFI firmware binary from the link above, you just have to uncompress the zip file onto the root of your media, and everything will be set as it should be.
  • Extract the content of the Debian ISO you downloaded to the root of the FAT partition. On Windows you can use a utility such as 7-zip to do just that (or you can mount the ISO in File Explorer then copy the files).
Once you have completed the steps above, eject your SD card, insert it in your Pi 3 and power it up. Make sure no other media is plugged in besides the SD card. Especially, make sure that there aren't any USB Flash Drives or USB HDDs connected.

Initial Boot


Unless you did something wrong, you should see the multicoloured boot screen, which indicates that the Raspberry Pi properly detected your SD media and is loading the low level CPU bootloader from it.

Then you should see the black and white Raspberry logo, which indicates that the Raspberry Pi UEFI firmware is running.



Wait for the GNU GRUB menu to appear (which it should do by default after the Pi logo disappears) and choose *Install (which should already be the default) and let the Debian installer process start.

Debian Installer


Note: In case anything goes wrong during install, remember that you can use Alt-F4 to check the current installation log for details about the error.
  • Select your Language, Country and Keyboard and let the installer proceed until it reports that No Common CD-ROM drive was detected.
  • At this stage, on Load CD-ROM drivers from removable media select No.
  • On Manually select a CD-ROM module and device select Yes.
  • On Module needed for accessing the CD-ROM select none.
  • On Device file for accessing the CD-ROM type exactly the following:

    -t vfat -o rw /dev/mmcblk0p1

    For the reasons why you need to type this, see Appendix B below.
  • With the "CD-ROM" device set, let the installation process proceed and retrieve the base packages from the media until it asks you for the non-free firmware files on the network hardware detection. If you plan to use the wired connection, you can skip the (Optional) step below.
  • (Optional) If you plan to use WLAN for the installation, choose Yes for Load missing firmware from removable media. If you created the media from that Raspberry Pi 3 firmware archive linked above, the relevant firmware files will be detected under the firmware/ directory.

    Note 1: Because there are multiple files to load, you will be prompted multiple times for different firmware files (look closely at their names, you will see that they are actually different). This is normal. Just select Yes for each new file.

    Note 2: Though they are included in the UEFI firmware zip archive we linked above, it is most likely okay not to provide the .clm_blob if you don't have it (the Wifi drivers should work without that file), so don't be afraid to select No here if needed.
  • Set up your network as requested by the installer by (optionally) choosing the network interface you want to use for installation and (also optionally) setting up your access point and credentials if you use Wifi.
  • Go through the hostname, user/password set up and customize those as you see fit.
  • Let the installer continue until you get to the Partition disks screen. There, for Partitioning method select Manual. You should see something like this:

    MMC/SD card #1 (mmcblk0) - 16.0 GB SD 2WCGO
         #1  primary  314.6 MB  B  K  ESP
             pri/log                  FREE SPACE

    If, instead, you see something like this:

    MMC/SD card #1 (mmcblk0) - 16.0 GB SD 2WCGO
         #1  primary  314.6 MB  B   fat16
             pri/log                FREE SPACE

    In other words, if you don't see B K ESP for the first partition, then it means that you didn't partition or format your drive as explained above and you will need to reference Appendix C (Help, I screwed up my partitioning!) to sort you out.
  • From there select the FREE SPACE partition and use the partition manager's menu to create two new primary partitions (one for swap and one for the root file system), until you have something like this:

    MMC/SD card #1 (mmcblk0) - 16.0 GB SD 2WCGO
         #1  primary  314.6 MB  B  K  ESP
         #2  primary    1.0 GB     f  swap    swap
         #3  primary   14.7 GB     f  ext4    /
    
  • Select Finish partitioning and write changes to disk and then Yes on Write the changes to disks? and let the installer continue with the base system installation.
  • After a while, the installer will produce a big red ominous message that says:

    [!!] Configure the package manager
      
    apt-configuration problem
    An attempt to configure apt to install additional packages from the CD failed.

    This, however, is actually a completely benign message and you can safely ignore it by selecting Continue . That's because, since we are conducting a net install, we couldn't care less about no longer being to access the "CD-ROM" files after install...
  • Once you have dimissed the message above, pick the mirror closest to your geographical location and let the installer proceed with some more software installation (this time, the software will be picked from that network mirror rather than the media).
    When prompted for the "package usage survey" pick whichever option you like.
  • Finally, at the Software selection screen, select any additional software package you wish to install. Note that the "Debian desktop environment" should work out of the box if you decide to install it (though I have only tested Xfce so far). It's probably a good idea to install at least "SSH server".
  • Let the process finalize the software and GRUB bootloader installation and, provided you didn't screw up your partitioning (i.e. you saw B K ESP when you entered the partition manager, otherwise see Appendix C) select Continue to reboot your machine on the Installation complete prompt.

If everything worked properly, your system will now boot into your brand new vanilla Debian ARM64 system. Enjoy!

Post install fixes


Here are a few things that you might want to fix post install:
  1. You may find a cdrom0 drive on your desktop, which can't seem to be accessible. This is a leftover from the installer process not knowing how to handle the installation media device. You should edit /etc/fstab to remove it.
     
  2. If you installed the cups package, you may get an error while loading modules (systemctl --failed will report that systemd-modules-load.service is in failed state). This is all due to the current cups package trying to load IBM PC kernel modules... on a non PC device. To fix this, simply delete /etc/modules-load.d/cups-filters.conf and reboot.
  3. If using UEFI firmware v1.6 or later, you can enable the serial console by editing /etc/default/grub and changing GRUB_CMDLINE_LINUX="" to GRUB_CMDLINE_LINUX="console=ttyS0,115200", and then running update-grub.
    You may also enable serial console access for GRUB by adding the following in the same file:
    GRUB_TERMINAL=serial
    GRUB_SERIAL_COMMAND="serial --unit=0 --speed=115200 --stop=1" 

Appendix A: How to create and format the SD partition for installation


IMPORTANT NOTE 1: Please make sure to select the disk that matches your SD media before issuing any of these commands. Using the wrong disk will irremediably destroy your data!

IMPORTANT NOTE 2: Do not be tempted to "force" FAT32 in DISKPART or mkfs and do not forget to set the bootable/active flag, else you will afoul of the issue described in Appendix C. 

Windows

C:>diskpart

Microsoft DiskPart version 10.0.18362.1

Copyright (C) Microsoft Corporation.
On computer: ########

DISKPART> list disk

  Disk ###  Status         Size     Free     Dyn  Gpt
  --------  -------------  -------  -------  ---  ---
  Disk 0    Online          238 GB      0 B        *
  Disk 1    Online          465 GB  1024 KB        *
  Disk 4    Online         4657 GB  1024 KB        *
  Disk 5    Online         4649 GB      0 B        *
  Disk 6    Online           14 GB    14 GB

DISKPART> select disk 6

Disk 6 is now the selected disk.

DISKPART> clean

DiskPart succeeded in cleaning the disk.

DISKPART> convert mbr

DiskPart successfully converted the selected disk to MBR format.

DISKPART> create partition primary size=300

DiskPart succeeded in creating the specified partition.

DISKPART> active

DiskPart marked the current partition as active.

DISKPART> format fs=fat quick

  100 percent completed

DiskPart successfully formatted the volume.

DISKPART> exit

Leaving DiskPart...

C:>

Note, if needed you can also force a specific partition type (e.g. set id=0e to force FAT16 LBA), but that shouldn't be needed as DISKPART should set the appropriate type accordingly.

Linux


The following assumes /dev/sdf is your SD/MMC device. Change it in all the commands below to use your actual device.

(Optional) If your drive was partitioned as GPT, or if you're not sure, you may want to issue the two following commands first. If it's MBR you can skip this step:

# Delete the primary GPT:
dd if=/dev/zero of=/dev/sdf bs=512 count=34
# Delete the backup GPT.:
dd if=/dev/zero of=/dev/sdf bs=512 count=34 seek=$((`blockdev --getsz /dev/sdf` - 34))

Now use fdisk and mkfs to partition the drive:

root@debian:~# fdisk /dev/sdf

Welcome to fdisk (util-linux 2.34).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0x7d188929.

Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-31291391, default 2048):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-31291391, default 31291391): +300M

Created a new partition 1 of type 'Linux' and of size 300 MiB.

Command (m for help): t
Selected partition 1
Hex code (type L to list all codes): e
Changed type of partition 'Linux' to 'W95 FAT16 (LBA)'.

Command (m for help): a
Selected partition 1
The bootable flag on partition 1 is enabled now.

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

root@debian:~# mkfs.vfat -F 16 /dev/sdf1
mkfs.fat 4.1 (2017-01-24)
root@debian:~#

Appendix B: Why do we need to use -t vfat -o rw /dev/mmcblk0p1 as the CD-ROM device?

  • Why this weird device name with options? Because these are actually mount command line parameters and the Debian installer actually calls mount behind the scenes and feeds it exactly what we write here. This means we can hijack the device name field to invoke the additional mount parameters we need.
  • Why /dev/mmcblk0p1? That's simply name of the device for the first partition (p1) on the SD/MMC media (mmcblk0) as seen by the Linux kernel on a Raspberry Pi.
  • Why -t vfat? Because the Debian installer appends fstype=iso9660 to the mount option which prevents automount and forces us to override the file system type.
  • Why -o rw? Because the Debian installer won't be able to use the first partition for /boot/efi otherwise or load the WLAN firmware from the media (you get a device or resource busy when trying to remount the media).

Appendix C: Help I screwed up my partitioning!


Of course you did. You thought you knew better, and now you are paying the price...

The problem in  a nutshell is that:
  1. You can't use a regular ESP on a Raspberry Pi, on account that GPT or an MBR partition with type 0xef are not handled by the Broadcom CPU bootloader. And there is nothing you can do about this, because this is a behaviour that's hardcoded in the CPU silicon itself.
     
  2. The Debian installer's partition manager is very temperamental about what it will recognize as an ESP. In other words, if you don't use the perfect combination of boot flag, partition type and file system, it will fail to see it as an ESP.
Now the good news is that this is recoverable, but you need to know what you're doing.
  • The first thing you should do in the Debian partition manager is set the first partition to be used as ESP. In other words, you will need to edit the first partition until you get this:
    MMC/SD card #1 (mmcblk0) - 16.0 GB SD 2WCGO
         #1  primary  314.6 MB  B  K  ESP
             pri/log                  FREE SPACE
  • Then you can proceed as the guide describe, but you need to bear in mind that, as soon as you choose to write the partition changes, the partition manager will have changed your first partition type to 0xef, which, as we have seen is ubootable by the CPU. Therefore, DO NOT PROCEED WITH THE SYSTEM REBOOT AT THE END UNTIL YOU HAVE CHANGED THE PARTITION TYPE BACK.
  • To do that, once you get to the Installation complete prompt that asks you to select Continue to reboot, you need to press Alt-F2 then Enter to activate a console.
  • Then type exactly the following command:
    chroot /target fdisk /dev/mmcblk0
    Then press the keys t, 1, e, w
  • No you can go back to the installer console (Alt-F1) and select Continue to reboot

Appendix D: Installing to a SD from an USB Flash Drive


As I explained above, and though it may seem simpler, I would discourage to use this method to install Debian on a Raspberry Pi. But I can understand that, if you don't have a card reader, you may be constrained to using this method.

For the most part, this should work fine out of the box. As a matter of fact, if you do it this way, you won't have to fiddle with the "CD-ROM" media detection. However, I will now list some of the caveat you'll face if you proceed like:

Caveat 1: If you use guided partitioning your SD/MMC media will be formatted as GPT (because this is a UEFI system after all) which the Broadcom CPU used in the Raspberry Pi can not boot. It has to be MBR. How you are supposed to force MBR over GPT in the Debian partition manager, I'll let you figure out.

Caveat 2: Similarly, you need to go through the 0xef to 0x0e conversion of your ESP, as the Pi won't boot from that partition otherwise.

Caveat 3: Of course you will also need to duplicate all the bootcode.bin, fixup.bat and so on from your USB boot media onto the SD ESP partition if you want it to boot (which is the reason why is is much more convenient to just set the ESP and Debian installer on the SD right of the bat, so you don't risk forgetting to copy a file).

Caveat 4: When I tried USB to SD install, I found that the GRUB installer somehow didn't seem to create an efi/boot/bootaa64.efi, which, if left uncorrected, will prevent the system from booting automatically.

2014-11-18

Applying a series of Debian patches to an original source

Say you have a nice original source package, as well as bunch of extra Debian patches, which you want to apply to that source (for instance, you may want to compile Debian's grub 2.00-22 using the tarballs you picked here.

However, since Debian uses quilt, or whatever it's called, to automate the application of a series of patches, and you either don't have it on your system or don't want to bother with it (since you're only interested in the patches), you end up wanting to apply all the files from the patches directory of the .debian addon, and there's of course no way you'll want to do that manually.

The solution: Copy the patches/ directory from the Debian addon to the root of your orig source, and run the following shell script.

#!/bin/bash
while read p; do
  patch -p1 < ./patches/$p
done < ./patches/series

By Grabthar's hammer, what a timesaver!

2014-11-16

Compiling Grub4DOS on MinGW

Since chenall committed a handful of patches that I submitted to make the compilation of Grub4DOS on MinGW easier, I'm just going to jolt down some quick notes on how you can produce a working Grub4DOS on Windows.
Note that part of this guide is shamelessly copied from the RMPrepUSB Grub4DOS compilation guide.
  • If you don't already have a git client, download and install msys-git (a.k.a. "Git for Windows") from here.
  • Download the latest MinGW32 installer (mingw-get-setup.exe) by clicking the "Download Installer" button on the top right corner of the main MinGW site.
  • Keep the default options on the first screen (but you can change the destination directory if you want)
  • On the package selection screen, select
    • mingw-developer-toolkit
    • mingw-base
    • msys-base
  • Select menu Installation → Apply Changes and click Apply
  • Now navigate to your msys directory, e.g.. C:\MinGW\msys\1.0\, and open the file etc\profile in a text editor.
  • Assuming that you installed msys-git in C:\Program Files (x86)\Git, change the following:
    if [ $MSYSTEM == MINGW32 ]; then
      export PATH=".:/usr/local/bin:/mingw/bin:/bin:$PATH"
    else
      export PATH=".:/usr/local/bin:/bin:/mingw/bin:$PATH"
    fi
    to
    if [ $MSYSTEM == MINGW32 ]; then
      export PATH=".:/usr/local/bin:/mingw/bin:/bin:/c/Program Files (x86)/Git/bin:$PATH"
    else
      export PATH=".:/usr/local/bin:/bin:/mingw/bin:/c/Program Files (x86)/Git/bin:$PATH"
    fi
    This is to ensure that your system will be able to invoke git. Of course, if you use a different git client, you can ignore this step.
  • Download nasm (current build is: http://www.nasm.us/pub/nasm/releasebuilds/2.11.06/win32/nasm-2.11.06-win32.zip) extract and copy nasm.exe to C:\MinGW\msys\1.0\bin (the other files in the zip archive can be discarded).
  • Download upx (current build is: ftp://ftp.heanet.ie/mirrors/sourceforge/u/up/upx/upx/3.91/upx391w.zip) extract and copy upx.exe to C:\MinGW\msys\1.0\bin (the other files in the zip archive can be discarded).
  • In C:\MinGW\msys\1.0\ launch msys.bat
  • In the shell that appears, issue the following command (this may be necessary to locate mingw-get):
    /postintall/pi.sh
    You should accept all the default options.
  • Now issue the following commands:
    mingw-get upgrade gcc=4.6.2-1
    mingw-get install mpc=0.8.1-1
    This will effectively downgrade your compiler to gcc 4.6.2, which is necessary as gcc 4.7 or later doesn't seem to produce a working grldr for the time being.
  • Download the latest Grub4DOS source from github by issuing the following command
    git clone https://github.com/chenall/grub4dos.git
    Note: By default this will download the source into C:\MinGW\msys\1.0\home\<your_user_name>\grub4dos\, but you can of course navigate to a different directory before issuing the git clone command if you want it elsewhere.
  • Run the following commands:
    cd grub4dos
    ./autogen.sh
    make
At the end of all this, you should end up with a grldr and grldr.mbr in the C:\MinGW\msys\1.0\home\<your_user_name>\grub4dos\stage2\ directory, which is what you want

IMPORTANT: Do not try to invoke ./configure directly on MinGW, as compilation will fail. Instead should ensure that you call autotools to re-generate configure and Makefiles that MinGW will be happy with. Note that you can run ./bootstrap.sh instead of ./autogen.sh, if you don't want configure to be invoked with the default options.

What's the deal with gcc 4.7 or later on MinGW?

I haven't really investigated the issue, but the end result is that grldr is 303 KB, vs 307 KB for gcc 4.6.2, and freezes at boot after displaying:
A20 Debug: C806 Done! ...

I'm getting an error about objcopy during the configure test...

That's because you're not listening to what I say and try to compile a version of Grub4DOS that doesn't contain the necessary updates for MinGW. You must use a version of the source that's more recent than 2014.11.14 and right now, that source is only available if you clone from git.

Dude, could you, like, also provide the steps to compile from Linux?

Sigh... Alright, since I'm a nice guy, and it's a lot simpler, I'll give you the steps for a bare Debian 7.7.0 x64 Linux setup:
aptitude install gcc glibc-devel.i686 gcc-multilib make autotools autoconf git nasm upx
git clone https://github.com/chenall/grub4dos.git
cd grub4dos
./autogen.sh
make
Happy now? Note that the Linux compiled version is usually a lot smaller than the MinGW32 compiled one.

2014-05-31

Restoring EFI NVRAM boot entries with rEFInd and Rufus

So, you reinstalled Windows, and it somehow screwed the nice EFI entry you had that booted your meticulously crafted EFI system partition? You know, the one you use with rEFInd or ELILO or whatever, to multiboot Linux, Windows, etc., and that has other goodies such as the EFI shell...

Well, here's how you can sort yourself out (shamelessly adapted from the always awesome and extremely comprehensive Arch Linux documentation):
  • Download the latest rEFInd CD-R image from here.
  • Extract the ISO and use Rufus to create a bootable USB drive. Make sure that, when you create the USB, you have "GPT partition scheme for UEFI computer" selected, under "Partition scheme and target system type".
  • Boot your computer in UEFI mode, and enter the EFI BIOS to select the USB as your boot device
  • On the rEFInd screen select "Start EFI shell".
  • At the 2.0 Shell > prompt type:
    bcfg boot dump

    This should confirm that some of your old entries have been unceremoniously wiped out by Windows.
  • Find the disk on which your old EFI partition resides by issuing something like:
    dir fs0:\efi

    NB: you can use the map command to get a list of all the disks and partitions detected during boot.
  • Once you have the proper fs# information (and provided you want to add an entry that boots into a rEFInd installed on your EFI system partition under EFI\refind\refind_x64.efi), issue something like:
    bcfg boot add 0 fs0:\EFI\refind\refind_x64.efi rEFInd
    Note: If needed you can also use bcfg boot rm # to remove existing entries.
  • Confirm that your entry has been properly installed as the first option, by re-issuing bcfg boot dump. Then remove the USB, reset your machine, and you should find that everything is back to normal.
NOTE: Make sure you use the latest rEFInd if you want an EFI shell that includes bcfg. Not all EFI shells will contain that command!

2014-05-29

Compiling and installing Grub2 for standalone USB boot

The goal here, is to produce the necessary set of files, to be written to an USB Flash Drive using dd (rather than using the Grub installer), so that it will boot through Grub 2.x and be able to process an existing grub.cfg that sits there.

As usual, we start from nothing. I'll also assume that you know nothing about the intricacies of Grub 2 with regards to the creation of a bootable USB, so let me start with a couple of primers:

  1. For a BIOS/USB boot, Grub 2 basically works on the principle of a standard MBR (boot.img), that calls a custom second stage (core.img), which usually sits right after the MBR (sector 1, or 0x200 on the UFD) and which is a flat compressed image containing the Grub 2 kernel plus a user hand-picked set of modules (.mod).
    These modules, which get added to the base kernel, should usually limit themselves to the ones required to access the set of file systems you want Grub to be able to read a config file from and load more individual modules (some of which need to be loaded to parse the config, such as normal.mod or terminal.mod).
    As you may expect, the modules you embed with the Grub kernel and the modules you load from the target filesystem are exactly the same, so you have some choice on whether to add them to the core image or load them from the filesystem.
     
  2. You most certainly do NOT want to use the automated Grub installer in order to boot an UFD. This is because the Grub installer is designed to try to boot the OS it is running from, rather than try to boot a random target in generic fashion. Thus, if you try to follow the myriad of quick Grub 2 guides you'll find floating around, you'll end up nowhere in terms of booting a FAT or NTFS USB Flash Drive, that should be isolated of everything else.
With the above in mind, it's time to get our hands dirty. Today, I'm going to use Linux, because my attempts to try to build the latest Grub 2 using either MinGW32 or cygwin failed miserably (crypto compilation issue for MinGW, Python issue for cygwin on top of the usual CRLF annoyances for shell scripts due to the lack of a .gitattributes). I sure wish I had the time to produce a set of fixes for Grub guys, but right now, that ain't gonna happen ⇒ Linux is is.

First step is to pick up the latest source, and, since we like living on the edge, we'll be using git rather than a release tarball:

git clone git://git.savannah.gnu.org/grub.git

Then, we bootstrap and attempt to configure for the smallest image size possible, by disabling NLS (which I had hoped would remove anything gettext but turns out not to be the case - see below).

cd grub
./autogen.sh
./configure --disable-nls
make -j2

After a few minutes, your compilation should succeed, and you should find that in the grub-core/ directory, you have a boot.img, kernel.img as well as a bunch of modules (.mod).

As explained above, boot.img is really our MBR, so that's good, but we're still missing the bunch of sectors we need to write right after that, that are meant to come from a core.img file.

The reason we don't have a core.img yet is because it is generated dynamically, and we need to tell Grub exactly what modules we want in there, as well as the disk location we want the kernel to look for additional modules and config files. To do just that, we need to use the Grub utility grub-mkimage.

Now that last part (telling grub that it should look at the USB generically and in isolation, and not give a damn about our current OS or disk setup) is what nobody on the Internet seems to have the foggiest clue about, so here goes: We'll want to tell Grub to use BIOS/MBR mode (not UEFI/GPT) and that we'll have one MBR partition on our UFD containing the boot data that's not included in boot.img/core.img and that it may need to proceed. And with BIOS setting our bootable UFD as the first disk (whatever gets booted is usually the first disk BIOS will list), we should tell Grub that our disk target is hd0. Furthermore, the first MBR partition on this drive will be identified as msdos1 (Grub calls MBR-like partitions msdos#, and GPT partitions gpt#, with the index starting at 1, rather than 0 as is the case for disks).

Thus, if we want to tell Grub that it needs to look for the first MBR partition on our bootable UFD device, we must specify (hd0,msdos1) as the root for our target.
With this being sorted, the only hard part remaining is figure out the basic modules we need, so that Grub has the ability to actually identify and read stuff on a partition that may be FAT, NTFS or exFAT. To cut a long story short, you'll need at least biosdisk and part_msdos, and then a module for each type of filesystem you want to be able to access. Hence the complete command:

cd grub-core/
../grub-mkimage -v -O i386-pc -d. -p\(hd0,msdos1\)/boot/grub biosdisk part_msdos fat ntfs exfat -o core.img

NB: If you want to know what the other options are for, just run ../grub-mkimage --help
Obviously, you could go crazy adding more file systems, but the one thing you want to pay attention is the size of core.img. That's because if you want to keep it safe and stay compatible with the largest choice of disk partitioning tools, you sure want to have core.img below 32KB - 512 bytes. The reason is there still exists a bunch of partitioning utilities out there that default to creating their first partition on the second "track" of the disk. And for most modern disks, including flash drives, a track will be exactly 64 sectors. What this all means is, if you don't want to harbour the possibility of overflowing core.img onto your partition data, you really don't want it to be larger than 32256 or 0x7E00 bytes.
OK, so now that we have core.img, it's probably a good idea to create a single partition on our UFD (May I suggest using Rufus to do just that? ;)) and format it to either FAT/FAT32, NTFS or exFAT.

Once this is done, we can flat-copy both the MBR, a.k.a. boot.img, and core.img onto those first sectors. The one thing you want to pay attention to here is, while copying core.img is no sweat, because we can just use a regular 512 byte sector size, for the MBR, you need to make sure that only the first 440 bytes of boot.img are copied, so as not to overwrite the partition data and the disk signature that also resides in the MBR and that has already been filled. So please pay close attention to the bs values below:

dd if=boot.img of=/dev/sdb bs=440 count=1
dd if=core.img of=/dev/sdb bs=512 seek=1 # seek=1 skips the first block (MBR)

Side note: Of course, instead of using plain old dd, one could have used Grub's custom grub-bios-setup like this:

../grub-bios-setup -d. -b ./boot.img -c ./core.img /dev/sdb

However, the whole point of this little post is to figure out a way to add Grub 2 support to Rufus, in which we'll have to do the copying of the img files without being able to rely on external tools. Thus I'd rather demonstrate that a dd copy works just as good as the Grub tool for this.
After having run the above, you may think that all that's left is copying a grub.cfg to /boot/grub/ onto your USB device, and watch the magic happen... but you'll be wrong.

Before you can even think about loading a grub.cfg, and at the very least, Grub MUST have loaded the following modules (which you'll find in your grub-core/ directory and that need to be copied on the target into a /boot/grub/i386-pc/ folder):
  • boot.mod
  • bufio.mod
  • crypto.mod
  • extcmd.mod
  • gettext.mod
  • normal.mod
  • terminal.mod
As to why the heck we still need gettext.mod, when we made sure we disabled NLS, and also why we must have crypto, when most usages of Grub don't care about it, your guess is as good as mine...

Finally, to confirm that everything works, you can add echo.mod to the list above, and create a /boot/grub/grub.cfg on your target with the following:

insmod echo
set timeout=5

menuentry "test" {
    echo "hello"
}

Try it, and you should find that your Grub 2 config is executing at long last, whether your target filesystem in FAT, NTFS or exFAT, and you can now build custom bootable Grub 2 USBs on top of that. Isn't that nice?

FINAL NOTE: In case you're using this to try boot an existing Grub 2 based ISO from USB (say Aros), be mindful that, since we are using the very latest Grub code, there is a chance that the modules from the ISO and the kernel we use in core may have some incompatibility. Especially, you may run into the obnoxious:

error: symbol 'grub_isprint' not found.

What this basically means is that there is a mismatch between your Grub 2 kernel version and Grub 2 module. To fix that you will need to use kernel and modules from the same source.

2013-10-08

Compiling and running your own Android kernel on the Nexus 7 2013

Now at last we're getting to the interesting part: compiling and running our own modified kernel on the Nexus!

We are of course going to pick most of our information from the official kernel building guide, though, unlike the guide, we're going to go all the way through, and not stop before we are actually running our recompiled kernel.

This time, we'll use Linux as our development platform (more convenient). The first thing I need to warn you about then is, if you're running a pure 64 bit distro (such as Slackware x64), you're going to find that you must install a 32-bit compatibility layer to run the 32 bit debug utilities. This is a bit annoying, and I'm not going to guide you through that, so make sure you sort your platform issues, okay?

Installing the toolchain


Anyway, first order of the day is to get the official arm toolchain, which I'm going to install in /usr/local/share/ since I plan to keep using it for some time:

# cd /usr/local/share/
# git clone https://android.googlesource.com/platform/prebuilts/gcc/linux-x86/arm/arm-eabi-4.6
Cloning into 'arm-eabi-4.6'...
remote: Sending approximately 124.64 MiB ...
remote: Counting objects: 33, done
remote: Finding sources: 100% (33/33)
remote: Total 580 (delta 146), reused 580 (delta 146)
Receiving objects: 100% (580/580), 124.64 MiB | 715 KiB/s, done.
Resolving deltas: 100% (146/146), done.

Now, let's add that arm toolchain to our path:

# export PATH=$PATH:/usr/local/share/arm-eabi-4.6/bin
# arm-eabi-gcc --version
arm-eabi-gcc (GCC) 4.6.x-google 20120106 (prerelease)
Copyright (C) 2011 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Ideally, of course, you'd want to add this export into your .profile or something so that you don't have to set it up every time.

Dude, where's the kernel source?


OK, now we want to fetch the kernel source of course, so let's do just that. The one thing you need to know is that the platform name for the Nexus 7 2013 is msm, thus:
# mkdir /usr/src/android
# cd /usr/src/android/
# git clone https://android.googlesource.com/kernel/msm.git kernel
Cloning into 'kernel'...
remote: Sending approximately 850.89 MiB ...
remote: Counting objects: 50642, done
remote: Finding sources: 100% (5223/5223)
remote: Getting sizes: 100% (699/699)
remote: Compressing objects:  99% (11626/11627)
remote: Total 3195022 (delta 2625647), reused 3194640 (delta 2625583)
Receiving objects: 100% (3195022/3195022), 850.15 MiB | 716 KiB/s, done.
Resolving deltas: 100% (2634067/2634067), done.

As can be seen from the above, expect a download of about 850 MB.
# cd kernel
# ls
#

Huh, there's nothing in there?

# git show
commit f688a7654b339885e689d0f95f38b9daa3d85c0f
Author: Jean-Baptiste Queru <jbq@google.com>
Date:   Thu Nov 17 12:44:41 2011 -0800

    empty commit
lines 1-5/5 (END)
# git branch -a
* master
  remotes/origin/HEAD -> origin/master
  remotes/origin/android-msm-2.6.35
  remotes/origin/android-msm-3.9-usb-and-mmc-hacks
  remotes/origin/android-msm-flo-3.4-jb-mr2
  remotes/origin/android-msm-mako-3.4-jb-mr1
  remotes/origin/android-msm-mako-3.4-jb-mr1-fr
  remotes/origin/android-msm-mako-3.4-jb-mr1-kgsl
  remotes/origin/android-msm-mako-3.4-jb-mr1.1
  remotes/origin/android-msm-mako-3.4-jb-mr2
  remotes/origin/android-msm-sony-cm-jb-3.0
  remotes/origin/master

Mmmm, OK, maybe the flo branch then?

# git checkout android-msm-flo-3.4-jb-mr2
Checking out files: 100% (41678/41678), done.
Branch android-msm-flo-3.4-jb-mr2 set up to track remote branch android-msm-flo-3.4-jb-mr2 from origin.
Switched to a new branch 'android-msm-flo-3.4-jb-mr2'
root@stella:/usr/src/android/kernel# git show
commit 9e52a2195889f4f03ddfcaefb19c0b42ec1d9070
Author: Praveen Chavan <pchavan@codeaurora.org>
Date:   Fri Mar 29 17:28:31 2013 -0700

    qseecom: Fix issue with incomplete command exiting prematurely

    A process waiting on a signal can be awaken by any signal. We
    need to only continue processing when the condition of the
    wait event is met.

    Change-Id: Ib2102babbb505876f89b04399729e6ff5a475605
    Signed-off-by: Mona Hossain <mhossain@codeaurora.org>
    Signed-off-by: Praveen Chavan <pchavan codeaurora.org>

Let me spare you the suspense there: As "documented" on their page, Google don't actually use branches or even tags for their development. Instead they force you to use an sha1 reference that's not attached to any helpful entity, in order to figure out the branch you should pick. Moreover, what they advise you to do to get that SHA is go for another large download of pointless binary data (650 MB) containing all the revision of their kernel files. Well, not everybody's running on Google Fibre... yet, and my ISP also enforces quotas, so, since we're smarter than this, we're going to avoid downloading 650 MB in exchange of a few SHA bytes. Instead, we'll head directly to https://android.googlesource.com/device/asus/flo-kernel and look at the most recent "flo: prebuilt kernel", which, at the time of this post, is:

79816e3  flo: prebuilt kernel by Jean-Baptiste Queru - 2 months ago jb-mr2-dev master tools_r22.2 android-4.3_r2.1

If you click on the link, you'll see part of the SHA we are really after here:

365a6e0 gpu: ion: Minimize allocation fallback delay

That 365a6e0 is the ID we're seeking, so let's use that (and also reuse the official kernel name, 'android-4.3_r2.1', for our branch name):

# git checkout -b android-4.3_r2.1 365a6e0
Switched to a new branch 'android-4.3_r2.1'
# ls
AndroidKernel.mk  CREDITS         Kbuild   MAINTAINERS  README          arch/   crypto/   firmware/  include/  ipc/     lib/                mm/   samples/  security/  tools/  virt/
COPYING           Documentation/  Kconfig  Makefile     REPORTING-BUGS  block/  drivers/  fs/        init/     kernel/  make_defconfig.sh*  net/  scripts/  sound/     usr/

That's more like it!

NB: We could be even smarter by just looking at the kernel revision provided by our device (3.4.0-g365a6e0), drop the whole part before the g and recognize that this is the partial SHA1 we are after. Then again, since there's a chance a new kernel was out and you're running an old or custom one...

Compiling the Android kernel


Now we need to set the variables that will tell the OS that we're cross compiling (again, something that you may want to do in your .profile using something like alias cross='export ARCH="arm";export SUBARCH="arm";export CROSS_COMPILE="arm-eabi-"'):

# export ARCH=arm
# export SUBARCH=arm
# export CROSS_COMPILE=arm-eabi-

Finally, we can initialize our default config and get going:

# make flo_defconfig
  HOSTCC  scripts/basic/fixdep
  HOSTCC  scripts/kconfig/conf.o
  SHIPPED scripts/kconfig/zconf.tab.c
  SHIPPED scripts/kconfig/zconf.lex.c
  SHIPPED scripts/kconfig/zconf.hash.c
  HOSTCC  scripts/kconfig/zconf.tab.o
  HOSTLD  scripts/kconfig/conf
warning: (ARCH_MSM_KRAITMP && ARCH_MSM_CORTEX_A5) selects HAVE_HW_BRKPT_RESERVED_RW_ACCESS which has unmet direct dependencies (HAVE_HW_BREAKPOINT)
warning: (ARCH_MSM_KRAITMP && ARCH_MSM_CORTEX_A5) selects HAVE_HW_BRKPT_RESERVED_RW_ACCESS which has unmet direct dependencies (HAVE_HW_BREAKPOINT)
# make menuconfig
  HOSTCC  scripts/kconfig/lxdialog/checklist.o
  HOSTCC  scripts/kconfig/lxdialog/inputbox.o
  HOSTCC  scripts/kconfig/lxdialog/menubox.o
  HOSTCC  scripts/kconfig/lxdialog/textbox.o
  HOSTCC  scripts/kconfig/lxdialog/util.o
  HOSTCC  scripts/kconfig/lxdialog/yesno.o
  HOSTCC  scripts/kconfig/mconf.o
  HOSTLD  scripts/kconfig/mconf
scripts/kconfig/mconf Kconfig
warning: (ARCH_MSM_KRAITMP && ARCH_MSM_CORTEX_A5) selects HAVE_HW_BRKPT_RESERVED_RW_ACCESS which has unmet direct dependencies (HAVE_HW_BREAKPOINT)
warning: (ARCH_MSM_KRAITMP && ARCH_MSM_CORTEX_A5) selects HAVE_HW_BRKPT_RESERVED_RW_ACCESS which has unmet direct dependencies (HAVE_HW_BREAKPOINT)

Now that we're in the kernel config, let's start with something real basic, and change General setup ---> Local version - append to kernel release setting it to something like "hello". Hopefully, we'll be able to boot our recompiled kernel and see that string appear in the system info, to confirm that we are indeed using our own kernel. Off we go with building the whole thing then:

# make -j8
(...)
  OBJCOPY arch/arm/boot/zImage
  Kernel: arch/arm/boot/zImage is ready

We have a kernel — now what?


Aaaaand, that's pretty much where the Google guide stops. What do you mean, you also want to run your newly compiled kernel on your device?

If that's the case, then you should download more than 16 GB worth of build files from the infamous aosp (Android Open Source Project) suite —I wish I was kidding, see below— and then figure out how to inject your kernel in there. Good luck!

My usage data (in MB) after downloading the full aosp.
(Careful analysis may be required to notice the small difference in data usage)

The 16 GB of the full aosp (I was already 2 GB in for that day, hence the 18 GB above) sure put me dangerously close of getting a not so friendly call from my ISP for going over the monthly quota (again, not everybody is running on Google Fibre), so I might try to spare you this trouble.

If you're not going to download the aosp to try to figure out this mess, the first thing you should know then is that, as opposed to what is the case on most devices, testing your kernel on Android isn't as simple as pointing your bootloader to it. It must be included into a custom boot image (boot.img) along with a RAM disk and some other stuff.

Crafting an Android boot.img


The format of this Android boot images can be found in the bootimg.h of the mkbootimg tool from the android platform/system/core source (at least up to Android 10). I guess we have to start somewhere, and we'll need to produce our own images, so let's start by fetching and recompiling mkbootimg then.

NB: Since I duplicated and improved on the original tools (see below), you can, and probably should, use https://github.com/pbatard/bootimg-tools.git instead of https://android.googlesource.com/platform/system/core for the git clone URL below.

# git clone https://android.googlesource.com/platform/system/core bootimg-tools
Cloning into 'bootimg-tools'...
remote: Counting objects: 92, done
remote: Finding sources: 100% (92/92)
remote: Total 19302 (delta 11674), reused 19302 (delta 11674)
Receiving objects: 100% (19302/19302), 5.87 MiB | 655 KiB/s, done.
Resolving deltas: 100% (11674/11674), done.
# cd bootimg-tools/libmincrypt/
# gcc -c *.c -I../include
# ar rcs libmincrypt.a *.o
# cd ../mkbootimg
# gcc mkbootimg.c -o mkbootimg -I../include ../libmincrypt/libmincrypt.a
mkbootimg.c: In function 'main':
mkbootimg.c:245:9: warning: assignment discards 'const' qualifier from pointer target type [enabled by default]
# cp mkbootimg /usr/local/bin/
# cd ../cpio
# gcc mkbootfs.c  -o mkbootfs -I../include
# cp mkbootfs /usr/local/bin/

With the above, we'll be able to invoke mkbootimg and optionally mkbootfs to create an image. If you follow this guide closely, you'll see that we're not even actually going to use mkbootfs, but I kept it there just in case.

Now, we don't want to recreate our RAM disk from scratch, or have to guess all the parameters we'll need to use to create one, so let's pick a working one. For this you have two choices:
  • Use a backup utility on your rooted Android device, and save the partition that contains the boot image. For instance, if you installed TWRP when rooting your device, and boot into recovery, then you will be able to save the 16MB Boot partition, which is what we are after.
  • Download the latest factory image (at the time of this writing, it's JSS15R, 359MB big), extract it in full (tar then zip), and then pick the boot.img image from the zip (in JSS15R, that would be boot.img within image-razor-jss15r.zip). You also want to be mindful that bootloader-flo-flo-03.14.img is the bootloader image, and NOT what we are after.
Also, don't let the fact that in one case you get a 16MB file and in the other one that is less than 7MB worry you. The 16MB one is just padded with zeroes, which won't matter when we access its content.
For the remainder of this guide, even as it requires a 359 MB download, I'll go with using boot.img from the factory image.

Next thing we need is a way to unpack that boot.img file. There are a couple of unpack-bootimg.pl/split-bootimg.pl perl scripts dating back from 2008, so we can pick one of those and then use a binary editor take a look at the .img and extract the addresses and parameters we should provide mkbootimg, and try to cross-reference those with... Aaaaagh, I can't stand it any more!!!

That's "Aaaaagh!" with five A's


What the £$%^& is wrong with Android developers?!? Why the heck aren't these unpacker scripts providing you with the full set of parameters you need to use to repack an image, instead of having to look at a binary header, or some mysterious settings file in a 16GB repository to find them? Also, why, in 5 years, hasn't anyone, including Google, improved these tools and scripts? The thing is, even mkbootimg has issues and won't work on Windows (when compiled with MinGW) due to the use of non-Windows friendly open(), read(), write() et cetera.

That does it!

I'm gonna write my own unpack tool in C, fix mkbootimg in the process, and put it all on github.
Sheesh, as if I didn't have better things to do... I hope you're happy!

So off we go to the directory you compiled mkbootimg (NB: I you cloned from https://github.com/pbatard/bootimg-tools.git earlier, you can skip the wget line):

# cd /usr/src/android/bootimg-tools/mkbootimg/
# wget https://raw.github.com/pbatard/bootimg-tools/master/mkbootimg/unmkbootimg.c
# gcc -o unmkbootimg unmkbootimg.c
# cp unmkbootimg /usr/local/bin/

Now at last, you have the minimum of unmkbootimg, mkbootimg and mkbootfs installed in your path, and we can get going with our testing of the kernel.
I'll assume that you have a /usr/src/android/boot where you copied your boot.img, so let's get cracking:

# cd /usr/src/android/boot/
# ls
boot.img
# unmkbootimg -i boot.img
kernel written to 'kernel' (6640200 bytes)
ramdisk written to 'ramdisk.cpio.gz' (399979 bytes)

To rebuild this boot image, you can use the command:
  mkbootimg --base 0 --pagesize 2048 --kernel_offset 0x80208000 --ramdisk_offset 0x82200000 --second_offset 0x81100000 --tags_offset 0x80200100 --cmdline 'console=ttyHSL0,115200,n8 androidboot.hardware=flo user_debug=31 msm_rtb.filter=0x3F ehci-hcd.park=3' --kernel kernel --ramdisk ramdisk.cpio.gz -o boot.img
# ls
boot.img  kernel  ramdisk.cpio.gz

As an aside that you don't need to run, but since it should be elementary that this is the basic functionality you want from a proper boot image unpack tool, we can confirm that the data provided by the unpack tool will produce a boot.img that is binary identical to the original one:
# mkbootimg --base 0 --pagesize 2048 --kernel_offset 0x80208000 --ramdisk_offset 0x82200000 --second_offset 0x81100000 --tags_offset 0x80200100 --cmdline 'console=ttyHSL0,115200,n8 androidboot.hardware=flo user_debug=31 msm_rtb.filter=0x3F ehci-hcd.park=3' --kernel kernel --ramdisk ramdisk.cpio.gz -o myboot.img
# ls
boot.img  kernel  myboot.img  ramdisk.cpio.gz
# cmp -l boot.img myboot.img
#

Now, THIS is what you want to be able to do from an Android image unpacking tool. Is it really that much to ask?

Crafting an Android boot.img 2: 'Electric boogaloo'


Moving on. Since we just want to test a kernel, we shouldn't really have to touch the cpio image (ramdisk), but then again, my goal here is to give you as many pointers as I can, so we might as well see how we craft our own ramdisk while we're at it. What we're going to do here, as an academic exercise, is add an it_works file at the root of the filesystem, which we'll look for after we booted, to confirm that we are able to use our modified stuff all the way through:

# mkdir ramdisk
# cd ramdisk
# gunzip -c ../ramdisk.cpio.gz | cpio -iu
# ls
charger*       init*             proc/              sys/
data/          init.flo.rc*      property_contexts  system/
default.prop   init.flo.usb.rc*  res/               ueventd.flo.rc
dev/           init.rc*          sbin/              ueventd.rc
file_contexts  init.trace.rc*    seapp_contexts
fstab.flo      init.usb.rc*      sepolicy
# touch it_works
# ls
charger*       init*             it_works           sepolicy
data/          init.flo.rc*      proc/              sys/
default.prop   init.flo.usb.rc*  property_contexts  system/
dev/           init.rc*          res/               ueventd.flo.rc
file_contexts  init.trace.rc*    sbin/              ueventd.rc
fstab.flo      init.usb.rc*      seapp_contexts
# find . | cpio -o -H newc | gzip > ../myramdisk.cpio.gz
1410 blocks
# cd ..
# ls
boot.img  kernel  myramdisk.cpio.gz  ramdisk/  ramdisk.cpio.gz

We're finally set for the last part, where we copy the kernel we compiled earlier, and invoke mkbootimg with the set of parameters we got from unmkbootimg, and use both our modified kernel and cpio image:

# cp /usr/src/android/kernel/arch/arm/boot/zImage .
# ls
boot.img  kernel  myramdisk.cpio.gz  ramdisk/  ramdisk.cpio.gz  zImage*
# mkbootimg --base 0 --pagesize 2048 --kernel_offset 0x80208000 --ramdisk_offset 0x82200000 --second_offset 0x81100000 --tags_offset 0x80200100 --cmdline 'console=ttyHSL0,115200,n8 androidboot.hardware=flo user_debug=31 msm_rtb.filter=0x3F ehci-hcd.park=3' --kernel zImage --ramdisk myramdisk.cpio.gz -o myboot.img
# ls
boot.img  kernel  myboot.img  myramdisk.cpio.gz  ramdisk/  ramdisk.cpio.gz  zImage*

Finally, a custom boot.img we can test. Let's press on by copying this myboot.img file into the directory we have adb and fastboot installed (see our earlier post about rooting the Nexus 7) and run the following set of commands which, unlike what many other guides seem to advise (what the heck, guys?), is NOT going to flash the kernel/boot.img but simply run it from memory. This means that, in case there's any mishap, you can simply power the Nexus off and you'll be good as new:

# ./adb start-server
* daemon not running. starting it now on port 5037 *
* daemon started successfully *
# ./adb reboot bootloader
# ./fastboot boot myboot.img
downloading 'boot.img'...
OKAY [  0.223s]
booting...
OKAY [  0.023s]
finished. total time: 0.246s

The moment of truth!

After a long few seconds of anxiety, where the device doesn't seem to do anything, you should see either one of these two things:
  • The Nexus' multicoloured 'X', telling you that your boot.img file was fine and that the tablet is now booting it, or
  • The Google logo with the open lock, indicating that the device didn't accept your boot image and is restarting with the one it has in flash (or complete freezout, if which case you should press the power button until your device turns off)
If you've seen the multicoloured 'X' straight away, then there's a good chance that, if you go to your tablet settings, you'll see the following reported:


A few things you should recognize above:
  • That hello appended to the 3.4.0 kernel version, that's the "local version" string we chose to add to our kernel when we recompiled
  • root@stella as the user who compiled this kernel (yeah, I tend to compile stuff as root - so sue me!)
  • That 365a6e0 after the g (for 'git') in the version number, that's the SHA1 from the git commit we had to pick to compile our kernel.
All in all, this looks fairly good. And a quick look to the root filesystem (through adb shell or through ssh) will also confirm that our it_works file is there, so we can also add whatever we want on the initial filesystem. Neat!

From there on, you can go ahead and tweak your kernel and initial filesystem exactly as you see fit. And after you have tested that everything works as it should, you can go ahead and flash the boot partition with your shiny custom boot.img, using:

fastboot flash boot boot.img

For the record, if you actually modify the source of your kernel, you should see a -dirty qualifier appended to your kernel version, which is normal, and won't prevent you to run a modified kernel.

Now what are you doing still reading this guide: get hacking your device already!

2013-09-30

Rooting your Nexus 7 (2013) for the Android virgin...

...but Linux experienced.

Having recently purchased a Nexus 7 32GB (2013 model) as my very first Android mobile device, I am finding that the information on how to gain complete control is both all over the place and supposes you're already familiar with the whole rooting process, which I'm not. Therefore, I'm going to write yet another step by step guide on what you need to do, to be able to remotely ssh into it and run a shell as root, which, IMO, is the one stage where we can pretend that we are actually running a GNU/Linux system under the hood and get things done.

Incidentally, if you don't care about commandline Linux, this process will also grant you full control and let you do whatever you please with the device.

Being a complete Android noob, I will also make sure not to skim over all the "Oh, but you're supposed to know that already!" details, so that you will hopefully be as successful as I was, in rooting your device. On the other hand, I will assume that you are somewhat familiar with Linux. But don't let that worry you as the Linux specific items will come at the very end, at which stage your device will already be fully rooted.

Finally, since it's somewhat convenient, I will be using Windows as the platform we'll use to communicate with the Nexus 7 during this process, and I'll assume that you are familiar enough with it to find your way around the Device Manager, manual driver installation and the Windows command line.

Why *I* want to root my device


I'm not going to venture a guess as to what your reasons are for rooting your device. But one of the first reasons I have, in a very long list, is to be able to edit the device's /etc/hosts and get rid of most of the ads and unwanted trackers, which every other device and computer that I have doesn't have to put up with, through a skilful use of the ever so awesome WinHelp 2002 host file. And yeah, I know there are other ways to block ads, but the /etc/hosts method is still the simplest, most universal and effective way, since it'll block ads for all apps running on your system, rather than the ones in a single browser application.

Now, the ironic part is: the reason I purchased a Nexus 7 in the first place is through revenue that I got from placing ads on a successful little web site (Hurray, for unexpected disposable income!).
Am I really going to bite the hand that feeds me? Watch me!

Besides eliminating ads, there are of course plenty of other reasons to want to root your device, such as recompiling and running your own custom kernel (we'll come to that in a next post), but we might as well start small, by improving our mobile surfing experience.

What the other guides omit to tell you...


For the most part, I am going to follow the video guides at xda-developers and most specifically, the third and second to last from this link. Oh, and if you want a less verbose, yet noob friendly and up to the point guide, I can only recommend the "How to build Cyanogen for Nexus 7 2013" one, as it has a very easy to follow section on rooting. Still, part of these guides suppose some familiarity with the process and omit some important details that you may be interested in knowing beforehand, such as:

The internal name of the Nexus 7 2013 platform is 'flo'

This might be relevant if you are flashing an Android image file that came from a website that isn't dedicated exclusively to the Nexus 7 2013. If you do that, you may want to check that 'flo' is mentioned somewhere in what you're looking at. Alternatively, 'msm', which is the development branch for the 'flo' platform might be a good indicator. Remembering these two can be a good way of avoiding looking at data that is only relevant for the "old" Nexus 7, or other incompatible Android hardware.

You are going to lose all your data!

More specifically, as soon as you unlock the device, which is pretty much the first step invoked in rooting, your device will be completely reset to factory settings.
In the notice they display when you do that (see screenshots below) Google say that this is to prevent someone who got unwarranted access to your phone from accessing your personal data (which is probably only a half truth, but hey...).
My advice then is: if your device is new, you want to root it as soon as possible, so that you don't have to go through the pain of restoring your apps, settings and personal data.

Then again, if you have let some time pass and installed a few things, provided you did set the device to use your google account for backup, a lot of it will be restored automatically as you re-indentify yourself after unlock. Especially, all the apps you have installed, whether free or not, will be re-downloaded and re-installed. Also, if you're lucky, some of the customization and personal data files associated with them may be restored... though you shouldn't really count on that and take your precautions.

But really, the less customized your device is before you root it, the better. Before attempting this procedure then, you probably want to back up as much as you can... though of course, it takes a rooted device to be able to perform a comprehensive backup of your data! As if we needed more reasons to want to root our device early.

As I assume that you are as new to the process as I was, here is a quick overview of the steps we are going to follow:
  1. Enable Developer access on your device - this will allow to connect your Nexus 7, through USB, to the Android debug console as well as the boot/flash utility. We'll need this to unlock the device and flash the custom (non Google) recovery image, that'll help with installing root access
  2. Install the Developer tools on your PC, so that you can communicate with your Nexus in developer mode.
  3. Unlock the stock bootloader, so that it will let you flash a non-stock recovery image. This is also the process that resets your device and data to factory settings.
  4. Install (flash) a custom recovery image - This will allow the installation of the binary that will supervise the granting of root access to applications (SuperSU).
  5. Boot into that recovery image and install (flash) the SuperSU system binaries and deamon.
  6. Reboot the device into standard operation mode, and install the SuperSU app. (SuperSU comes in two parts: A regular "client" app, that can be downloaded from the App Store, and a system executable part that needs to be installed separately into the flash filesystem).
  7. Install Busybox
  8. Install an SSH server
  9. Customize our GNU/Linux system, including /etc/hosts.

Enabling Developer access


So you've backed up whatever you could and accepted the idea that your Nexus will be restored to the clean slate state you found it right after you purchased it. Good.

The first thing we need then is enable developer access, which is disabled by default. To do that, you need to navigate to "Settings" → "About Tablet" (at the very end) and tap the "Build number" section 7 times. Why 7 times? Who knows...

Once you do that, you will get a small prompt indicating "You are now a developer" and after that has occurred, if you go back into the Settings menu, you will see a new "{ } Developers options" category. There, you will find a bunch of additional parameters that you can modify, that may come handy if you do development, but the only one we need right now, under the "DEBUGGING" section, is "USB debugging". So please go ahead and tap that one.

As soon as you do that, you will receive a warning about "USB debugging being intended for development purposes only", which you should accept. This should lead to the following checkmark being displayed:


Installing the USB driver and developers tools


Unfortunately, even as it is WinUSB based, the current version of Android is not WCID enabled (but I'm working on that!), so that means you need to manually install some drivers, to be able to communicate with the device in debug mode.

You should therefore head to http://developer.android.com/sdk/win-usb.html click the "Download Google USB Driver" button, then pretend you accept whatever agreement they want you to agree by not reading it (so that you can feign complete ignorance of the terms if you break them, which I hear is how EULAs work these days) and unzip the files into an usb_driver\ directory somewhere.

Now, you should connect your device to an USB port on your Windows machine. Windows will try, and fail, to locate a driver for the new debugging USB interface, at which stage you should select the "Nexus 7" unrecognised device in Device Manager, select install driver, and point to the usb_driver\ directory you previously extracted.

Once you have done that, you should see something very similar to this:


All that is left, with regards to the developers tool installation, is to go to http://developer.android.com/sdk/index.html and Download the SDK ADT Bundle for Windows (about 450 MB in size). Depending on whether you picked the 32 or 64 bit version, you should end up with a zip file such as adt-bundle-windows-x86_64-20130917.zip which you can then extract somewhere like D:\Android\adt-bundle-windows-x86_64-20130917\.

Unlocking the bootloader


The only thing we are really interested in package above are the adb.exe (Android Debug Bridge - The application that communicates with the Android OS in debug mode) and fastboot.exe (The application that communicates with the bootloader) and that are located in the sdk\platform-tools\ directory. That's right; we downloaded more than 450 MB worth of junk just to access a couple of executables that don't even occupy 1 MB worth of space... talk about bandwidth wastage! (if you are concerned about this, you may want to look for smaller unofficial packages containing only adb and fastboot, or recompiling these from git)

Next, you should open a command prompt into your sdk\platform-tools\ directory and issue:

adb start-server

The command window will display the following:

D:\Android\adt-bundle-windows-x86_64-20130917\sdk\platform-tools>adb start-server
* daemon not running. starting it now on port 5037 *
* daemon started successfully *

D:\Android\adt-bundle-windows-x86_64-20130917\sdk\platform-tools> 

And you will also receive a request similar to this one on your device, which you should accept:


Once you have accepted the request, you will get a short message on the top status bar, indicating "USB debugging connected". Then a persistent USB debug icon, similar to the one displayed on the left, will appear in the top left corner.

The next command we want to enter, in our command prompt, is:

adb reboot bootloader

Your Nexus will now reboot in bootloader mode and once that is done, you should see a screen similar to this one, indicating "LOCK STATE - locked":


Note 1: If needed, you can use Volume Up/Volume Down to select the option highlighted at the top, and press  Power to execute that option.

Note 2: You can also enter the bootloader without using the adb command, by powering your table off, and then pressing Volume Down while powering it up.

Of course, we're going to change that LOCK STATE. To do that, and since we are in bootloader mode, we can use the fastboot command:

fastboot oem unlock

You will get a warning similar to this:


Use Volume Up/Down to select "Yes" and press the Power button. It should return to bootloader menu, but this time you should see "LOCK STATE - unlocked":


Now, you can now reboot your device by issuing the following:

fastboot reboot

Note: Because you unlocked the bootloader, the device will now reset to its factory settings. This is the part where you lose ALL YOUR DATA AND CONFIGURATION.

Installing the TWRP custom recovery image


Make sure your device boots properly, and recreate your account.

You probably also want to take some time to let the backup process reinstall your apps, as well as manually download and reconfigure whatever personal settings and files you did backup, before you unlocked the bootloader. The rest of our operations here should not affect this data.

While your Nexus is being restored, you also want to download the following 2 files:

  • The latest TeamWin Open Recovery image from http://techerrata.com/browse/twrp2/flo, which you should save into your sdk\platform-tools folder\
  • The latest SuperSU zip, from http://download.chainfire.eu/supersu, which you should save, on your Nexus device, in the "Downloads" folder, without extracting it (i.e. as a zip file). This file will be named something like "UPDATE-SuperSU-vX.YZ.zip".
For safety, once you have restored apps, settings and files, you should, issue a complete poweroff. Then boot it back again, confirm that you have USB debugging running (The small Android icon at the top left corner) and, in the command prompt Window issue:

adb reboot bootloader

Now that we're back in the bootloader, we are going to replace the stock recovery image, by flashing the TWRP we just downloaded over it. Provided the TWRP file you downloaded above is named "recovery openrecovery-twrp-2.6.3.0-flo.img", the command you will need to issue:

fastboot flash recovery openrecovery-twrp-2.6.3.0-flo.img

Now you need to reboot in recovery mode. You can do that by using Volume Up/Down until "Recovery Mode" is displayed at the top, and then pressing Power.

You should be greeted with the following screens:



Once you see the above, tap "Install" and select the "UPDATE-SuperSU-vX.YZ.zip" you should have in your "Download" folder:


You should then see the following:



At this stage, you should proceed to swipe, to install the files contained in the zip into the OS partition. What this process actually does is copy two executables su and chattr (Change Attribute) in your /system/xbin/ directory, as well as install a daemon that, will be launched at boottime.

You can then proceed to reboot the device, by selecting "Reboot" on the initial screen and then "System".

Installing BusyBox


Now that we have the system part of SuperSU installed, we can go to the Play Store and install the SuperSU client app, so just go ahead and do that.

Congratulations! At this stage, you have supposedly completed the hardest part, and your device is effectively rooted.

First thing you probably want to do then, is confirm that this is actually the case by installing "Root Checker" from the Play Store. If you launch it and click on the "Verify Root Access" you should get a prompt such as this one:


This is a SuperSU prompt, one that you should become exceedingly familiar with as you install more apps that request root access. When you see it, you should tap on "Grant", and after that, Root Checker should confirm that you have root access indeed.

Alrighty, moving on to our next target. We're of course not going to be sitting around installing rooted apps all day. What we want is an actual Linux shell, that we can access remotely. Thus, the first application we want to install is Busybox, that provides a basic set of base Linux utilities, as well as a basic shell.

On to the Play Store it is. If you search for BusyBox, you'll find a bunch of 'em. The one I installed was the free version from Stephen Ericson (Stericson) that will probably be the first result you'll see.

After it's installed, you should run the app, let it scan your system for a bit, and then go for the Smart Install. In the process, you will be greeted by another "Superuser request" prompt from SuperSU, which you should of course grant.

Once we've done that, and to confirm that BusyBox is properly installed, we're can to go back to our Windows prompt and issue:

adb shell

This opens an interactive shell on our Linux box (with a terminal that sucks, but we're not going to worry about that). You can then confirm that BusyBox was properly installed by issuing the command busybox. The result you get should be similar to this:

shell@flo:/ $ busybox
busybox
BusyBox v1.21.1-Stericson (2013-07-08 15:58:11 BST) multi-call binary.
BusyBox is copyrighted by many authors between 1998-2012.
Licensed under GPLv2. See source distribution for detailed
copyright notices.

Usage: busybox [function [arguments]...]
   or: busybox --list[-full]
   or: busybox --install [-s] [DIR]
   or: function [arguments]...

        BusyBox is a multi-call binary that combines many common Unix
        utilities into a single executable.  Most people will create a
        link to busybox for each function they wish to use and BusyBox
        will act like whatever it was invoked as.

Currently defined functions:
        [, [[, ash, awk, base64, basename, blkid, bunzip2, bzcat, bzip2, cal,
        cat, chat, chattr, chgrp, chmod, chown, chroot, chrt, cksum, clear,
        comm, cp, crond, crontab, cut, date, dd, depmod, devmem, df, diff,
        dirname, dmesg, dnsd, dnsdomainname, dos2unix, du, echo, egrep, env,
        ether-wake, expand, expr, fakeidentd, fdflush, fdformat, fdisk, fgrep,
        find, fold, free, freeramdisk, fsck, fsync, ftpd, ftpget, ftpput,
        fuser, getopt, grep, groups, gunzip, gzip, hd, head, hexdump, hostid,
        hostname, httpd, hwclock, id, ifconfig, ifenslave, inetd, inotifyd,
        insmod, install, ionice, iostat, ip, ipaddr, ipcalc, iplink, iproute,
        iprule, iptunnel, kill, killall, killall5, less, ln, logname, losetup,
        ls, lsattr, lsmod, lsof, lsusb, lzop, lzopcat, md5sum, microcom, mkdir,
        mkdosfs, mke2fs, mkfifo, mkfs.ext2, mkfs.vfat, mknod, mkswap, modinfo,
        modprobe, more, mount, mountpoint, mt, mv, nameif, nanddump, nandwrite,
        nc, netstat, nice, nmeter, nslookup, ntpd, od, patch, pgrep, pidof,
        ping, ping6, pkill, pmap, powertop, printenv, printf, ps, pscan, pwd,
        rdate, rdev, readahead, readlink, realpath, renice, reset, rev, rfkill,
        rm, rmdir, rmmod, route, run-parts, script, scriptreplay, sed, seq,
        setkeycodes, setlogcons, setsid, sha1sum, sha256sum, sha3sum,
        sha512sum, showkey, sleep, smemcap, sort, split, start-stop-daemon,
        stat, strings, stty, sum, swapoff, swapon, sync, sysctl, tac, tail,
        tar, tee, telnet, telnetd, test, tftp, tftpd, time, timeout, top,
        touch, tr, traceroute, traceroute6, tty, ttysize, tunctl, umount,
        uname, uncompress, unexpand, uniq, unix2dos, unlzop, unzip, uptime,
        usleep, uudecode, uuencode, vconfig, vi, watch, wc, wget, which, who,
        whoami, whois, xargs, zcat

shell@flo:/ $

Looks good enough. Now you can type exit to exit the shell and go back to the Windows command prompt.

And since we're done with the Windows command, you may want to issue:

adb kill-server

So that you can later disconnect your device without Windows complaining about it.

Installing an SSH server


At last, we've come to the final piece of our puzzle! Surprisingly, if you visit the Play Store, you'll find that there aren't that many SSH servers on offer.

The one I picked then is "Rooted SSH/SFTP Daemon", which looks a bit basic but does its job very well.

Once you launch it, tap the gear icon in the middle of the screen, and change at least the Login ID (I used 'root') and Password. You probably also want to enable "Wifi Lock" so that the SSH server will keep running in the background as you switch apps.


Now you want to press the large ON/OFF button so that it is highlighted red and so that our server is running. You should also get a small grey round ON/OFF icon in the top left area (see previous or next screenshot). Oh, and you'll probably have gotten another SuperSU grant request at this stage.

"Well, I seem to have an SSH server running alright, you say, but how do I find the name or IP address I should use to connect to it?"

The simplest method, without having to install another app or check your Wifi router, is to swipe down the Wifi icon in the top right corner and tap the Wifi properties.
Oh yeah, did you know that, if you swipe down any of the small status icons that appear at the top, you can quickly access the application and settings behind them? That's a good trick to know, and I sure didn't.

In this case, you'll be able to tap your current Wifi connection and get the IP address you need to use for SSH:



Give a go to that IP address in an ssh client, such as putty, and lo and behold, you should be connected to your Nexus 2013 Android device under a nice shiny shell. That's more like it!

Customizing our GNU/Linux Android system


Let's now push things to their logical conclusion, by performing 2 last items:
  1. As you may have noticed if you scan your network, and if you let the DHCP client provide its desired hostname, your Nexus 7 will be identified as 'android_12345678' or something. Ugh, let's change that!
  2. Edit that /etc/hosts to filter ads, as we said we'd do.
With regards to item 1, a quick search tells us that the file we want to edit is /system/build.prop, where you want to add a net.hostname=my_name line. So you go ahead and try that, only to find that the file is read-only?!?
What gives? Weren't we supposed to be root here?

Well, that's Android security for ya (which may or may not be that bad a thing). By default, only the directories under /sdcard (such as "Download" and stuff) are read/write. The rest of the system is read-only.

Fixing this is easy though. If you want to modify system files then, you just need to issue the following to be granted full read/write access:

mount -o rw,remount -t rootfs /system

Much better!

Now you can edit your /system/build.prop to your hearts content and change the DHCP hostname.

Let's finish the job by going ads busting then. Still from the ssh shell:

cd /sdcard/Download
wget http://winhelp2002.mvps.org/hosts.zip
unzip hosts.zip
mv HOSTS hosts
dos2unix hosts
cp hosts /etc/

The last line will overwrite the existing /etc/hosts, which is fine since it only contains one entry to alias 127.0.0.1 to localhost (which is also present in the new file). Oh, and the dos2unix is necessary as the original HOSTS file was saved in DOS mode.

Do that, try to surf, and watch in awe as the ads pollution has now gone away...

I think my work here is done...