2013-10-03

Nexus 7 (2013) - The good, the bad and the ugly

Since I have been using it for a few weeks, here's my short pictureless yet picturesque review of the Nexus 7 2013.

The good

  • The screen resolution of course. 1920x1200 is the minimum resolution I consider reasonable for a 7" tablet, period. Anything less is crappy DPI. It's 2013, we're not using pixelated DOS any more.
  • Plays any 720p mkv you throw at it without trouble, including Hi10 (at least with the excellent MX Player). Also seems to play a fair share of (non Hi10) 1080p mkv's alright, but more about this below.
  • This auto-brightness feature is great. It's probably been a feature of Android for some time, and most likely didn't originate from Android in the first place, but with the Nexus 7 being my first mobile Android device, I can only go "Of course, you'd want to use the front camera sensor to automatically adjust the screen brightness according to the environment!". 
  • Likewise, the independent new message notification light, is a another "Of course!"-type of feature.
  • Screen is actually readable while standing outside on a bright sunny day. Not that I care that much about using a tablet outside, but it's nice to know you should be able to do so if you need to.
  • Passed a few minor hurdles (unlocking and rooting), the device is very developer-friendly. At least, it's open enough to let run your own customized kernel in a reasonable amount of time, as well as configure the OS exactly as you see fit. Bite me, Apple!
  • Supports > 54 Mbps WiFi (a.k.a. 802.11n). This is likely to be useful if, unlike everybody else out there, you actually plan to keep using your Nexus in 5 years time or more. Yes, there exist consumers who don't go around and change their mobile device every other year!
  • Standard USB port for charging and data. None of this proprietary connector crap.
  • Supports wireless charging. Being able to rely on two options for charging your device could be a life saver in the long run, if the other option breaks down.
  • Can actually store and play mkv's that are larger than 8 GB. If it isn't obvious by now, I am quite interested in HD movie playout.
    I mean, it would be a shame to have a high resolution 7" screen, and not use it as an HD player wouldn't it?
  • Sleek design. This is my first tablet after all, and I do consider it as a direct "upgrade" from my last portable device, which was a PSP-1000 (bought in 2006), so I'll reserve the right to be pleased by the form factor. Oh, and I sure hope that tablet will last as long in terms of use as my old sturdy PSP.
  • Automatically recognizes USB keyboards and/or mice. Though you have to use an OTG adapter, this too could come handy.
  • Quad core CPU. Unless recompilation of a full Linux kernel on your device doesn't sound like a ridiculous prospect, your device is underpowered.
  • Decent battery life. Could probably watch a couple movies over a transatlantic flight, and have some power left. I like that idea. Of course, the real killer would be to have enough power to play a full game of Civ 5, should Civ 5 ever been ported (or portable) to Android. Oh well, a man can dream...
  • Storage can (awkwardly) be extended through the use of a cheap USB OTG adapter. Doesn't compensate for an SD card expansion, but makes the pill somewhat easier to swallow.
  • The Play Store is furnished with more than enough good free apps to find something that will appeal to you when it comes to performing common tasks.
  • From having used some of their hardware, I have at least some confidence that Asus do actually know a thing or two about designing devices that can last.
  • Awesome pick as a first Android device. It does feel polished and mature enough to be considered as something a consumer truly deserves, i.e. a device where a sufficient amount of thought and effort seems to have gone into designing the various core functionalities.

 The bad

  • Uses a micro-USB connector #1. I cannot seriously see this connector last. Just like with mini-USB (and boy have I seen mini-USB connectors fail...), there's nothing sturdy about it, and the more you use the connector, the more you feel like it's gonna break like a twig and render your device close to unusable the day you inadvertently yank a cable or an OTG extension, or insert the plug in an odd position... which is apparently what all USB plugs are really designed for.
    Having seen the allegedly more sturdy USB 3.0 micro connector rendered worthless from a cable mishap, as well as having probably performed more than a hundred plugs/unplugs in the few weeks I got the device, I can only dread the day the connector will break. A meagre consolation will be that, when that happens, you're not going to be stranded with a complete paperweight, as you'll be able to use a wireless charger. But then you can kiss goodbye to USB debugging, device reset and any form or development...
    As an aside, were they actually listening to end-users' woes, the USB committee would have devised a sturdy small factor connector, that could be inserted either way, long before Apple did. Then again, that would take letting representatives of the general public have a say in the committee's agenda, which, as we all know, is pure unadulterated heresy!
  • 32 GB max internal flash max is way too small!! 32GB was already too small in 2010 for crying out loud! In 2013, 128GB should come as standard, with 256GB for high end. I can only imagine how ridiculously limited a tablet with "only" 32GB will be in 5 years time... This is all the more incomprehensible as anybody can purchase 64GB micro SD cards off Amazon for less than $50, ergo this is neither a question of technological limitation nor price.
  • No SD slot for you! Google and its minions can lie as much as they want, but everybody knows there's just one reason not to have an SD card slot on a portable device, and that is to force users to get their multimedia content from the restrictive cloud. :(
  • Uses a micro-USB connector #2: The lack of an SD slot above means you pretty much have to use a micro to regular USB OTG adapter, for anything that involves expanding the ridiculous amount of internal flash, which is both awkward and uncomfortable. That is unless you are using a micro SD card with meenova Mini MicroSD Card Reader (which, while a bit pricey, is well designed and well worth the satisfaction of sending a big fat "screw you and your stupid restrictions" message to Google)
  • I've seen a bunch of artefacts (banding in gradients) when displaying pictures, which I haven't seen when the same picture is shown on a regular monitor. That makes me think that the screen is not actually able to display 16M nuances of colours. I have waited 8 years to upgrade from the lousy screen that Sony decided to ship with original PSP (Oh, the remanence!), and this is this only as far we've been able to make the technology evolve? That is a (small) letdown... 
  • Uses USB 2.0 rather than USB 3.0. I don't care if the read and write speed of your flash can't reach USB 3.0 speeds, the RAM can, and one might want to use the device in the future for applications connecting to an USB device where high bandwidth is a must (software oscilloscope, software radio).
  • The default picture and movie viewers from Android are severely lacking functionality. Thankfully, the free QPic and MX Player can be installed, that do a much better job. Still, having a default picture viewer that you can't set to display pictures in alphabetical order is unacceptable.
  • Can't play Hi10 (a.k.a. 10 bits) 1080p mkv's (unless you consider a 2 FPS rate as acceptable). I don't think even a fully optimized video player will ever be able to so on this device, because the GPU and CPU are unlikely to be that powerful. Sure hope the future will prove me wrong though...
  • Video playout of some 8 bits 1080p mkv can be very jerky too, if MX Player decides that the HW+ decoder cannot apply, and, besides Hi10, I have no idea what decides whether HW+ can be used for decoding or not. Thus, depending on how the video file was produced, 1080p playout can be a bit of a hit and miss...
  • Touch screens suck. Having to wipe finger smudges all the time, and wonder where the OS will have decided the tip of your finger went is annoying. Oh and don't even get me started about trying to play arcade games using touch controls. At least for gaming, it shouldn't be that hard to add two 4+1 positions knobs (think microswitch joystick) in a very shallow depressed speaker-like shape, at the rear of the device, to at least allow some acceptable [game|arrow-keys]-like input.
  • As with any tablet, there's no "good way" to get a firm yet relaxed grip when using it.
  • When you try to raise the volume past a completely arbitrary point when using headphones, you get a MOST ANNOYING POPUP about listening at high volume for a long time being potentially damaging, that you have to accept EVERY FRIGGING TIME. The problem is, if you are using a high quality headphone with relatively large speakers, the volume at which this message appears is way too low! I think I'll have to figure out a way to disable this little sucker, because this chronic nagging is really degrading the whole media player experience.
  • No pouch? My old PSP came with a pouch! A portable device whose screen isn't expected to survive a 1.5m drop should come with a pouch for transport and (minimal) protection. What's more, if it has a touch screen, it should also come with a microfibre cloth, that's at least as big as the width of the device. This may sound like a trifle, but it shows that you might actually value your customers.

The Ugly

  • The Android community.
    Trying to find information that is actually of interest for development is such a massive ordeal! You have to go through page upon page of ill-informed posts, as well as scores of people jumping on the Android bandwagon to try to make a quick adversing buck, before reaching anything of actual substance... And don't get me started on Google's "Welp, that 5 line overview of the development process'll do". I'd like to remind Google that, as opposed to most Open Source projects, they do have paid employees that could take care of providing comprehensive documentation...

I guess that will do. This should be different enough from any other review of the Nexus 7 2013 you've seen. And I know some who have made a small jump from a 2012 Nexus 7 to a 2013 Nexus 7, and have been all ecstatic in their review about it, whereas here I am, having jumped leaps and bounds from an outdated portable device, yet with loads to complain about. Still, by my standards, this is a fairly positive review, and I would recommend the device (even over the new more powerful Amazon Kindle for instance), especially as I can't really fault Asus and Google for some items I have a gripe with, such as the USB committee being led by a bunch of monkeys...

2013-09-30

Rooting your Nexus 7 (2013) for the Android virgin...

...but Linux experienced.

Having recently purchased a Nexus 7 32GB (2013 model) as my very first Android mobile device, I am finding that the information on how to gain complete control is both all over the place and supposes you're already familiar with the whole rooting process, which I'm not. Therefore, I'm going to write yet another step by step guide on what you need to do, to be able to remotely ssh into it and run a shell as root, which, IMO, is the one stage where we can pretend that we are actually running a GNU/Linux system under the hood and get things done.

Incidentally, if you don't care about commandline Linux, this process will also grant you full control and let you do whatever you please with the device.

Being a complete Android noob, I will also make sure not to skim over all the "Oh, but you're supposed to know that already!" details, so that you will hopefully be as successful as I was, in rooting your device. On the other hand, I will assume that you are somewhat familiar with Linux. But don't let that worry you as the Linux specific items will come at the very end, at which stage your device will already be fully rooted.

Finally, since it's somewhat convenient, I will be using Windows as the platform we'll use to communicate with the Nexus 7 during this process, and I'll assume that you are familiar enough with it to find your way around the Device Manager, manual driver installation and the Windows command line.

Why *I* want to root my device


I'm not going to venture a guess as to what your reasons are for rooting your device. But one of the first reasons I have, in a very long list, is to be able to edit the device's /etc/hosts and get rid of most of the ads and unwanted trackers, which every other device and computer that I have doesn't have to put up with, through a skilful use of the ever so awesome WinHelp 2002 host file. And yeah, I know there are other ways to block ads, but the /etc/hosts method is still the simplest, most universal and effective way, since it'll block ads for all apps running on your system, rather than the ones in a single browser application.

Now, the ironic part is: the reason I purchased a Nexus 7 in the first place is through revenue that I got from placing ads on a successful little web site (Hurray, for unexpected disposable income!).
Am I really going to bite the hand that feeds me? Watch me!

Besides eliminating ads, there are of course plenty of other reasons to want to root your device, such as recompiling and running your own custom kernel (we'll come to that in a next post), but we might as well start small, by improving our mobile surfing experience.

What the other guides omit to tell you...


For the most part, I am going to follow the video guides at xda-developers and most specifically, the third and second to last from this link. Oh, and if you want a less verbose, yet noob friendly and up to the point guide, I can only recommend the "How to build Cyanogen for Nexus 7 2013" one, as it has a very easy to follow section on rooting. Still, part of these guides suppose some familiarity with the process and omit some important details that you may be interested in knowing beforehand, such as:

The internal name of the Nexus 7 2013 platform is 'flo'

This might be relevant if you are flashing an Android image file that came from a website that isn't dedicated exclusively to the Nexus 7 2013. If you do that, you may want to check that 'flo' is mentioned somewhere in what you're looking at. Alternatively, 'msm', which is the development branch for the 'flo' platform might be a good indicator. Remembering these two can be a good way of avoiding looking at data that is only relevant for the "old" Nexus 7, or other incompatible Android hardware.

You are going to lose all your data!

More specifically, as soon as you unlock the device, which is pretty much the first step invoked in rooting, your device will be completely reset to factory settings.
In the notice they display when you do that (see screenshots below) Google say that this is to prevent someone who got unwarranted access to your phone from accessing your personal data (which is probably only a half truth, but hey...).
My advice then is: if your device is new, you want to root it as soon as possible, so that you don't have to go through the pain of restoring your apps, settings and personal data.

Then again, if you have let some time pass and installed a few things, provided you did set the device to use your google account for backup, a lot of it will be restored automatically as you re-indentify yourself after unlock. Especially, all the apps you have installed, whether free or not, will be re-downloaded and re-installed. Also, if you're lucky, some of the customization and personal data files associated with them may be restored... though you shouldn't really count on that and take your precautions.

But really, the less customized your device is before you root it, the better. Before attempting this procedure then, you probably want to back up as much as you can... though of course, it takes a rooted device to be able to perform a comprehensive backup of your data! As if we needed more reasons to want to root our device early.

As I assume that you are as new to the process as I was, here is a quick overview of the steps we are going to follow:
  1. Enable Developer access on your device - this will allow to connect your Nexus 7, through USB, to the Android debug console as well as the boot/flash utility. We'll need this to unlock the device and flash the custom (non Google) recovery image, that'll help with installing root access
  2. Install the Developer tools on your PC, so that you can communicate with your Nexus in developer mode.
  3. Unlock the stock bootloader, so that it will let you flash a non-stock recovery image. This is also the process that resets your device and data to factory settings.
  4. Install (flash) a custom recovery image - This will allow the installation of the binary that will supervise the granting of root access to applications (SuperSU).
  5. Boot into that recovery image and install (flash) the SuperSU system binaries and deamon.
  6. Reboot the device into standard operation mode, and install the SuperSU app. (SuperSU comes in two parts: A regular "client" app, that can be downloaded from the App Store, and a system executable part that needs to be installed separately into the flash filesystem).
  7. Install Busybox
  8. Install an SSH server
  9. Customize our GNU/Linux system, including /etc/hosts.

Enabling Developer access


So you've backed up whatever you could and accepted the idea that your Nexus will be restored to the clean slate state you found it right after you purchased it. Good.

The first thing we need then is enable developer access, which is disabled by default. To do that, you need to navigate to "Settings" → "About Tablet" (at the very end) and tap the "Build number" section 7 times. Why 7 times? Who knows...

Once you do that, you will get a small prompt indicating "You are now a developer" and after that has occurred, if you go back into the Settings menu, you will see a new "{ } Developers options" category. There, you will find a bunch of additional parameters that you can modify, that may come handy if you do development, but the only one we need right now, under the "DEBUGGING" section, is "USB debugging". So please go ahead and tap that one.

As soon as you do that, you will receive a warning about "USB debugging being intended for development purposes only", which you should accept. This should lead to the following checkmark being displayed:


Installing the USB driver and developers tools


Unfortunately, even as it is WinUSB based, the current version of Android is not WCID enabled (but I'm working on that!), so that means you need to manually install some drivers, to be able to communicate with the device in debug mode.

You should therefore head to http://developer.android.com/sdk/win-usb.html click the "Download Google USB Driver" button, then pretend you accept whatever agreement they want you to agree by not reading it (so that you can feign complete ignorance of the terms if you break them, which I hear is how EULAs work these days) and unzip the files into an usb_driver\ directory somewhere.

Now, you should connect your device to an USB port on your Windows machine. Windows will try, and fail, to locate a driver for the new debugging USB interface, at which stage you should select the "Nexus 7" unrecognised device in Device Manager, select install driver, and point to the usb_driver\ directory you previously extracted.

Once you have done that, you should see something very similar to this:


All that is left, with regards to the developers tool installation, is to go to http://developer.android.com/sdk/index.html and Download the SDK ADT Bundle for Windows (about 450 MB in size). Depending on whether you picked the 32 or 64 bit version, you should end up with a zip file such as adt-bundle-windows-x86_64-20130917.zip which you can then extract somewhere like D:\Android\adt-bundle-windows-x86_64-20130917\.

Unlocking the bootloader


The only thing we are really interested in package above are the adb.exe (Android Debug Bridge - The application that communicates with the Android OS in debug mode) and fastboot.exe (The application that communicates with the bootloader) and that are located in the sdk\platform-tools\ directory. That's right; we downloaded more than 450 MB worth of junk just to access a couple of executables that don't even occupy 1 MB worth of space... talk about bandwidth wastage! (if you are concerned about this, you may want to look for smaller unofficial packages containing only adb and fastboot, or recompiling these from git)

Next, you should open a command prompt into your sdk\platform-tools\ directory and issue:

adb start-server

The command window will display the following:

D:\Android\adt-bundle-windows-x86_64-20130917\sdk\platform-tools>adb start-server
* daemon not running. starting it now on port 5037 *
* daemon started successfully *

D:\Android\adt-bundle-windows-x86_64-20130917\sdk\platform-tools> 

And you will also receive a request similar to this one on your device, which you should accept:


Once you have accepted the request, you will get a short message on the top status bar, indicating "USB debugging connected". Then a persistent USB debug icon, similar to the one displayed on the left, will appear in the top left corner.

The next command we want to enter, in our command prompt, is:

adb reboot bootloader

Your Nexus will now reboot in bootloader mode and once that is done, you should see a screen similar to this one, indicating "LOCK STATE - locked":


Note 1: If needed, you can use Volume Up/Volume Down to select the option highlighted at the top, and press  Power to execute that option.

Note 2: You can also enter the bootloader without using the adb command, by powering your table off, and then pressing Volume Down while powering it up.

Of course, we're going to change that LOCK STATE. To do that, and since we are in bootloader mode, we can use the fastboot command:

fastboot oem unlock

You will get a warning similar to this:


Use Volume Up/Down to select "Yes" and press the Power button. It should return to bootloader menu, but this time you should see "LOCK STATE - unlocked":


Now, you can now reboot your device by issuing the following:

fastboot reboot

Note: Because you unlocked the bootloader, the device will now reset to its factory settings. This is the part where you lose ALL YOUR DATA AND CONFIGURATION.

Installing the TWRP custom recovery image


Make sure your device boots properly, and recreate your account.

You probably also want to take some time to let the backup process reinstall your apps, as well as manually download and reconfigure whatever personal settings and files you did backup, before you unlocked the bootloader. The rest of our operations here should not affect this data.

While your Nexus is being restored, you also want to download the following 2 files:

  • The latest TeamWin Open Recovery image from http://techerrata.com/browse/twrp2/flo, which you should save into your sdk\platform-tools folder\
  • The latest SuperSU zip, from http://download.chainfire.eu/supersu, which you should save, on your Nexus device, in the "Downloads" folder, without extracting it (i.e. as a zip file). This file will be named something like "UPDATE-SuperSU-vX.YZ.zip".
For safety, once you have restored apps, settings and files, you should, issue a complete poweroff. Then boot it back again, confirm that you have USB debugging running (The small Android icon at the top left corner) and, in the command prompt Window issue:

adb reboot bootloader

Now that we're back in the bootloader, we are going to replace the stock recovery image, by flashing the TWRP we just downloaded over it. Provided the TWRP file you downloaded above is named "recovery openrecovery-twrp-2.6.3.0-flo.img", the command you will need to issue:

fastboot flash recovery openrecovery-twrp-2.6.3.0-flo.img

Now you need to reboot in recovery mode. You can do that by using Volume Up/Down until "Recovery Mode" is displayed at the top, and then pressing Power.

You should be greeted with the following screens:



Once you see the above, tap "Install" and select the "UPDATE-SuperSU-vX.YZ.zip" you should have in your "Download" folder:


You should then see the following:



At this stage, you should proceed to swipe, to install the files contained in the zip into the OS partition. What this process actually does is copy two executables su and chattr (Change Attribute) in your /system/xbin/ directory, as well as install a daemon that, will be launched at boottime.

You can then proceed to reboot the device, by selecting "Reboot" on the initial screen and then "System".

Installing BusyBox


Now that we have the system part of SuperSU installed, we can go to the Play Store and install the SuperSU client app, so just go ahead and do that.

Congratulations! At this stage, you have supposedly completed the hardest part, and your device is effectively rooted.

First thing you probably want to do then, is confirm that this is actually the case by installing "Root Checker" from the Play Store. If you launch it and click on the "Verify Root Access" you should get a prompt such as this one:


This is a SuperSU prompt, one that you should become exceedingly familiar with as you install more apps that request root access. When you see it, you should tap on "Grant", and after that, Root Checker should confirm that you have root access indeed.

Alrighty, moving on to our next target. We're of course not going to be sitting around installing rooted apps all day. What we want is an actual Linux shell, that we can access remotely. Thus, the first application we want to install is Busybox, that provides a basic set of base Linux utilities, as well as a basic shell.

On to the Play Store it is. If you search for BusyBox, you'll find a bunch of 'em. The one I installed was the free version from Stephen Ericson (Stericson) that will probably be the first result you'll see.

After it's installed, you should run the app, let it scan your system for a bit, and then go for the Smart Install. In the process, you will be greeted by another "Superuser request" prompt from SuperSU, which you should of course grant.

Once we've done that, and to confirm that BusyBox is properly installed, we're can to go back to our Windows prompt and issue:

adb shell

This opens an interactive shell on our Linux box (with a terminal that sucks, but we're not going to worry about that). You can then confirm that BusyBox was properly installed by issuing the command busybox. The result you get should be similar to this:

shell@flo:/ $ busybox
busybox
BusyBox v1.21.1-Stericson (2013-07-08 15:58:11 BST) multi-call binary.
BusyBox is copyrighted by many authors between 1998-2012.
Licensed under GPLv2. See source distribution for detailed
copyright notices.

Usage: busybox [function [arguments]...]
   or: busybox --list[-full]
   or: busybox --install [-s] [DIR]
   or: function [arguments]...

        BusyBox is a multi-call binary that combines many common Unix
        utilities into a single executable.  Most people will create a
        link to busybox for each function they wish to use and BusyBox
        will act like whatever it was invoked as.

Currently defined functions:
        [, [[, ash, awk, base64, basename, blkid, bunzip2, bzcat, bzip2, cal,
        cat, chat, chattr, chgrp, chmod, chown, chroot, chrt, cksum, clear,
        comm, cp, crond, crontab, cut, date, dd, depmod, devmem, df, diff,
        dirname, dmesg, dnsd, dnsdomainname, dos2unix, du, echo, egrep, env,
        ether-wake, expand, expr, fakeidentd, fdflush, fdformat, fdisk, fgrep,
        find, fold, free, freeramdisk, fsck, fsync, ftpd, ftpget, ftpput,
        fuser, getopt, grep, groups, gunzip, gzip, hd, head, hexdump, hostid,
        hostname, httpd, hwclock, id, ifconfig, ifenslave, inetd, inotifyd,
        insmod, install, ionice, iostat, ip, ipaddr, ipcalc, iplink, iproute,
        iprule, iptunnel, kill, killall, killall5, less, ln, logname, losetup,
        ls, lsattr, lsmod, lsof, lsusb, lzop, lzopcat, md5sum, microcom, mkdir,
        mkdosfs, mke2fs, mkfifo, mkfs.ext2, mkfs.vfat, mknod, mkswap, modinfo,
        modprobe, more, mount, mountpoint, mt, mv, nameif, nanddump, nandwrite,
        nc, netstat, nice, nmeter, nslookup, ntpd, od, patch, pgrep, pidof,
        ping, ping6, pkill, pmap, powertop, printenv, printf, ps, pscan, pwd,
        rdate, rdev, readahead, readlink, realpath, renice, reset, rev, rfkill,
        rm, rmdir, rmmod, route, run-parts, script, scriptreplay, sed, seq,
        setkeycodes, setlogcons, setsid, sha1sum, sha256sum, sha3sum,
        sha512sum, showkey, sleep, smemcap, sort, split, start-stop-daemon,
        stat, strings, stty, sum, swapoff, swapon, sync, sysctl, tac, tail,
        tar, tee, telnet, telnetd, test, tftp, tftpd, time, timeout, top,
        touch, tr, traceroute, traceroute6, tty, ttysize, tunctl, umount,
        uname, uncompress, unexpand, uniq, unix2dos, unlzop, unzip, uptime,
        usleep, uudecode, uuencode, vconfig, vi, watch, wc, wget, which, who,
        whoami, whois, xargs, zcat

shell@flo:/ $

Looks good enough. Now you can type exit to exit the shell and go back to the Windows command prompt.

And since we're done with the Windows command, you may want to issue:

adb kill-server

So that you can later disconnect your device without Windows complaining about it.

Installing an SSH server


At last, we've come to the final piece of our puzzle! Surprisingly, if you visit the Play Store, you'll find that there aren't that many SSH servers on offer.

The one I picked then is "Rooted SSH/SFTP Daemon", which looks a bit basic but does its job very well.

Once you launch it, tap the gear icon in the middle of the screen, and change at least the Login ID (I used 'root') and Password. You probably also want to enable "Wifi Lock" so that the SSH server will keep running in the background as you switch apps.


Now you want to press the large ON/OFF button so that it is highlighted red and so that our server is running. You should also get a small grey round ON/OFF icon in the top left area (see previous or next screenshot). Oh, and you'll probably have gotten another SuperSU grant request at this stage.

"Well, I seem to have an SSH server running alright, you say, but how do I find the name or IP address I should use to connect to it?"

The simplest method, without having to install another app or check your Wifi router, is to swipe down the Wifi icon in the top right corner and tap the Wifi properties.
Oh yeah, did you know that, if you swipe down any of the small status icons that appear at the top, you can quickly access the application and settings behind them? That's a good trick to know, and I sure didn't.

In this case, you'll be able to tap your current Wifi connection and get the IP address you need to use for SSH:



Give a go to that IP address in an ssh client, such as putty, and lo and behold, you should be connected to your Nexus 2013 Android device under a nice shiny shell. That's more like it!

Customizing our GNU/Linux Android system


Let's now push things to their logical conclusion, by performing 2 last items:
  1. As you may have noticed if you scan your network, and if you let the DHCP client provide its desired hostname, your Nexus 7 will be identified as 'android_12345678' or something. Ugh, let's change that!
  2. Edit that /etc/hosts to filter ads, as we said we'd do.
With regards to item 1, a quick search tells us that the file we want to edit is /system/build.prop, where you want to add a net.hostname=my_name line. So you go ahead and try that, only to find that the file is read-only?!?
What gives? Weren't we supposed to be root here?

Well, that's Android security for ya (which may or may not be that bad a thing). By default, only the directories under /sdcard (such as "Download" and stuff) are read/write. The rest of the system is read-only.

Fixing this is easy though. If you want to modify system files then, you just need to issue the following to be granted full read/write access:

mount -o rw,remount -t rootfs /system

Much better!

Now you can edit your /system/build.prop to your hearts content and change the DHCP hostname.

Let's finish the job by going ads busting then. Still from the ssh shell:

cd /sdcard/Download
wget http://winhelp2002.mvps.org/hosts.zip
unzip hosts.zip
mv HOSTS hosts
dos2unix hosts
cp hosts /etc/

The last line will overwrite the existing /etc/hosts, which is fine since it only contains one entry to alias 127.0.0.1 to localhost (which is also present in the new file). Oh, and the dos2unix is necessary as the original HOSTS file was saved in DOS mode.

Do that, try to surf, and watch in awe as the ads pollution has now gone away...

I think my work here is done...

2013-08-28

Does a clean install of Windows 8 recover existing Storage Space(s)?

Yes it does. of course, it would make the whole thing pretty useless if it didn't... Moving your Storage Space drives to a different Windows 8 machine should work too, though I haven't tested this specific scenario.

My advice however is to (re)install Windows 8 with all your storage space drives unplugged, then make sure the newly installed Windows 8 can access a non Storage Space HDD over all the mass storage controllers you will be using for your Storage Space array, and only then replug your Storage Space disks.


2012-07-06

Using rss2email and github feeds to only notify of issues

Long story short:
  • We want any github issue notification related to the libusbx github project being pushed as an e-mail to libusbx-devel mailing list.
  • The default notification mechanism of github is next to useless to do anything like that, especially if you're working with a project that is associated with a github organization.
  • As a result, we have created a libusbx-devel user that, monitors the libusbx project as a pull-only participant, and that receives NewsFeed about issues among other things. This NewsFeed can then be used as RSS/Atom (https://github.com/libusbx-devel.private.atom), and fed through rss2email to send e-mail notifications to the mailing list.
  • However, besides report about issues, the NewsFeed also contains reports about wiki updates and whatnot, i.e. loads of polluting events that we don't want the mailing list to get notified about. And neither github or rss2email offer the possibility to filter events on their own.
  • A quick look at the feed with curl indicates that RSS/Atom entries are tagged, with something like: <id>tag:github.com,2008:IssuesEvent/1570141350</id> or <id>tag:github.com,2008:IssueCommentEvent/1570141345</id> for the ones that are of interest to us, and <id>tag:github.com,2008:GollumEvent/1570245314</id> for the ones that aren't
  • Thus, if you use a dedicated copy of rss2email for the github feed, it is possible to modify the rss2email.py code, and add a filter that ensures only entries that have an issue related tag are processed, with:
    for entry in r.entries:
            id = getID(entry)
            if id[:25] != "tag:github.com,2008:Issue": continue
            
            # If TRUST_GUID isn't set, we get back hashes of the content.
    so that anything that isn't issue related is filtered out.
Of course, the proper way would be to add an INCLUDE_TAGS/EXCLUDE_TAGS section in rss2email's config.py, that can take wildcards, and then cross reference these in the code above, but since we don't have a all day...

Bonus 1: If you need authentication with rss2email, assuming that libusbx-devel is your user, the following is an example of the URI you should use for the RSS: https://libusbx-devel:PASSWORD@github.com/libusbx-devel.private.atom
Bonus 2: In case you want to play with the github JSON API, that pertains to issues, rather than the RSS, and provide libusbx/libusbx is your project, you can issue something like

curl -i -u libusbx-devel:PASSWORD https://api.github.com/repos/libusbx/libusbx/issues/events

2012-04-02

Crafting an MBR from scratch

If you follow this site, you'll remember that we previoulsy crafted a BIOS from scratch. Of course we may as well follow that up with writing an MBR while we're at it!

This time, the problem that was put to us was as follows:

As part of XP/2003 installation support in Rufus, which, if needed to be reminded, is your friendly bootable USB stick creation tool, we thought it'd be nice if, rather than having to fiddle with boot.ini options like other XP ISO → USB apps do for the second part of the XP setup process, we did something similar to what the original optical installation medium provides, with a "Press any key to boot from CD/DVD..." prompt.

More technically, the issue is that XP was not exactly designed by Microsoft to be installable from and USB drive. Therefore Windows expects to see the BIOS disk ID of the HDD it boots from, during the later stage of the installation process, as the first bootable device = 0x80, whereas 0x80 is the disk ID the BIOS assigns to the USB whenever it boots from it.
Thus, scripted methods of installing XP from USB would install an ntldr + boot.ini on the USB and prompt the user to select the second drive (first bootable HDD) to continue the process. Else the other workaround is to unplug the USB drive during reboot after the first part of the installation process is complete, and plug it back later, as Windows still needs to read files from it. Since these methods deviate from what users would see from a regular installation from CD/DVD, we tried to see if we couldn't come up with something better.

Our solution, then, is to craft an MBR that does the following:
  1. If a bootable HDD is detected as second BIOS bootable device (0x81), the MBR prompts the user whether they want to boot from USB and, if no input is given, will fall back to booting from the HDD (0x81) instead of USB (0x80)
  2. According to the bootable disk ID provided in the USB partition table, the MBR will swap the 0x80 device with the ID provided.
    This means that for instance, if the first partition on the USB drive has disk ID 0x81, then the USB disk is remapped to this ID, whereas the original 0x81, which would typically be the first bootable HDD, is remapped as 0x80 (first bootable device). Because this approach falls between swap and remapping, we call it masquerading, as each bootable drive is now being masqueraded as a different one.
Once the above is properly set, then the setup process on the HD can be led to think it always boots from 0x80, even if it was the USB that actually booted the system, and operate as if that was the case, leaving the installation process as close as possible to what users would experience when installing from CD/DVD.

For a more technical breakdown, of our process is as follows, knowing that the BIOS would have copied over the 512 bytes MBR at address 0x00007C00 when we start to run it.
  1. Because later stages need to copy boot records at address 0x00007C00, the first thing we do is move our 512 bytes code out of the way, by allocating 1 KB of RAM, duplicating our code there and then jumping to it.
  2. With our code now safely out of the way, we attempt to read the MBR of the second bootable device (0x81, as 0x80 would be the USB) into the 0x00007C00 address we just moved out from using INT_13h (disk), with either function 02h or 42h depending on the extensions found (Some BIOSes, such as older DELL, can only use 42h, so we must handle both 02h and 42h). If the read is unsuccessful, we just boot the USB.
  3. If the read is successful, we check the partition entries of the HDD MBR we just read, to see if there exists one that is bootable/active. If none is found, we give up and boot the USB.
  4. If an active partition is found,  we prompt the user to hit a key if they want to boot from USB by installing an override for INT_08h (timer) to provide us with both a timeout and the ability to print a dot every second.
  5. If the user presses a key, we install an override for INT_13h that does the following:
    - masquerades boot device 0x81 (first bootable HDD, second bootable device after the USB) as 0x80 (first bootable device)
    - masquerades the USB boot device (0x80) with the disk ID provided for its first partition (typically 0x81, but could be higher)
  6. We then handle the rest of the boot process to the relevant boot record (after removing the INT_08h override if needed). If the HDD is booted, we simply jump to the MBR we read at address 0x00007C00. If booting from USB, we read the partition boot record (eg. NTFS boot record) into address 0x00007C00 and jump there.
All of the above (and more!) is done in 440 bytes and with 2 bytes to spare (or 9 if you count the ones spent on convenience instructions). Not bad...

On a side note, the process of overriding INT_13h is used by various MBR viruses (eg. Michelangelo), and we actually had to take some measures to prevent anti virus applications from detecting our MBR as one. Our override of INT_08h for "Please press any key to boot from USB..." is also very close to what Microsoft does in the bootfix.bin it provides on its XP/2003 installation media.

The x86 assembly source of the MBR can be accessed here. If you're interested in writing your own MBR, feel free to have a look at it. Or, if you just want to see the MBR in action, feel free to download the latest version of Rufus, and use it to install Windows XP or Windows 2003 from USB.

Chkdskx and Formatx by Mark Russinovich


There used to be a time where, before they got purchased by Microsoft, Mark Russinovich's SysInternals provided a slew of very useful Windows utilities, with source. Of these, one of the most interesting had to be Chkdskx and Formatx, 2 utilities that leverage the fmifs.dll to duplicate as much as the Microsoft chkdsk and format functionality. Mark had the following to say about these utilities:
Have you ever wondered how exactly NT's two file system management utilities, chkdsk and format, work? Maybe you've had an application that would have been perfect if you could have incorporated chkdsk or format functionality into it. I present Chkdskx and Formatx, two utilities that very precisely clone the command-line chkdsk and format utilities that come with NT. In fact, the clones support the same switches as the standard chkdsk and format and produce almost exactly the same output. So while they don't do anything special, their source code demonstrates how you can write your own interfaces to chkdsk and format functionality.
The truth is, fmifs usage has barely changed since the days of Windows NT and what applied then to formatting and checking volumes still applies today if you want to format or check a volume in Windows 7 or Windows 8. Unfortunately however, the original SysInternals site went down around 2006, along with its utilities and source... Fortunately Mick's blog provided a link to a torrent of a full SysInternals site rip issued in 2006 (which I also duplicate here as well), and I will now make the original FMIFS.zip archive available here for your convenience, as it can be quite useful.

Or, if you want to have an overview of how fmifs.dll is used in a modern application, I can only invite you to have a look at the CheckDisk() and FormatExCallback()/FormatThread() calls of Rufus's format.c/format.h.

2011-12-16

Compiling FreeDOS on Windows, using VMWare player

Might as well follow up on the previous post...

Here is how I set up an environment to be able to recompile the FreeDOS kernel, on Windows 7 x64, using the VMWare player.
  1. Download the "FreeDOS Base CD" from here. Yes, I am well aware that there is also a "FreeDOS Base CD with source code" available, but we are going to recompile the latest kernel from SVN and in case I decide to upload my VMWare image to help y'all, I want the disk to be as little bloated as possible
  2. Create a new VMWare image in Player and select "I will install the OS later" when prompted. A 512MB (0.5 GB) disk is all you need in size, and I'm probably being generous. On the other hand since we are going to compile, setting at least 512 MB RAM for the VM (rather than VMWare's default 16 MB) is probably a good idea. You can also remove virtual floppy and sound card as we don't need them
  3. Set the VM to boot from the ISO you just downloaded in step 1, and start it. When prompted by FreeDOS, select "1) Continue to boot from FreeDOS from the CD-ROM" or press Enter, then select "1. Install to harddisk using FreeDOS SETUP (default)" and go through the installation steps. You may have to enter the VMWare BIOS (by pressing the F2 key) to make the CD-ROM first bootable device after partitioning in order to conplete the installation.
  4. One installed, FreeDOS should boot from the hard drive and the first bad surprise you'll get is that the default of "2 - Load FreeDOS with EMM386+EMS and SHARE" results in "CONFIG.SYS error in line 24 - Illegal Instruction occured". You should select instead "1- Load FreeDOS with EMM386, no EMS (most UMBs), max RAM free" or "3 - Load FreeDOS including HIMEM XMS-memory driver"
  5. Actually, let us start by fixing this inconvenience by issuing "edit fdconfig.sys" and changing the "MENUDEFAULT=2, 5" to "MENUDEFAULT=3, 5". Now that's better.
  6. Power off the Virtual Machine and mount its virtual disk for transfer as highlighted in my previous post (or find another way that is convenient to you to transfer files to the disk)
  7. Download the latest DOS 32 bit nasm from https://sourceforge.net/projects/nasm/files/DOS%2032-bit%20binaries/ and extract nasm.exe as well as cwsdpmi.exe from the archive to a \BIN directory on the FreeDOS hard drive
  8. Download the latest DOS/i386 version of UPX from http://upx.sourceforge.net and extract upx.exe to \BIN
  9. Download the latest open-watcom-c-dos-x.y.exe from http://ftp.openwatcom.org/ftp/ and copy it to your virtual FreeDOS hard drive (where is irrelevant)
  10. Fetch the latest FreeDOS kernel using svn (eg. using TortoiseSVN on Windows) from https://freedos.svn.sourceforge.net/svnroot/freedos/kernel/trunk and copy the whole kernel\ directory to \src\kernel on your FreeDOS disk
  11. Boot the FreeDOS VM again, and now run the open-watcom-c-dos-x.y.exe from wherever you copied it (which may now be listed as open-w~1.exe). Install Open Watcom as "Selective installation" and in "Toolkits and other components" remove "Sample programs". Also in that section, select the "Helpfiles" sub-menu and remove the "DOS Hosted Help Files". The installation will take a little while, but at least it's unattended. At the end, tell Open Watcom NOT to modify the AUTOEXEC.BAT and CONFIG.SYS files
  12. Edit AUTOEXEC.BAT and edit the PATH line to have the following:
    set PATH %dosdir%\bin;C:\BIN;C:\WATCOM\BINW
  13. Also in AUTOEXEC.BAT, after the line lh doslfn, add:
    set INCLUDE=C:\WATCOM\H
    set WATCOM=C:\WATCOM
    set EDPATH=C:\WATCOM\EDDAT
    set WIPFC=C:\WATCOM\WIPFC
  14. Reboot the VM
  15. Go to C:\src\kernel and issue "copy config.b config.bat"
  16. Edit config.bat and update the following:
    set XNASM=c:\bin\nasm.exe
    Uncomment all the relevant WATCOM options in the file
    It is also recommended to modify config.bat to have:
    set XCPU=386
    set XFAT=32
  17. Issue the command: build
  18. If you followed everything properly, you should end up with a successful build of KERNEL.SYS and other files in C:\src\kernel\bin
    Note that you can clean up a previous build by issuing clean
Bonus: For those who don't want to have to go through all these steps, please find a ready-to-use VMWare image here (Player 4.x, 86 MB, 7z compressed), created using the exact steps above.

How to get a vmdk mapped in Windows Explorer when using VMWare Player 4.x and Windows 7

VMWare... Yet another textbook story of a technological company that has become a bit too complacent with its users, possibly without even realizing it (Disclaimer: I have been a virtualization evangelist, as well as direct contributor to sales of VMWare products in the previous company I worked for, so I do feel somewhat entitled to exert criticism).

So, you have VMWare Player 4.x, installed on a Windows7 platform, and let's say you fancy recompiling FreeDOS in a virtualized environment. For this you have set up your VM, with a small virtual disk and everything, but since it's DOS, no shared folders or VMWare Tools will be available there. Yet you need a solution to exchange data in a convenient manner, between the Windows 7 host and the VM disk image. How are you going to do it then? Why, by mounting the vmdk as a virtual drive of course. And since VMWare is a Virtual hardware company, they made sure that such a staple of virtualization was as easy and as straightforward as possible, right?

Wrong. Below is what you actually need to go through:
  1. The VMWare Player does not natively offer any tools to map/mount a virtual drive. You have to install extra software.
  2. That software is the Virtual Disk Development Kit (presently in its version 5.5.1) which can be obtained here (after registration of course! It's not like you were about to sort this virtual disk mapping issue in 30 seconds anyway).
  3. Once installed (by running bin\vstor2install.bat), and of course without any obvious mention of it (since officially it's an SDK, not an application plugin!), you will find out that the following becomes available when editing your disk properties:

    Brilliant: a GUI with the ability to map drives is exactly what we've been looking for.
  4. Except, at least in Windows 7 x64, and if you run VMWare Player 4.x under your regular credentials, you will be greeted with the following error message "Error reading volume information. Please select another disk file":

    Same occurs if you try the vmware-mount.exe from from the VDK's bin directory (and this even if you open a command prompt as admin, as I have also found)
  5. To avoid the "Error reading volume information..." message then, and be able to mount the disk, you must run VMWare Player as admin
  6. HOWEVER, "Open drive in Windows Explorer after mapping" still will not work, because Explorer is running under your user credentials, and the drive was mounted under the Administrator's credentials. Thus, you must run your File Explorer as administrative.
  7. Luckily, if you happen to run a proper File Manager (such as the most excellent Directory Opus, which I can't recommend enough as it's well worth its license), this isn't a major problem and you will actually get the friggin' vmdk mapped at long last, with the ability to transfer files. If you're stuck with the default Windows Explorer, running it as admin may or may not work —can't test, as on my machine, Directory Opus is the explorer that will be launched by default by VMWare)
Yay, at last VMWare may actually prove useful in getting some work done. Now let's see if the FreeDOS guys also see it this way...

Update (2012.01.06): The FreeDOS people kindly indicated that vmsmount might also be able to help you out if you want to acces VMWare shares from DOS.

2011-12-14

Rufus - The bootable USB Formatting Utility

It is my very great pleasure to introduce Rufus (direct downloads here), my own GPL'd version of a bootable USB formatting utility (DOS + ISO).


If you have been using the old HP Utility to create DOS bootable USB, you can throw that old thing away! In a small executable, and with no requirement for an installer, Rufus offers you a much better and up to date interface, with better features, and a DOS creation that doesn't rely on external files. It can also create bootable USB from ISO images. Plus you will find welcome additional features, such as the ability to check your USB stick for bad blocks. Best of all, and as you have come to expect from this site, it is 100% Free Software.

Please make sure you check the official Rufus page, and stop looking further when you need a DOS bootable USB stick. Of course, Rufus is compatible with all versions of Windows starting with Windows XP and lets you use the always awesome FreeDOS alongside the rather old and not up to date Windows Millenium DOS.

Why are you doing this?

Well, the truth of the matter is that, after having used the HPUSBFW utility for some time, it turns out that I really can't stand proprietary software utilities (as well as Windows' glaring shortcomings), so I decided to create my own Open Source version of an equivalent tool. Also, the fact that the many people, who have taken a stab at creating their own DOS bootable USB formatting utility, decided to go closed source doesn't really help. Simple utilities should only be Open Source, period.

Some interesting technical details (or yet another annoying technical rant)

You'd think there wouldn't be much to formatting an USB flash drive for DOS on Windows, but you would be wrong. As I already explained, there's some reason why Windows doesn't do it natively. Also, you may be surprised to hear that Windows doesn't actually provide a public API to format a drive, and instead you have to hijack an undocumented one called FormatEx, and which can be found in fmifs.dll. Then you will find that FormatEx kind of destroys your partition table when you want to use LBA, so you need to fix it manually. There's also this whole business of allowed cluster sizes. And then there's all the usual traps, such as having the partition boot records needing to be patched on XP, because unlike Vista or later, it equates an USB Flash drive to a floppy, as well as Windows' somewhat mysterious handling of Physical Drives vs. Logical Volumes. One will let you access the MBR and the other one the Partition Boot Record, yet, you still need to hold a lock to the latter to be able to access its underlying sectors with the former. Straightforward, it is not.

If you are so inclined, you'll find Rufus' FormatThread() in format.c as a good starting point. Oh, and would be quite ungrateful if I didn't acknowledge other OSS projects, such as ms-sys (boot records handling) or e2fsprogs (bad blocks check) for providing some of the building blocks used by the utility, as well as the talented designers from PC-Unleashed for the Rufus icon.

Finally, for those interested, the acronym stands for "The Reliable USB Formatting Utility (with Source").

Enjoy!

IMPORTANT NOTE: If you want support, please use the Github issue tracker. Any requests for support in the comments will be left unanswered.

2011-12-12

Finding which preprocessor macros are defined in gcc

gcc -dM -E -x c /dev/null
Can be quite useful. Also works for Clang.

2011-12-02

Why Windows doesn't let you create an USB MS-DOS startup disk

Today, I'll provide an answer to the following question, which, if you're like me, must have irked you some when formatting an USB flash drive:

Yes why? Why is is that Windows has no problems creating an MS-DOS startup disk for the now useless floppy technology, but does not allow you to do so for the ubiquitous USB memory stick one?
It's not really like this is an impossible feat on Windows, as there exist quite a few tools that allow you to create one already (and I have now added my MUCH BETTER ONE to the mix). So why doesn't Microsoft do it?

If it ain't broke, don't fix it... But don't upgrade it either.

Ah, conservatism. I'm not talking about the political kind here, but rather its no less sinister computer equivalent: "Hey, we worked hard enough to get there. We don't see why we should take the risk of changing existing code, just so newcomers can reap the benefits."

You see, the way Windows creates an MS-DOS startup disk is not by doing anything smart, but by doing something lazy —which some also say is the second smartest thing to do after "smart"— such as including a 1:1 image of a 1.4MB floppy disk in a DLL (diskcopy.dll) and just splashing it wholesale onto a floppy when you request a startup disk.
Oh and yes, that's L-A-Z-Y-lazy since, if you actually have a look at the image file, you'll see that Microsoft didn't even spend much time creating a well thought-out one, but instead picked up the first Windows Millenium Edition (Me) startup disk they had lying around and just proceeded to "delete" stuff until it looked OK enough. The proof of it is that the image still contains much of the "deleted" content from the original disk and for a good read on how exactly the Windows Me Startup disk was "emasculated", you are very much invited to read Daniel Sedory's entry on the subject.

Of course, this whole technique of imaging a whole floppy does fall apart as soon as you're trying to create an unknown size USB bootdisk and this, my little children, is why Windows won't let you create DOS bootable USB flash drives to this day. End of story. Good night.

But wait, there's more!

Alright, alright, I hear you: that was a pretty underwhelming bedtime story. So of course there's more. And to answer your follow up question on why Microsoft didn't just compensate for the issue above by dropping the floppy image from diskcopy.dll and simply include the files themselves, we'll press on with a little bit of history.

As stated previously, the "DOS" files included in the image file that Windows uses when creating an "MS-DOS" startup disk actually come from a Windows Me boot disk, which, amongst its many disappointments, is also well known for crippling the DOS version it came with. As Wikipedia indicates, "One of the most publicized changes from Windows 98 is that Windows Me does not include real mode MS-DOS". Furthermore, departing from any MS-DOS that came prior, "the Autoexec.bat and Config.sys files are used only to set global environment variables". Thus DOS 8.0, which is the one included with Windows Millenium, is indeed seen by many as crippled, whereas DOS 7.1, a.k.a. the one from Windows 98 SE, is often labelled the "last good DOS".

But much more relevant to the USB boot issue is that the Windows Me DOS is not actually able to produce a command prompt on anything else but a floppy media. That is to say, even if you were to create all the required boot records on an USB drive, and then copy the unmodified COMMAND.COM and IO.SYS from the diskcopy.dll floppy image, your drive will never boot. You can try this for yourself, if you have an existing bootable stick, as 7-zip will happily extract the files from diskcopy.dll if you want to copy them over.

That still doesn't explain anything...

"OK fine," you say, "Windows Me is unusable but Microsoft should have the rights to their own DOS stuff anyway, so why don't they either just use the DOS files from Windows 98, or produce an uncrippled new version of the WinMe DOS, and go with that?". That's actually a very valid question, which people have only been able to speculate onto this day. Technically and most likely legally, there doesn't exist a reason why Microsoft shouldn't be able to provide a proper set of DOS files to boot an USB stick with. Furthermore, and this is the trick people use when they want to legally provide MS-DOS files for USB boot on Windows, uncrippling the WinMe DOS files from the diskcopy.dll to get them working on USB is actually trivial and documented here. For the record, even the well known HPUSBFW tool, from HP will patch the COMMAND.COM and IO.SYS files according to the above, when provided with WinMe DOS files. Yet, Microsoft went as far as explicitly discouraging people from doing so...

Thus, the moral of our story is that, while there is no real technical limitation to it, there is no satisfying explanation as to why Microsoft decided to inconvenience millions of their users by not allowing to create a DOS bootable USB drive. However, if you keep watching this blog, you will soon be provided with a 100% Open Source tool that does just that (and more!), so stay tuned.

Bonus

By the way, if you get your hands on an older version of the HPUSBFW tool --the one that is about 400 KB in size, you should be advised that it contains, as a resource, a self extracting UPX compressed executable that will extract the Windows 98 DOS files. The utility itself is designed to only enable extraction of these files if specific HP USB hardware (VID:PID 03F0:0023) is detected, however that can easily be patched to enable any device (F0 03 75 15F0 03 90 90 & 23 00 75 0C23 90 90 0C) and one could use HPUSBFW as a standalone USB DOS boot utility.

2011-11-14

Using an older version of VMWare Tools

Lost enough time with this already so I'll be short. First of all, yes, sometimes the latest version of the VMWare Tools can fail and you want to downgrade. I'm looking at you VMWare Tools 8.8.0 from VMWare Player 4.0.0 using a Windows 2000 image! And as you may have already found out, when that occurs, the VMWare Software as well as the VMWare community are less than helpful.

In case you're running the VMWare Player on Windows, to emulate another Windows virtual platform, here's what you should know:
  1. After they are downloaded by the Player, the VMWare tools are installed, as a windows.iso file, in your VMWare installation directory (typically C:\Program Files (x86)\VMWare\VMWare Player\). It is that ISO image that is mounted in the virtual machine during installation.
  2. You can obtain older versions of the VMWare Tools by navigating the not-intended-for-users VMWare Software Update directory, starting at http://softwareupdate.vmware.com/cds/vmw-desktop/player/.
  3. The VMWare Tools download is then found in <version>/<build>/windows/packages/ (eg. http://softwareupdate.vmware.com/cds/vmw-desktop/player/3.1.5/491717/windows/packages/ )
  4. Of course the download is not provided as an ISO, because that would be too easy. Instead, it's a tar of an exe that silently extracts and copies (overwrites) a new windows.iso in the VMWare directory. Thus, before you run that exe, you may want to rename your old windows.iso to something else, so that you can both tell whether the downgrade process completed successfully and have the ability to switch back between ISOs if needed.

2011-11-10

cathash

cathash is a multiplatform MS CAT/Authenticode SHA-1 generation, inspired by cathash by Michel I. Gallant, but not using CryptCATAdminCalcHashFromFileHandle, and thus able to run on UNIX platforms, including big-endian ones.

It is intended for the computation of the custom SHA-1, used by Microsoft and others, to verify the authenticity of Windows executable (or regular files), using the algorithm detailed at the end of the "Windows Authenticode Portable Executable Signature Format" specifications.

The utility was heavily validated against Windows system files (32 and 64 bit), MSVC generated files, MinGW/MinGW-w64 generated files and so on, so it is expected to be fully compliant with the MS SHA-1.
If compiled on Windows, and unless you comment the VALIDATE_HASH define, the program will also validate its computation against the one from CryptCATAdminCalcHashFromFileHandle, for extra safety.

By the way, if you are interested in producing your own implementation, you may want to note that the MS specs omit the fact that the optional extra PE data, starting at SUM_OF_BYTES_HASHED, needs to be padded to the next 8 byte boundary for hashing.

Outside of its upcoming libwdi usage, this utility may come handy for UNIX users needing to validate the authenticity of Windows files.
The source, which comes along with a (signed) 32 bit Windows executable can either be downloaded here (direct link) or here (SourceForge).
You can also directly access cathash.c, to compile it using any of Visual Studio, WDK, MinGW, cygwin or UNIX based gcc.

IMPORTANT NOTE: As reported by Abid Bhat, some Windows XP system files, such as C:\Windows\system32\drivers\update.sys, C:\Windows\Driver Cache\i386\sp3.cab or C:\Windows\Driver Cache\i386\driver.cab, are not getting the expected hash when computed by the current version of cathash. I'll look into it when I get a chance, but it may take a while...

2011-11-07

LINK : warning LNK4044: unrecognized option '/lib'; ignored

Error above from the WDK driving you crazy? And when you try link /lib everything is fine?

Well, How about removing that /nologo you have before /lib. Better now? And yes, I agree, whoever designed the MS linker to be this mind-numbingly picky about its option order should get a well deserved pie in face:
E:\WinDDK\7600.16385.0>link /lib /nologo

E:\WinDDK\7600.16385.0>link /nologo /lib
LINK : warning LNK4044: unrecognized option '/lib'; ignored
LINK : warning LNK4001: no object files specified; libraries used
LINK : warning LNK4068: /MACHINE not specified; defaulting to X86
LINK : fatal error LNK1561: entry point must be defined

E:\WinDDK\7600.16385.0>

2011-11-05

bin2coff

As part of the ongoing effort to add 7z compression to libwdi, I am publishing bin2coff, which is an improved version of the tool of the same name by Vortex. bin2coff is aimed at generating MS COFF object files that are simple wrappers for binary data, and that can be used by the MS (Visual Studio, WDK), MinGW (MinGW32, MinGW-w64) and other linker tools. Such a tool can be especially useful if you want to generate a library that contains a large chunk binary data. For instance, and this is what I plan to use it for in libwdi, you can use it to embed a complete 7z archive in a cross-compiled Win32 static library file, which you will then be able to extract at runtime.

What this version of bin2coff brings is 64 bit compatibility (MinGW-w64), the provision of a size variable besides the binary data, as well as the full sourcecode. A 32 bit Windows executable is also included in the archive, which you can either download here (direct link) or here (SourceForge).
Usage: bin2coff bin obj [label] [64bit]

  bin  : source binary data
  obj  : target object file, in MS COFF format.
  label: identifier for the extern data. If not provided, the name of the
         binary file without extension is used.
  64bit: produce a 64 bit compatible object - symbols are generated without
         leading underscores and machine type is set to x86_x64.

With your linker set properly, typical access from a C source is:

    extern uint8_t  label[]     /* binary data         */
    extern uint32_t label_size  /* size of binary data */

Addon:
If, for some foolish reason, or just because you like it when things blow up in your face, you want to integrate bin2coff generation with libtool, you will find that you need to generate a .lo wrapper. In that case, the following Makefile excerpt might be of interest to you:
wdi_data.lo: bin2coff
# $* is the target without extension - here 'wdi_data'
 bin2coff $*.7z $*.o $(bin2coff_OPTIONS)
# libtool won't let us get away with a mere .o - we have to generate a .lo wrapper.
# What's more, the first comment from the .lo is mandatory and must contain the libtool version
 @echo "# Generated by $(shell $(LIBTOOL) --version | head -n 1)" > $@
 @echo "pic_object='$*.o'" >> $@
 @echo "non_pic_object='$*.o'" >> $@

2011-11-03

Access denied and Windows 7

One of the annoying aspects of using NTFS is that, from time to time (eg. if Windows 8 preview corrupted your Windows 7 partition so bad that you had to reinstall the whole OS - see post #7), Windows will decide that you are no longer entitled to access your files, and while performing some operation or other you'll get something like:
There's nothing more infuriating in terms of computing than being denied access to files you should have access to.

The solution? Open an elevated command prompt and issue:
takeown /R  /F <Drive or directory>
Icacls <Drive or directory> /reset /T

2011-10-25

Building and running Clang static-analyzer on Windows/MinGW

With the Clang static-analyzer becoming more and more popular these days, MinGW users on Windows might be looking for some way to also bring the Clang goodness to their shores.

However, well, let's just say that the LLVM documentation isn't that intuitive for newcomers, especially if you were expecting to be able to download a nice Windows binary package and roll. This quick recipe on setting up a complete Clang static-analyzer environment for MinGW attempts to remedy that. As usual, what is presented below worked on my environment at the time of the post - I make no guarantee that it will work elsewhere, especially if you choose to deviate from the guide.

Also, before you start, please be aware that this whole process will send you back about 11 GB in terms of disk space, so make sure you have adequate room.

  1. Install MinGW (32 bit!) using the latest msys-get-inst (just run the GUI installer) picking up the very latest packages.
    For the installation directory, I used D:\Clang, which I will assume to also be yours for for the rest of the guide - please modify accordingly. With regards to the installation packages, select MinGW Compiler SuiteC compiler (which should be selected by default), MinGW Compiler SuiteC++ compiler, and MinGW Developer Toolkit to ensure that binutils and all the necessary tools to build clang, except Python, are available. You should also select MSYS Basic System, so that a shell is available. You probably also want to play it safe by picking up the pre-packaged repository catalog for MinGW. The retrieval and installation of the packages will take a few minutes.
  2. Download Python v2.x (32 bit installer) from http://www.python.org/download/.
    Use D:\Clang\bin as the installation directory and unselect everything but Python Interpreter and Libraries, as this is all you really need from Python.
    IMPORTANT: Using python 3.2.2 produced an error on my machine during the build, so make sure you pick v2.x.
  3. [OBSOLETE: You can skip to the next step as per comments below]
    Download the latest (Experimental) Clang Binaries for Mingw32/x86 from http://llvm.org/releases/download.html. Extract its content (bin/, include/ ...) into D:\Clang, overwriting any existing files. We do not care about headers being overwritten as this setup will be used for Clang and Clang only. Always make sure llvm-gcc is installed in the /bin directory of MinGW, unless you want to see a CreateProcess() error when running it (or want to have to duplicate the binutils files which we already got installed from picking up the MinGW Developer Toolkit).
  4. After installing an Subversion client such as Tortoise-SVN if needed:
    • Checkout llvm from http://llvm.org/svn/llvm-project/llvm/trunk (this is a Subversion URI), into D:\Clang\msys\1.0\src\llvm
    • Checkout clang from http://llvm.org/svn/llvm-project/cfe/trunk, into D:\Clang\msys\1.0\src\llvm\tools\clang
      NB: you may have to checkout clang outside of llvm first, then move it to the right location if your Subversion client doesn't let you checkout a repository inside an existing repository.
  5. Open an msys shell by launching D:\Clang\msys\1.0\msys.bat and issue the following commands:
    cd /src
    mkdir build
    cd build
    export CC=gcc
    export CXX=g++
    ../llvm/configure --disable-docs --enable-optimized --enable-targets=x86,x86_64 --prefix=/mingw
    make -j2
    The --prefix ensures that our files will be installed in the bin directory of MinGW and the two export lines avoid the need to install the Clang binaries to be able to recompile (As per now obsolete Step 3). Note that, in the msys shell, /mingw always point to the root directory where you installed MinGW (here D:\clang) regardless of the name you used. I would strongly advise against using a different directory, or not specifying the --prefix option (in which case /usr/local will be used as default), as I ran into issues when I tried (the static-analyzer failed to locate <stdio.h> when parsing sources). As usual, the -j2, which specifies the number of jobs to run in parallel, is there to make use of as many cores as we have (here two). Adjust according to your platform.
    Oh, and if you want to know why we use --enable-optimized, see the very end of this post...
  6. Go for a walk or something. Some of the libraries and executables are huge (hundreds of MB in size) so this whole process will take a while. During the various compilation runs I conducted, I chanced in one that froze (one make process using 100% of the CPU and never completing) and another where Windows reported "llvm-tblgen has stopped working", so don't be surprised if the compilation doesn't work quite right first time 'round -- it's fairly heavy on resources.
  7. If the compilation completed, then you can run:
    make install -j2
    Note that this operation will take some time too!
  8. Copy all the content of D:\Clang\msys\1.0\src\llvm\tools\clang\tools\scan-build, except c++-analyzer to D:\Clang\bin.
    Copy all the content of D:\Clang\msys\1.0\src\llvm\tools\clang\tools\scan-view to D:\Clang\bin.
  9. Issue the command:
    ln -s /mingw/bin/ccc-analyzer /mingw/bin/c++-analyzer
    The reason we're re-creating this symbolic link (which on Windows ends up being a duplicate rather than a link anyway) is that the one from the file provided by LLVM only works on UNIX systems.
  10. Edit D:\clang\bin\scan-build, line 100, and remove the /bin there to have:
    my $ClangSB = Cwd::realpath("$RealBin/clang");
    If you don't do that, scan-build will look for clang in /mingw/bin/bin and obviously fail to locate it.
  11. Finally, since we had to override them, you shouldn't forget to undefine the CC and CXX environmental variables:
    export -n CC CXX
    unset CC CXX

Success looks like what?

$ cd /d/libusb-pbatard
$ scan-build ./autogen.sh
(...)
$ scan-build make
make  all-recursive
make[1]: Entering directory `/d/libusb-pbatard'
Making all in libusb
make[2]: Entering directory `/d/libusb-pbatard/libusb'
  CC     libusb_1_0_la-core.lo
core.c:1626:3: warning: Value stored to 'r' is never read
                r = 0;
                ^   ~
1 warning generated.
  CC     libusb_1_0_la-descriptor.lo
  CC     libusb_1_0_la-io.lo
io.c:1819:8: warning: Call to 'malloc' has an allocation size of 0 bytes
        fds = malloc(sizeof(*fds) * nfds);
              ^      ~~~~~~~~~~~~~~~~~~~
1 warning generated.
  CC     libusb_1_0_la-sync.lo
  CC     libusb_1_0_la-poll_windows.lo
  CC     libusb_1_0_la-windows_usb.lo
os/windows_usb.c:1218:26: warning: Value stored to 'discdevs' during its initialization is never read
        struct discovered_devs *discdevs = *_discdevs;
                                ^          ~~~~~~~~~~
os/windows_usb.c:1297:4: warning: Value stored to 'session_id' is never read
                        session_id = 0;
                        ^            ~
2 warnings generated.
  RC     libusb-1.0.lo
  CC     libusb_1_0_la-threads_windows.lo
os/threads_windows.c:115:2: warning: Value stored to 'prev_pos' is never read
        prev_pos = pos = NULL;
        ^          ~~~~~~~~~~
1 warning generated.
  CCLD   libusb-1.0.la
Creating library file: .libs/libusb-1.0.dll.a
make[2]: Leaving directory `/d/libusb-pbatard/libusb'
Making all in doc
make[2]: Entering directory `/d/libusb-pbatard/doc'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/d/libusb-pbatard/doc'
Making all in examples
make[2]: Entering directory `/d/libusb-pbatard/examples'
  CC     xusb.o
xusb.c: In function 'test_mass_storage':
xusb.c:443:9: warning: variable 'junk' set but not used [-Wunused-but-set-variable]
xusb.c:520:4: warning: Value stored to 'junk' is never read
                        junk = fwrite(data, 1, size, fd);
                        ^      ~~~~~~~~~~~~~~~~~~~~~~~~~
1 warning generated.
  CCLD   xusb.exe
./.libs/lt-xusb.c:692:11: warning: Value stored to 'len' during its initialization is never read
      int len = strlen (new_value);
          ^     ~~~~~~~~~~~~~~~~~~
1 warning generated.
  CC     lsusb.o
  CCLD   lsusb.exe
./.libs/lt-lsusb.c:692:11: warning: Value stored to 'len' during its initialization is never read
      int len = strlen (new_value);
          ^     ~~~~~~~~~~~~~~~~~~
1 warning generated.
make[2]: Leaving directory `/d/libusb-pbatard/examples'
make[2]: Entering directory `/d/libusb-pbatard'
make[2]: Leaving directory `/d/libusb-pbatard'
make[1]: Leaving directory `/d/libusb-pbatard'
scan-build: 8 bugs found.
scan-build: Run 'scan-view /tmp/scan-build-2011-10-24-1' to examine bug reports.
This is a typical result that Clang will (eventually - the analysis takes some time!) churn through. One false positive (that 'junk' variable is meant to be ignored, and only serves to eliminate a warning about not reading the return value from fwrite) and apart from that malloc code, everything looks fairly minor. Even on its very first report from clang, which of course we will act upon, libusb-1.0 on Windows doesn't appear to fare too bad!

Bonus:
If you followed the guide above, chances are you'll be using separate installations for the standard MinGW/MinGW-w64 and the Clangified ones. As such, you may want a title on the msys shell window that reflects that you are in a Clang environment, to identify which is which more easily.
To change the msys commandline window title, just edit your D:\clang\msys\1.0\etc\profile, locate the line:
export PS1='\[\033]0;$MSYSTEM:\w\007
and replace $MSYSTEM with the title you want.

Bonus 2:
Displayed after spending more than one hour of compilation, using gcc:
llvm[0]: ***** Completed Debug+Asserts Build
llvm[0]: ***** Note: Debug build can be 10 times slower than an
llvm[0]: ***** optimized build. Use make ENABLE_OPTIMIZED=1 to
llvm[0]: ***** make an optimized build. Alternatively you can
llvm[0]: ***** configure with --enable-optimized.
Now you tell me!?! Shouldn't that be the kind of info you want to display after someone issues configure?

2011-10-14

On Windows 8 and WCID devices

With the 1.2.0 release of libwdi, which, for those who don't know, is a library aimed at making the installation of Windows USB drivers as simple and as non intrusive for end-users as possible, one of the most exciting features brought forward is WCID device driver support.

WCID, which was brought to the limelight with Windows 8, and which stands for Windows Compatible ID, is a set of additional USB Descriptors one can add to a device firmware, to make the OS automatically identify which type of driver should be used, and potentially automatically install it. In other words, what WCID does is bring the Plug-and-Play functionality that exists for HID or Mass Storage, to any USB device.

Thus, as upcoming Windows 8 comes with a WinUSB WCID driver, if you currently produce USB devices that require driver installation, you should seriously consider converting them to WCID as it'll make life a lot easier for everybody.

This is also good news for libusb-1.0 users, as libusb-1.0 relies on WinUSB for generic access on Windows, which means that you can actually produce libusb-1.0 applications for Windows 8, that do not require you or your users to take care of any driver installation. And for earlier versions of Windows, up to Windows XP SP2, libwdi and its friendly installer application Zadig can make it very easy to proceed to the one-off installation of the WinUSB (or libusb-win32 or libusbK) WCID driver, to make those platforms behave in the same way as Windows 8.

More information:

2011-10-12

Avoiding Program Compatibility Assistant warnings on Windows

If you're developing applications on Windows Vista or later, you may end up in a situation where this rather annoying little fellow pops up after your program execution:


"What gives?, you say, my program's not an installer and it doesn't have anything like 'setup' or 'install' in the executable name! Why does Windows pester me and my users with such a warning?" The answer: have a look at your resource file (RC, etc) and especially your versioning strings, such as "FileDescription" or "ProductName". If there's any or part of the taboo keywords in there, such as "setup", "install", "update", Windows assumes that the executable is an installer, and also assumes the worst. Thus, the solution is to never use any or part of these keywords in any of the location that Windows will look for.

For more information on how Windows "heuristically" detects installation programs, have a look at the Installer Detection Technology paragraph from the following TechNet note.

2011-10-03

Flashing a NEC/RENESAS USB 3.0 uPD720200/uPD720200A firmware

It looks like NEC/Renesas are releasing steady releases of their popular USB 3.0 controllers drivers and firmwares. In case you have such a controller, you probably want to check the following page for the latest version.

One of the problems I found with the latest firmware update (4015) is that the GUI Renesas upgrade utility just didn't work for my controller, on Windows 7 x64. Even worse, it created commandline processes ad-infinitum, which, apart from obviously clogging up the system, is the last thing you want to see, as repeated erase/flash cycles can really wear out a flash chip.

The good thing however is that the firmware upgrade comes with a commandline flashing utility (W200FW35.exe), so you might as well use that, and flash the firmware manually. Having a commandline utility may also help with a flashrom implementation in the future, since it should be fairly straightforward to reverse engineer. The one thing you have to be careful of however, when using W200FW35.exe, is that you need to select the right SPI-Flash-ROM device type for flashing, and that the included batch files, which are set to a specific type, may not match the one you have.

Issuing W200FW35.exe /? will provide more information. First, in an elevated command prompt, start by issuing /srom ? to identify your chip and check that the flashing utility can access it:

C:\uPD720200_uPD720200A_FW_Updater>W200FW35.exe /srom ?
Bus:0x04 Device:0x00 Function:0x00
This Device is uPD720200A(Revision 4).
    W25X10BV/20BV/40BV(WINBOND)/EN25F05/10/20/40/(EON)/A25L512/010/020/040(AMIC) Type : 3, PageSize = 0x100, Chip Erase = 0xC7
Then, if you have a uPD720200A based controller, you probably want to issue:
W200FW35.exe /srom 0 /dump backup.mem
W200FW35.exe /srom 0 /write F401502.MEM cfg.ini
/srom 0 above ensures that the flash type will be autoselected. Or, in this case, we could have used /srom 3. If you use an srom parameter other than 0, be mindful that using the wrong type will result in both garbage in and garbage out.
As digitaldiatribe also points out, if you have more than one Renesas chip on your system, you will also need to add the /address switch to select the right one. Update: As pointed out by Eluder, the latest firmwares from Renesas no longer come with the W200FW35.exe commandline utility. It can still be obtained from the "3.0.2.8.0.8 (uPD720200) & 4.0.1.5.0.2 (uPD720200a)" archive available on the link provided above.