Compiling FreeDOS on Windows, using VMWare player

Might as well follow up on the previous post...

Here is how I set up an environment to be able to recompile the FreeDOS kernel, on Windows 7 x64, using the VMWare player.

1. Download the "FreeDOS Base CD" from here. Yes, I am well aware that there is also a "FreeDOS Base CD with source code" available, but we are going to recompile the latest kernel from SVN and in case I decide to upload my VMWare image to help y'all, I want the disk to be as little bloated as possible
2. Create a new VMWare image in Player and select "I will install the OS later" when prompted. A 512MB (0.5 GB) disk is all you need in size, and I'm probably being generous. On the other hand since we are going to compile, setting at least 512 MB RAM for the VM (rather than VMWare's default 16 MB) is probably a good idea. You can also remove virtual floppy and sound card as we don't need them
3. Set the VM to boot from the ISO you just downloaded in step 1, and start it. When prompted by FreeDOS, select "1) Continue to boot from FreeDOS from the CD-ROM" or press Enter, then select "1. Install to harddisk using FreeDOS SETUP (default)" and go through the installation steps. You may have to enter the VMWare BIOS (by pressing the F2 key) to make the CD-ROM first bootable device after partitioning in order to conplete the installation.
4. One installed, FreeDOS should boot from the hard drive and the first bad surprise you'll get is that the default of "2 - Load FreeDOS with EMM386+EMS and SHARE" results in "CONFIG.SYS error in line 24 - Illegal Instruction occured". You should select instead "1- Load FreeDOS with EMM386, no EMS (most UMBs), max RAM free" or "3 - Load FreeDOS including HIMEM XMS-memory driver"
5. Actually, let us start by fixing this inconvenience by issuing "edit fdconfig.sys" and changing the "MENUDEFAULT=2, 5" to "MENUDEFAULT=3, 5". Now that's better.
6. Power off the Virtual Machine and mount its virtual disk for transfer as highlighted in my previous post (or find another way that is convenient to you to transfer files to the disk)
7. Download the latest DOS 32 bit nasm from https://sourceforge.net/projects/nasm/files/DOS%2032-bit%20binaries/ and extract nasm.exe as well as cwsdpmi.exe from the archive to a \BIN directory on the FreeDOS hard drive
8. Download the latest DOS/i386 version of UPX from http://upx.sourceforge.net and extract upx.exe to \BIN
9. Download the latest open-watcom-c-dos-x.y.exe from http://ftp.openwatcom.org/ftp/ and copy it to your virtual FreeDOS hard drive (where is irrelevant)
10. Fetch the latest FreeDOS kernel using svn (eg. using TortoiseSVN on Windows) from https://freedos.svn.sourceforge.net/svnroot/freedos/kernel/trunk and copy the whole kernel\ directory to \src\kernel on your FreeDOS disk
11. Boot the FreeDOS VM again, and now run the open-watcom-c-dos-x.y.exe from wherever you copied it (which may now be listed as open-w~1.exe). Install Open Watcom as "Selective installation" and in "Toolkits and other components" remove "Sample programs". Also in that section, select the "Helpfiles" sub-menu and remove the "DOS Hosted Help Files". The installation will take a little while, but at least it's unattended. At the end, tell Open Watcom NOT to modify the AUTOEXEC.BAT and CONFIG.SYS files
12. Edit AUTOEXEC.BAT and edit the PATH line to have the following:
    set PATH %dosdir%\bin;C:\BIN;C:\WATCOM\BINW
13. Also in AUTOEXEC.BAT, after the line lh doslfn, add:
    set INCLUDE=C:\WATCOM\H
set WATCOM=C:\WATCOM
set EDPATH=C:\WATCOM\EDDAT
set WIPFC=C:\WATCOM\WIPFC
14. Reboot the VM
15. Go to C:\src\kernel and issue "copy config.b config.bat"
16. Edit config.bat and update the following:
    set XNASM=c:\bin\nasm.exe
    Uncomment all the relevant WATCOM options in the file
    It is also recommended to modify config.bat to have:
    set XCPU=386
set XFAT=32
17. Issue the command: build
18. If you followed everything properly, you should end up with a successful build of KERNEL.SYS and other files in C:\src\kernel\bin
    Note that you can clean up a previous build by issuing clean

Bonus: For those who don't want to have to go through all these steps, please find a ready-to-use VMWare image here (Player 4.x, 86 MB, 7z compressed), created using the exact steps above.

How to get a vmdk mapped in Windows Explorer when using VMWare Player 4.x and Windows 7

VMWare... Yet another textbook story of a technological company that has become a bit too complacent with its users, possibly without even realizing it (Disclaimer: I have been a virtualization evangelist, as well as direct contributor to sales of VMWare products in the previous company I worked for, so I do feel somewhat entitled to exert criticism).

So, you have VMWare Player 4.x, installed on a Windows7 platform, and let's say you fancy recompiling FreeDOS in a virtualized environment. For this you have set up your VM, with a small virtual disk and everything, but since it's DOS, no shared folders or VMWare Tools will be available there. Yet you need a solution to exchange data in a convenient manner, between the Windows 7 host and the VM disk image. How are you going to do it then? Why, by mounting the vmdk as a virtual drive of course. And since VMWare is a Virtual hardware company, they made sure that such a staple of virtualization was as easy and as straightforward as possible, right?

Wrong. Below is what you actually need to go through:

1. The VMWare Player does not natively offer any tools to map/mount a virtual drive. You have to install extra software.
2. That software is the Virtual Disk Development Kit (presently in its version 5.5.1) which can be obtained here (after registration of course! It's not like you were about to sort this virtual disk mapping issue in 30 seconds anyway).
3. Once installed (by running bin\vstor2install.bat), and of course without any obvious mention of it (since officially it's an SDK, not an application plugin!), you will find out that the following becomes available when editing your disk properties:
   
   Brilliant: a GUI with the ability to map drives is exactly what we've been looking for.
4. Except, at least in Windows 7 x64, and if you run VMWare Player 4.x under your regular credentials, you will be greeted with the following error message "Error reading volume information. Please select another disk file":
   
   Same occurs if you try the vmware-mount.exe from from the VDK's bin directory (and this even if you open a command prompt as admin, as I have also found)
5. To avoid the "Error reading volume information..." message then, and be able to mount the disk, you must run VMWare Player as admin
6. HOWEVER, "Open drive in Windows Explorer after mapping" still will not work, because Explorer is running under your user credentials, and the drive was mounted under the Administrator's credentials. Thus, you must run your File Explorer as administrative.
7. Luckily, if you happen to run a proper File Manager (such as the most excellent Directory Opus, which I can't recommend enough as it's well worth its license), this isn't a major problem and you will actually get the friggin' vmdk mapped at long last, with the ability to transfer files. If you're stuck with the default Windows Explorer, running it as admin may or may not work —can't test, as on my machine, Directory Opus is the explorer that will be launched by default by VMWare)

Yay, at last VMWare may actually prove useful in getting some work done. Now let's see if the FreeDOS guys also see it this way...

Update (2012.01.06): The FreeDOS people kindly indicated that vmsmount might also be able to help you out if you want to acces VMWare shares from DOS.

Rufus - The bootable USB Formatting Utility

It is my very great pleasure to introduce Rufus (direct downloads here), my own GPL'd version of a bootable USB formatting utility (DOS + ISO).

If you have been using the old HP Utility to create DOS bootable USB, you can throw that old thing away! In a small executable, and with no requirement for an installer, Rufus offers you a much better and up to date interface, with better features, and a DOS creation that doesn't rely on external files. It can also create bootable USB from ISO images. Plus you will find welcome additional features, such as the ability to check your USB stick for bad blocks. Best of all, and as you have come to expect from this site, it is 100% Free Software.

Please make sure you check the official Rufus page, and stop looking further when you need a DOS bootable USB stick. Of course, Rufus is compatible with all versions of Windows starting with Windows XP and lets you use the always awesome FreeDOS alongside the rather old and not up to date Windows Millenium DOS.

Why are you doing this?

Well, the truth of the matter is that, after having used the HPUSBFW utility for some time, it turns out that I really can't stand proprietary software utilities (as well as Windows' glaring shortcomings), so I decided to create my own Open Source version of an equivalent tool. Also, the fact that the many people, who have taken a stab at creating their own DOS bootable USB formatting utility, decided to go closed source doesn't really help. Simple utilities should only be Open Source, period.

Some interesting technical details (or yet another annoying technical rant)

You'd think there wouldn't be much to formatting an USB flash drive for DOS on Windows, but you would be wrong. As I already explained, there's some reason why Windows doesn't do it natively. Also, you may be surprised to hear that Windows doesn't actually provide a public API to format a drive, and instead you have to hijack an undocumented one called FormatEx, and which can be found in fmifs.dll. Then you will find that FormatEx kind of destroys your partition table when you want to use LBA, so you need to fix it manually. There's also this whole business of allowed cluster sizes. And then there's all the usual traps, such as having the partition boot records needing to be patched on XP, because unlike Vista or later, it equates an USB Flash drive to a floppy, as well as Windows' somewhat mysterious handling of Physical Drives vs. Logical Volumes. One will let you access the MBR and the other one the Partition Boot Record, yet, you still need to hold a lock to the latter to be able to access its underlying sectors with the former. Straightforward, it is not.

If you are so inclined, you'll find Rufus' FormatThread() in format.c as a good starting point. Oh, and would be quite ungrateful if I didn't acknowledge other OSS projects, such as ms-sys (boot records handling) or e2fsprogs (bad blocks check) for providing some of the building blocks used by the utility, as well as the talented designers from PC-Unleashed for the Rufus icon.

Finally, for those interested, the acronym stands for "The Reliable USB Formatting Utility, with Source".

Enjoy!

IMPORTANT NOTE: If you want support, please use the Github issue tracker. Any requests for support in the comments will be left unanswered.

Finding which preprocessor macros are defined in gcc

gcc -dM -E -x c /dev/null

Can be quite useful. Also works for Clang.

Why Windows doesn't let you create an USB MS-DOS startup disk

Today, I'll provide an answer to the following question, which, if you're like me, must have irked you some when formatting an USB flash drive:

Yes why? Why is is that Windows has no problems creating an MS-DOS startup disk for the now useless floppy technology, but does not allow you to do so for the ubiquitous USB memory stick one?
It's not really like this is an impossible feat on Windows, as there exist quite a few tools that allow you to create one already (and I have now added my MUCH BETTER ONE to the mix). So why doesn't Microsoft do it?

If it ain't broke, don't fix it... But don't upgrade it either.

Ah, conservatism. I'm not talking about the political kind here, but rather its no less sinister computer equivalent: "Hey, we worked hard enough to get there. We don't see why we should take the risk of changing existing code, just so newcomers can reap the benefits."

You see, the way Windows creates an MS-DOS startup disk is not by doing anything smart, but by doing something lazy —which some also say is the second smartest thing to do after "smart"— such as including a 1:1 image of a 1.4MB floppy disk in a DLL (diskcopy.dll) and just splashing it wholesale onto a floppy when you request a startup disk.
Oh and yes, that's L-A-Z-Y-lazy since, if you actually have a look at the image file, you'll see that Microsoft didn't even spend much time creating a well thought-out one, but instead picked up the first Windows Millenium Edition (Me) startup disk they had lying around and just proceeded to "delete" stuff until it looked OK enough. The proof of it is that the image still contains much of the "deleted" content from the original disk and for a good read on how exactly the Windows Me Startup disk was "emasculated", you are very much invited to read Daniel Sedory's entry on the subject.

Of course, this whole technique of imaging a whole floppy does fall apart as soon as you're trying to create an unknown size USB bootdisk and this, my little children, is why Windows won't let you create DOS bootable USB flash drives to this day. End of story. Good night.

But wait, there's more!

Alright, alright, I hear you: that was a pretty underwhelming bedtime story. So of course there's more. And to answer your follow up question on why Microsoft didn't just compensate for the issue above by dropping the floppy image from diskcopy.dll and simply include the files themselves, we'll press on with a little bit of history.

As stated previously, the "DOS" files included in the image file that Windows uses when creating an "MS-DOS" startup disk actually come from a Windows Me boot disk, which, amongst its many disappointments, is also well known for crippling the DOS version it came with. As Wikipedia indicates, "One of the most publicized changes from Windows 98 is that Windows Me does not include real mode MS-DOS". Furthermore, departing from any MS-DOS that came prior, "the Autoexec.bat and Config.sys files are used only to set global environment variables". Thus DOS 8.0, which is the one included with Windows Millenium, is indeed seen by many as crippled, whereas DOS 7.1, a.k.a. the one from Windows 98 SE, is often labelled the "last good DOS".

But much more relevant to the USB boot issue is that the Windows Me DOS is not actually able to produce a command prompt on anything else but a floppy media. That is to say, even if you were to create all the required boot records on an USB drive, and then copy the unmodified COMMAND.COM and IO.SYS from the diskcopy.dll floppy image, your drive will never boot. You can try this for yourself, if you have an existing bootable stick, as 7-zip will happily extract the files from diskcopy.dll if you want to copy them over.

That still doesn't explain anything...

"OK fine," you say, "Windows Me is unusable but Microsoft should have the rights to their own DOS stuff anyway, so why don't they either just use the DOS files from Windows 98, or produce an uncrippled new version of the WinMe DOS, and go with that?". That's actually a very valid question, which people have only been able to speculate onto this day. Technically and most likely legally, there doesn't exist a reason why Microsoft shouldn't be able to provide a proper set of DOS files to boot an USB stick with. Furthermore, and this is the trick people use when they want to legally provide MS-DOS files for USB boot on Windows, uncrippling the WinMe DOS files from the diskcopy.dll to get them working on USB is actually trivial and documented here. For the record, even the well known HPUSBFW tool, from HP will patch the COMMAND.COM and IO.SYS files according to the above, when provided with WinMe DOS files. Yet, Microsoft went as far as explicitly discouraging people from doing so...

Thus, the moral of our story is that, while there is no real technical limitation to it, there is no satisfying explanation as to why Microsoft decided to inconvenience millions of their users by not allowing to create a DOS bootable USB drive. However, if you keep watching this blog, you will soon be provided with a 100% Open Source tool that does just that (and more!), so stay tuned.

Bonus

By the way, if you get your hands on an older version of the HPUSBFW tool --the one that is about 400 KB in size, you should be advised that it contains, as a resource, a self extracting UPX compressed executable that will extract the Windows 98 DOS files. The utility itself is designed to only enable extraction of these files if specific HP USB hardware (VID:PID 03F0:0023) is detected, however that can easily be patched to enable any device (F0 03 75 15 → F0 03 90 90 & 23 00 75 0C → 23 90 90 0C) and one could use HPUSBFW as a standalone USB DOS boot utility.