2011-12-16

Compiling FreeDOS on Windows, using VMWare player

Might as well follow up on the previous post...

Here is how I set up an environment to be able to recompile the FreeDOS kernel, on Windows 7 x64, using the VMWare player.
  1. Download the "FreeDOS Base CD" from here. Yes, I am well aware that there is also a "FreeDOS Base CD with source code" available, but we are going to recompile the latest kernel from SVN and in case I decide to upload my VMWare image to help y'all, I want the disk to be as little bloated as possible
  2. Create a new VMWare image in Player and select "I will install the OS later" when prompted. A 512MB (0.5 GB) disk is all you need in size, and I'm probably being generous. On the other hand since we are going to compile, setting at least 512 MB RAM for the VM (rather than VMWare's default 16 MB) is probably a good idea. You can also remove virtual floppy and sound card as we don't need them
  3. Set the VM to boot from the ISO you just downloaded in step 1, and start it. When prompted by FreeDOS, select "1) Continue to boot from FreeDOS from the CD-ROM" or press Enter, then select "1. Install to harddisk using FreeDOS SETUP (default)" and go through the installation steps. You may have to enter the VMWare BIOS (by pressing the F2 key) to make the CD-ROM first bootable device after partitioning in order to conplete the installation.
  4. One installed, FreeDOS should boot from the hard drive and the first bad surprise you'll get is that the default of "2 - Load FreeDOS with EMM386+EMS and SHARE" results in "CONFIG.SYS error in line 24 - Illegal Instruction occured". You should select instead "1- Load FreeDOS with EMM386, no EMS (most UMBs), max RAM free" or "3 - Load FreeDOS including HIMEM XMS-memory driver"
  5. Actually, let us start by fixing this inconvenience by issuing "edit fdconfig.sys" and changing the "MENUDEFAULT=2, 5" to "MENUDEFAULT=3, 5". Now that's better.
  6. Power off the Virtual Machine and mount its virtual disk for transfer as highlighted in my previous post (or find another way that is convenient to you to transfer files to the disk)
  7. Download the latest DOS 32 bit nasm from https://sourceforge.net/projects/nasm/files/DOS%2032-bit%20binaries/ and extract nasm.exe as well as cwsdpmi.exe from the archive to a \BIN directory on the FreeDOS hard drive
  8. Download the latest DOS/i386 version of UPX from http://upx.sourceforge.net and extract upx.exe to \BIN
  9. Download the latest open-watcom-c-dos-x.y.exe from http://ftp.openwatcom.org/ftp/ and copy it to your virtual FreeDOS hard drive (where is irrelevant)
  10. Fetch the latest FreeDOS kernel using svn (eg. using TortoiseSVN on Windows) from https://freedos.svn.sourceforge.net/svnroot/freedos/kernel/trunk and copy the whole kernel\ directory to \src\kernel on your FreeDOS disk
  11. Boot the FreeDOS VM again, and now run the open-watcom-c-dos-x.y.exe from wherever you copied it (which may now be listed as open-w~1.exe). Install Open Watcom as "Selective installation" and in "Toolkits and other components" remove "Sample programs". Also in that section, select the "Helpfiles" sub-menu and remove the "DOS Hosted Help Files". The installation will take a little while, but at least it's unattended. At the end, tell Open Watcom NOT to modify the AUTOEXEC.BAT and CONFIG.SYS files
  12. Edit AUTOEXEC.BAT and edit the PATH line to have the following:
    set PATH %dosdir%\bin;C:\BIN;C:\WATCOM\BINW
  13. Also in AUTOEXEC.BAT, after the line lh doslfn, add:
    set INCLUDE=C:\WATCOM\H
    set WATCOM=C:\WATCOM
    set EDPATH=C:\WATCOM\EDDAT
    set WIPFC=C:\WATCOM\WIPFC
  14. Reboot the VM
  15. Go to C:\src\kernel and issue "copy config.b config.bat"
  16. Edit config.bat and update the following:
    set XNASM=c:\bin\nasm.exe
    Uncomment all the relevant WATCOM options in the file
    It is also recommended to modify config.bat to have:
    set XCPU=386
    set XFAT=32
  17. Issue the command: build
  18. If you followed everything properly, you should end up with a successful build of KERNEL.SYS and other files in C:\src\kernel\bin
    Note that you can clean up a previous build by issuing clean
Bonus: For those who don't want to have to go through all these steps, please find a ready-to-use VMWare image here (Player 4.x, 86 MB, 7z compressed), created using the exact steps above.

How to get a vmdk mapped in Windows Explorer when using VMWare Player 4.x and Windows 7

VMWare... Yet another textbook story of a technological company that has become a bit too complacent with its users, possibly without even realizing it (Disclaimer: I have been a virtualization evangelist, as well as direct contributor to sales of VMWare products in the previous company I worked for, so I do feel somewhat entitled to exert criticism).

So, you have VMWare Player 4.x, installed on a Windows7 platform, and let's say you fancy recompiling FreeDOS in a virtualized environment. For this you have set up your VM, with a small virtual disk and everything, but since it's DOS, no shared folders or VMWare Tools will be available there. Yet you need a solution to exchange data in a convenient manner, between the Windows 7 host and the VM disk image. How are you going to do it then? Why, by mounting the vmdk as a virtual drive of course. And since VMWare is a Virtual hardware company, they made sure that such a staple of virtualization was as easy and as straightforward as possible, right?

Wrong. Below is what you actually need to go through:
  1. The VMWare Player does not natively offer any tools to map/mount a virtual drive. You have to install extra software.
  2. That software is the Virtual Disk Development Kit (presently in its version 5.5.1) which can be obtained here (after registration of course! It's not like you were about to sort this virtual disk mapping issue in 30 seconds anyway).
  3. Once installed (by running bin\vstor2install.bat), and of course without any obvious mention of it (since officially it's an SDK, not an application plugin!), you will find out that the following becomes available when editing your disk properties:

    Brilliant: a GUI with the ability to map drives is exactly what we've been looking for.
  4. Except, at least in Windows 7 x64, and if you run VMWare Player 4.x under your regular credentials, you will be greeted with the following error message "Error reading volume information. Please select another disk file":

    Same occurs if you try the vmware-mount.exe from from the VDK's bin directory (and this even if you open a command prompt as admin, as I have also found)
  5. To avoid the "Error reading volume information..." message then, and be able to mount the disk, you must run VMWare Player as admin
  6. HOWEVER, "Open drive in Windows Explorer after mapping" still will not work, because Explorer is running under your user credentials, and the drive was mounted under the Administrator's credentials. Thus, you must run your File Explorer as administrative.
  7. Luckily, if you happen to run a proper File Manager (such as the most excellent Directory Opus, which I can't recommend enough as it's well worth its license), this isn't a major problem and you will actually get the friggin' vmdk mapped at long last, with the ability to transfer files. If you're stuck with the default Windows Explorer, running it as admin may or may not work —can't test, as on my machine, Directory Opus is the explorer that will be launched by default by VMWare)
Yay, at last VMWare may actually prove useful in getting some work done. Now let's see if the FreeDOS guys also see it this way...

Update (2012.01.06): The FreeDOS people kindly indicated that vmsmount might also be able to help you out if you want to acces VMWare shares from DOS.

2011-12-14

Rufus - The bootable USB Formatting Utility

It is my very great pleasure to introduce Rufus (direct downloads here), my own GPL'd version of a bootable USB formatting utility (DOS + ISO).

https://rufus.ie


If you have been using the old HP Utility to create DOS bootable USB, you can throw that old thing away! In a small executable, and with no requirement for an installer, Rufus offers you a much better and up to date interface, with better features, and a DOS creation that doesn't rely on external files. It can also create bootable USB from ISO images. Plus you will find welcome additional features, such as the ability to check your USB stick for bad blocks. Best of all, and as you have come to expect from this site, it is 100% Free Software.

Please make sure you check the official Rufus page, and stop looking further when you need a DOS bootable USB stick. Of course, Rufus is compatible with all versions of Windows starting with Windows XP and lets you use the always awesome FreeDOS alongside the rather old and not up to date Windows Millenium DOS.

Why are you doing this?

Well, the truth of the matter is that, after having used the HPUSBFW utility for some time, it turns out that I really can't stand proprietary software utilities (as well as Windows' glaring shortcomings), so I decided to create my own Open Source version of an equivalent tool. Also, the fact that the many people, who have taken a stab at creating their own DOS bootable USB formatting utility, decided to go closed source doesn't really help. Simple utilities should only be Open Source, period.

Some interesting technical details (or yet another annoying technical rant)

You'd think there wouldn't be much to formatting an USB flash drive for DOS on Windows, but you would be wrong. As I already explained, there's some reason why Windows doesn't do it natively. Also, you may be surprised to hear that Windows doesn't actually provide a public API to format a drive, and instead you have to hijack an undocumented one called FormatEx, and which can be found in fmifs.dll. Then you will find that FormatEx kind of destroys your partition table when you want to use LBA, so you need to fix it manually. There's also this whole business of allowed cluster sizes. And then there's all the usual traps, such as having the partition boot records needing to be patched on XP, because unlike Vista or later, it equates an USB Flash drive to a floppy, as well as Windows' somewhat mysterious handling of Physical Drives vs. Logical Volumes. One will let you access the MBR and the other one the Partition Boot Record, yet, you still need to hold a lock to the latter to be able to access its underlying sectors with the former. Straightforward, it is not.

If you are so inclined, you'll find Rufus' FormatThread() in format.c as a good starting point. Oh, and would be quite ungrateful if I didn't acknowledge other OSS projects, such as ms-sys (boot records handling) or e2fsprogs (bad blocks check) for providing some of the building blocks used by the utility, as well as the talented designers from PC-Unleashed for the Rufus icon.

Finally, for those interested, the acronym stands for "The Reliable USB Formatting Utility (with Source").

Enjoy!

IMPORTANT NOTE: If you want support, please use the Github issue tracker. Any requests for support in the comments will be left unanswered.

2011-12-12

Finding which preprocessor macros are defined in gcc

gcc -dM -E -x c /dev/null
Can be quite useful. Also works for Clang.

2011-12-02

Why Windows doesn't let you create an USB MS-DOS startup disk

Today, I'll provide an answer to the following question, which, if you're like me, must have irked you some when formatting an USB flash drive:



Yes why? Why is is that Windows has no problems creating an MS-DOS startup disk for the now useless floppy technology, but does not allow you to do so for the ubiquitous USB memory stick one?
It's not really like this is an impossible feat on Windows, as there exist quite a few tools that allow you to create one already (and I have now added my MUCH BETTER ONE to the mix). So why doesn't Microsoft do it?

If it ain't broke, don't fix it... But don't upgrade it either.

Ah, conservatism. I'm not talking about the political kind here, but rather its no less sinister computer equivalent: "Hey, we worked hard enough to get there. We don't see why we should take the risk of changing existing code, just so newcomers can reap the benefits."

You see, the way Windows creates an MS-DOS startup disk is not by doing anything smart, but by doing something lazy —which some also say is the second smartest thing to do after "smart"— such as including a 1:1 image of a 1.4MB floppy disk in a DLL (diskcopy.dll) and just splashing it wholesale onto a floppy when you request a startup disk.
Oh and yes, that's L-A-Z-Y-lazy since, if you actually have a look at the image file, you'll see that Microsoft didn't even spend much time creating a well thought-out one, but instead picked up the first Windows Millenium Edition (Me) startup disk they had lying around and just proceeded to "delete" stuff until it looked OK enough. The proof of it is that the image still contains much of the "deleted" content from the original disk and for a good read on how exactly the Windows Me Startup disk was "emasculated", you are very much invited to read Daniel Sedory's entry on the subject.

Of course, this whole technique of imaging a whole floppy does fall apart as soon as you're trying to create an unknown size USB bootdisk and this, my little children, is why Windows won't let you create DOS bootable USB flash drives to this day. End of story. Good night.

But wait, there's more!

Alright, alright, I hear you: that was a pretty underwhelming bedtime story. So of course there's more. And to answer your follow up question on why Microsoft didn't just compensate for the issue above by dropping the floppy image from diskcopy.dll and simply include the files themselves, we'll press on with a little bit of history.

As stated previously, the "DOS" files included in the image file that Windows uses when creating an "MS-DOS" startup disk actually come from a Windows Me boot disk, which, amongst its many disappointments, is also well known for crippling the DOS version it came with. As Wikipedia indicates, "One of the most publicized changes from Windows 98 is that Windows Me does not include real mode MS-DOS". Furthermore, departing from any MS-DOS that came prior, "the Autoexec.bat and Config.sys files are used only to set global environment variables". Thus DOS 8.0, which is the one included with Windows Millenium, is indeed seen by many as crippled, whereas DOS 7.1, a.k.a. the one from Windows 98 SE, is often labelled the "last good DOS".

But much more relevant to the USB boot issue is that the Windows Me DOS is not actually able to produce a command prompt on anything else but a floppy media. That is to say, even if you were to create all the required boot records on an USB drive, and then copy the unmodified COMMAND.COM and IO.SYS from the diskcopy.dll floppy image, your drive will never boot. You can try this for yourself, if you have an existing bootable stick, as 7-zip will happily extract the files from diskcopy.dll if you want to copy them over.

That still doesn't explain anything...

"OK fine," you say, "Windows Me is unusable but Microsoft should have the rights to their own DOS stuff anyway, so why don't they either just use the DOS files from Windows 98, or produce an uncrippled new version of the WinMe DOS, and go with that?". That's actually a very valid question, which people have only been able to speculate onto this day. Technically and most likely legally, there doesn't exist a reason why Microsoft shouldn't be able to provide a proper set of DOS files to boot an USB stick with. Furthermore, and this is the trick people use when they want to legally provide MS-DOS files for USB boot on Windows, uncrippling the WinMe DOS files from the diskcopy.dll to get them working on USB is actually trivial and documented here. For the record, even the well known HPUSBFW tool, from HP will patch the COMMAND.COM and IO.SYS files according to the above, when provided with WinMe DOS files. Yet, Microsoft went as far as explicitly discouraging people from doing so...

Thus, the moral of our story is that, while there is no real technical limitation to it, there is no satisfying explanation as to why Microsoft decided to inconvenience millions of their users by not allowing to create a DOS bootable USB drive. However, if you keep watching this blog, you will soon be provided with a 100% Open Source tool that does just that (and more!), so stay tuned.

Bonus

By the way, if you get your hands on an older version of the HPUSBFW tool --the one that is about 400 KB in size, you should be advised that it contains, as a resource, a self extracting UPX compressed executable that will extract the Windows 98 DOS files. The utility itself is designed to only enable extraction of these files if specific HP USB hardware (VID:PID 03F0:0023) is detected, however that can easily be patched to enable any device (F0 03 75 15F0 03 90 90 & 23 00 75 0C23 90 90 0C) and one could use HPUSBFW as a standalone USB DOS boot utility.

2011-11-14

Using an older version of VMWare Tools

Lost enough time with this already so I'll be short. First of all, yes, sometimes the latest version of the VMWare Tools can fail and you want to downgrade. I'm looking at you VMWare Tools 8.8.0 from VMWare Player 4.0.0 using a Windows 2000 image! And as you may have already found out, when that occurs, the VMWare Software as well as the VMWare community are less than helpful.

In case you're running the VMWare Player on Windows, to emulate another Windows virtual platform, here's what you should know:
  1. After they are downloaded by the Player, the VMWare tools are installed, as a windows.iso file, in your VMWare installation directory (typically C:\Program Files (x86)\VMWare\VMWare Player\). It is that ISO image that is mounted in the virtual machine during installation.
  2. You can obtain older versions of the VMWare Tools by navigating the not-intended-for-users VMWare Software Update directory, starting at http://softwareupdate.vmware.com/cds/vmw-desktop/player/.
  3. The VMWare Tools download is then found in <version>/<build>/windows/packages/ (eg. http://softwareupdate.vmware.com/cds/vmw-desktop/player/3.1.5/491717/windows/packages/ )
  4. Of course the download is not provided as an ISO, because that would be too easy. Instead, it's a tar of an exe that silently extracts and copies (overwrites) a new windows.iso in the VMWare directory. Thus, before you run that exe, you may want to rename your old windows.iso to something else, so that you can both tell whether the downgrade process completed successfully and have the ability to switch back between ISOs if needed.

2011-11-10

cathash

cathash is a multiplatform MS CAT/Authenticode SHA-1 generation, inspired by cathash by Michel I. Gallant, but not using CryptCATAdminCalcHashFromFileHandle, and thus able to run on UNIX platforms, including big-endian ones.

It is intended for the computation of the custom SHA-1, used by Microsoft and others, to verify the authenticity of Windows executable (or regular files), using the algorithm detailed at the end of the "Windows Authenticode Portable Executable Signature Format" specifications.

The utility was heavily validated against Windows system files (32 and 64 bit), MSVC generated files, MinGW/MinGW-w64 generated files and so on, so it is expected to be fully compliant with the MS SHA-1.
If compiled on Windows, and unless you comment the VALIDATE_HASH define, the program will also validate its computation against the one from CryptCATAdminCalcHashFromFileHandle, for extra safety.

By the way, if you are interested in producing your own implementation, you may want to note that the MS specs omit the fact that the optional extra PE data, starting at SUM_OF_BYTES_HASHED, needs to be padded to the next 8 byte boundary for hashing.

Outside of its upcoming libwdi usage, this utility may come handy for UNIX users needing to validate the authenticity of Windows files.
The source, which comes along with a (signed) 32 bit Windows executable can either be downloaded here (direct link) or here (SourceForge).
You can also directly access cathash.c, to compile it using any of Visual Studio, WDK, MinGW, cygwin or UNIX based gcc.

IMPORTANT NOTE: As reported by Abid Bhat, some Windows XP system files, such as C:\Windows\system32\drivers\update.sys, C:\Windows\Driver Cache\i386\sp3.cab or C:\Windows\Driver Cache\i386\driver.cab, are not getting the expected hash when computed by the current version of cathash. I'll look into it when I get a chance, but it may take a while...

2011-11-07

LINK : warning LNK4044: unrecognized option '/lib'; ignored

Error above from the WDK driving you crazy? And when you try link /lib everything is fine?

Well, How about removing that /nologo you have before /lib. Better now? And yes, I agree, whoever designed the MS linker to be this mind-numbingly picky about its option order should get a well deserved pie in face:
E:\WinDDK\7600.16385.0>link /lib /nologo

E:\WinDDK\7600.16385.0>link /nologo /lib
LINK : warning LNK4044: unrecognized option '/lib'; ignored
LINK : warning LNK4001: no object files specified; libraries used
LINK : warning LNK4068: /MACHINE not specified; defaulting to X86
LINK : fatal error LNK1561: entry point must be defined

E:\WinDDK\7600.16385.0>

2011-11-05

bin2coff

As part of the ongoing effort to add 7z compression to libwdi, I am publishing bin2coff, which is an improved version of the tool of the same name by Vortex. bin2coff is aimed at generating MS COFF object files that are simple wrappers for binary data, and that can be used by the MS (Visual Studio, WDK), MinGW (MinGW32, MinGW-w64) and other linker tools. Such a tool can be especially useful if you want to generate a library that contains a large chunk binary data. For instance, and this is what I plan to use it for in libwdi, you can use it to embed a complete 7z archive in a cross-compiled Win32 static library file, which you will then be able to extract at runtime.

What this version of bin2coff brings is 64 bit compatibility (MinGW-w64), the provision of a size variable besides the binary data, as well as the full sourcecode. A 32 bit Windows executable is also included in the archive, which you can either download here (direct link) or here (SourceForge).
Usage: bin2coff bin obj [label] [64bit]

  bin  : source binary data
  obj  : target object file, in MS COFF format.
  label: identifier for the extern data. If not provided, the name of the
         binary file without extension is used.
  64bit: produce a 64 bit compatible object - symbols are generated without
         leading underscores and machine type is set to x86_x64.

With your linker set properly, typical access from a C source is:

    extern uint8_t  label[]     /* binary data         */
    extern uint32_t label_size  /* size of binary data */

Addon:
If, for some foolish reason, or just because you like it when things blow up in your face, you want to integrate bin2coff generation with libtool, you will find that you need to generate a .lo wrapper. In that case, the following Makefile excerpt might be of interest to you:
wdi_data.lo: bin2coff
# $* is the target without extension - here 'wdi_data'
 bin2coff $*.7z $*.o $(bin2coff_OPTIONS)
# libtool won't let us get away with a mere .o - we have to generate a .lo wrapper.
# What's more, the first comment from the .lo is mandatory and must contain the libtool version
 @echo "# Generated by $(shell $(LIBTOOL) --version | head -n 1)" > $@
 @echo "pic_object='$*.o'" >> $@
 @echo "non_pic_object='$*.o'" >> $@

2011-11-03

Access denied and Windows 7

One of the annoying aspects of using NTFS is that, from time to time (eg. if Windows 8 preview corrupted your Windows 7 partition so bad that you had to reinstall the whole OS - see post #7), Windows will decide that you are no longer entitled to access your files, and while performing some operation or other you'll get something like:
There's nothing more infuriating in terms of computing than being denied access to files you should have access to.

The solution? Open an elevated command prompt and issue:
takeown /R  /F <Drive or directory>
Icacls <Drive or directory> /reset /T

2011-10-25

Building and running Clang static-analyzer on Windows/MinGW

With the Clang static-analyzer becoming more and more popular these days, MinGW users on Windows might be looking for some way to also bring the Clang goodness to their shores.

However, well, let's just say that the LLVM documentation isn't that intuitive for newcomers, especially if you were expecting to be able to download a nice Windows binary package and roll. This quick recipe on setting up a complete Clang static-analyzer environment for MinGW attempts to remedy that. As usual, what is presented below worked on my environment at the time of the post - I make no guarantee that it will work elsewhere, especially if you choose to deviate from the guide.

Also, before you start, please be aware that this whole process will send you back about 11 GB in terms of disk space, so make sure you have adequate room.

  1. Install MinGW (32 bit!) using the latest msys-get-inst (just run the GUI installer) picking up the very latest packages.
    For the installation directory, I used D:\Clang, which I will assume to also be yours for for the rest of the guide - please modify accordingly. With regards to the installation packages, select MinGW Compiler SuiteC compiler (which should be selected by default), MinGW Compiler SuiteC++ compiler, and MinGW Developer Toolkit to ensure that binutils and all the necessary tools to build clang, except Python, are available. You should also select MSYS Basic System, so that a shell is available. You probably also want to play it safe by picking up the pre-packaged repository catalog for MinGW. The retrieval and installation of the packages will take a few minutes.
  2. Download Python v2.x (32 bit installer) from http://www.python.org/download/.
    Use D:\Clang\bin as the installation directory and unselect everything but Python Interpreter and Libraries, as this is all you really need from Python.
    IMPORTANT: Using python 3.2.2 produced an error on my machine during the build, so make sure you pick v2.x.
  3. [OBSOLETE: You can skip to the next step as per comments below]
    Download the latest (Experimental) Clang Binaries for Mingw32/x86 from http://llvm.org/releases/download.html. Extract its content (bin/, include/ ...) into D:\Clang, overwriting any existing files. We do not care about headers being overwritten as this setup will be used for Clang and Clang only. Always make sure llvm-gcc is installed in the /bin directory of MinGW, unless you want to see a CreateProcess() error when running it (or want to have to duplicate the binutils files which we already got installed from picking up the MinGW Developer Toolkit).
  4. After installing an Subversion client such as Tortoise-SVN if needed:
    • Checkout llvm from http://llvm.org/svn/llvm-project/llvm/trunk (this is a Subversion URI), into D:\Clang\msys\1.0\src\llvm
    • Checkout clang from http://llvm.org/svn/llvm-project/cfe/trunk, into D:\Clang\msys\1.0\src\llvm\tools\clang
      NB: you may have to checkout clang outside of llvm first, then move it to the right location if your Subversion client doesn't let you checkout a repository inside an existing repository.
  5. Open an msys shell by launching D:\Clang\msys\1.0\msys.bat and issue the following commands:
    cd /src
    mkdir build
    cd build
    export CC=gcc
    export CXX=g++
    ../llvm/configure --disable-docs --enable-optimized --enable-targets=x86,x86_64 --prefix=/mingw
    make -j2
    The --prefix ensures that our files will be installed in the bin directory of MinGW and the two export lines avoid the need to install the Clang binaries to be able to recompile (As per now obsolete Step 3). Note that, in the msys shell, /mingw always point to the root directory where you installed MinGW (here D:\clang) regardless of the name you used. I would strongly advise against using a different directory, or not specifying the --prefix option (in which case /usr/local will be used as default), as I ran into issues when I tried (the static-analyzer failed to locate <stdio.h> when parsing sources). As usual, the -j2, which specifies the number of jobs to run in parallel, is there to make use of as many cores as we have (here two). Adjust according to your platform.
    Oh, and if you want to know why we use --enable-optimized, see the very end of this post...
  6. Go for a walk or something. Some of the libraries and executables are huge (hundreds of MB in size) so this whole process will take a while. During the various compilation runs I conducted, I chanced in one that froze (one make process using 100% of the CPU and never completing) and another where Windows reported "llvm-tblgen has stopped working", so don't be surprised if the compilation doesn't work quite right first time 'round -- it's fairly heavy on resources.
  7. If the compilation completed, then you can run:
    make install -j2
    Note that this operation will take some time too!
  8. Copy all the content of D:\Clang\msys\1.0\src\llvm\tools\clang\tools\scan-build, except c++-analyzer to D:\Clang\bin.
    Copy all the content of D:\Clang\msys\1.0\src\llvm\tools\clang\tools\scan-view to D:\Clang\bin.
  9. Issue the command:
    ln -s /mingw/bin/ccc-analyzer /mingw/bin/c++-analyzer
    The reason we're re-creating this symbolic link (which on Windows ends up being a duplicate rather than a link anyway) is that the one from the file provided by LLVM only works on UNIX systems.
  10. Edit D:\clang\bin\scan-build, line 100, and remove the /bin there to have:
    my $ClangSB = Cwd::realpath("$RealBin/clang");
    If you don't do that, scan-build will look for clang in /mingw/bin/bin and obviously fail to locate it.
  11. Finally, since we had to override them, you shouldn't forget to undefine the CC and CXX environmental variables:
    export -n CC CXX
    unset CC CXX

Success looks like what?

$ cd /d/libusb-pbatard
$ scan-build ./autogen.sh
(...)
$ scan-build make
make  all-recursive
make[1]: Entering directory `/d/libusb-pbatard'
Making all in libusb
make[2]: Entering directory `/d/libusb-pbatard/libusb'
  CC     libusb_1_0_la-core.lo
core.c:1626:3: warning: Value stored to 'r' is never read
                r = 0;
                ^   ~
1 warning generated.
  CC     libusb_1_0_la-descriptor.lo
  CC     libusb_1_0_la-io.lo
io.c:1819:8: warning: Call to 'malloc' has an allocation size of 0 bytes
        fds = malloc(sizeof(*fds) * nfds);
              ^      ~~~~~~~~~~~~~~~~~~~
1 warning generated.
  CC     libusb_1_0_la-sync.lo
  CC     libusb_1_0_la-poll_windows.lo
  CC     libusb_1_0_la-windows_usb.lo
os/windows_usb.c:1218:26: warning: Value stored to 'discdevs' during its initialization is never read
        struct discovered_devs *discdevs = *_discdevs;
                                ^          ~~~~~~~~~~
os/windows_usb.c:1297:4: warning: Value stored to 'session_id' is never read
                        session_id = 0;
                        ^            ~
2 warnings generated.
  RC     libusb-1.0.lo
  CC     libusb_1_0_la-threads_windows.lo
os/threads_windows.c:115:2: warning: Value stored to 'prev_pos' is never read
        prev_pos = pos = NULL;
        ^          ~~~~~~~~~~
1 warning generated.
  CCLD   libusb-1.0.la
Creating library file: .libs/libusb-1.0.dll.a
make[2]: Leaving directory `/d/libusb-pbatard/libusb'
Making all in doc
make[2]: Entering directory `/d/libusb-pbatard/doc'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/d/libusb-pbatard/doc'
Making all in examples
make[2]: Entering directory `/d/libusb-pbatard/examples'
  CC     xusb.o
xusb.c: In function 'test_mass_storage':
xusb.c:443:9: warning: variable 'junk' set but not used [-Wunused-but-set-variable]
xusb.c:520:4: warning: Value stored to 'junk' is never read
                        junk = fwrite(data, 1, size, fd);
                        ^      ~~~~~~~~~~~~~~~~~~~~~~~~~
1 warning generated.
  CCLD   xusb.exe
./.libs/lt-xusb.c:692:11: warning: Value stored to 'len' during its initialization is never read
      int len = strlen (new_value);
          ^     ~~~~~~~~~~~~~~~~~~
1 warning generated.
  CC     lsusb.o
  CCLD   lsusb.exe
./.libs/lt-lsusb.c:692:11: warning: Value stored to 'len' during its initialization is never read
      int len = strlen (new_value);
          ^     ~~~~~~~~~~~~~~~~~~
1 warning generated.
make[2]: Leaving directory `/d/libusb-pbatard/examples'
make[2]: Entering directory `/d/libusb-pbatard'
make[2]: Leaving directory `/d/libusb-pbatard'
make[1]: Leaving directory `/d/libusb-pbatard'
scan-build: 8 bugs found.
scan-build: Run 'scan-view /tmp/scan-build-2011-10-24-1' to examine bug reports.
This is a typical result that Clang will (eventually - the analysis takes some time!) churn through. One false positive (that 'junk' variable is meant to be ignored, and only serves to eliminate a warning about not reading the return value from fwrite) and apart from that malloc code, everything looks fairly minor. Even on its very first report from clang, which of course we will act upon, libusb-1.0 on Windows doesn't appear to fare too bad!

Bonus:
If you followed the guide above, chances are you'll be using separate installations for the standard MinGW/MinGW-w64 and the Clangified ones. As such, you may want a title on the msys shell window that reflects that you are in a Clang environment, to identify which is which more easily.
To change the msys commandline window title, just edit your D:\clang\msys\1.0\etc\profile, locate the line:
export PS1='\[\033]0;$MSYSTEM:\w\007
and replace $MSYSTEM with the title you want.

Bonus 2:
Displayed after spending more than one hour of compilation, using gcc:
llvm[0]: ***** Completed Debug+Asserts Build
llvm[0]: ***** Note: Debug build can be 10 times slower than an
llvm[0]: ***** optimized build. Use make ENABLE_OPTIMIZED=1 to
llvm[0]: ***** make an optimized build. Alternatively you can
llvm[0]: ***** configure with --enable-optimized.
Now you tell me!?! Shouldn't that be the kind of info you want to display after someone issues configure?

2011-10-14

On Windows 8 and WCID devices

With the 1.2.0 release of libwdi, which, for those who don't know, is a library aimed at making the installation of Windows USB drivers as simple and as non intrusive for end-users as possible, one of the most exciting features brought forward is WCID device driver support.

WCID, which was brought to the limelight with Windows 8, and which stands for Windows Compatible ID, is a set of additional USB Descriptors one can add to a device firmware, to make the OS automatically identify which type of driver should be used, and potentially automatically install it. In other words, what WCID does is bring the Plug-and-Play functionality that exists for HID or Mass Storage, to any USB device.

Thus, as upcoming Windows 8 comes with a WinUSB WCID driver, if you currently produce USB devices that require driver installation, you should seriously consider converting them to WCID as it'll make life a lot easier for everybody.

This is also good news for libusb-1.0 users, as libusb-1.0 relies on WinUSB for generic access on Windows, which means that you can actually produce libusb-1.0 applications for Windows 8, that do not require you or your users to take care of any driver installation. And for earlier versions of Windows, up to Windows XP SP2, libwdi and its friendly installer application Zadig can make it very easy to proceed to the one-off installation of the WinUSB (or libusb-win32 or libusbK) WCID driver, to make those platforms behave in the same way as Windows 8.

More information:

2011-10-12

Avoiding Program Compatibility Assistant warnings on Windows

If you're developing applications on Windows Vista or later, you may end up in a situation where this rather annoying little fellow pops up after your program execution:


"What gives?, you say, my program's not an installer and it doesn't have anything like 'setup' or 'install' in the executable name! Why does Windows pester me and my users with such a warning?" The answer: have a look at your resource file (RC, etc) and especially your versioning strings, such as "FileDescription" or "ProductName". If there's any or part of the taboo keywords in there, such as "setup", "install", "update", Windows assumes that the executable is an installer, and also assumes the worst. Thus, the solution is to never use any or part of these keywords in any of the location that Windows will look for.

For more information on how Windows "heuristically" detects installation programs, have a look at the Installer Detection Technology paragraph from the following TechNet note.

2011-10-03

Flashing a NEC/RENESAS USB 3.0 uPD720200/uPD720200A firmware

It looks like NEC/Renesas are releasing steady releases of their popular USB 3.0 controllers drivers and firmwares. In case you have such a controller, you probably want to check the following page for the latest version.

One of the problems I found with the latest firmware update (4015) is that the GUI Renesas upgrade utility just didn't work for my controller, on Windows 7 x64. Even worse, it created commandline processes ad-infinitum, which, apart from obviously clogging up the system, is the last thing you want to see, as repeated erase/flash cycles can really wear out a flash chip.

The good thing however is that the firmware upgrade comes with a commandline flashing utility (W200FW35.exe), so you might as well use that, and flash the firmware manually. Having a commandline utility may also help with a flashrom implementation in the future, since it should be fairly straightforward to reverse engineer. The one thing you have to be careful of however, when using W200FW35.exe, is that you need to select the right SPI-Flash-ROM device type for flashing, and that the included batch files, which are set to a specific type, may not match the one you have.

Issuing W200FW35.exe /? will provide more information. First, in an elevated command prompt, start by issuing /srom ? to identify your chip and check that the flashing utility can access it:

C:\uPD720200_uPD720200A_FW_Updater>W200FW35.exe /srom ?
Bus:0x04 Device:0x00 Function:0x00
This Device is uPD720200A(Revision 4).
    W25X10BV/20BV/40BV(WINBOND)/EN25F05/10/20/40/(EON)/A25L512/010/020/040(AMIC) Type : 3, PageSize = 0x100, Chip Erase = 0xC7
Then, if you have a uPD720200A based controller, you probably want to issue:
W200FW35.exe /srom 0 /dump backup.mem
W200FW35.exe /srom 0 /write F401502.MEM cfg.ini
/srom 0 above ensures that the flash type will be autoselected. Or, in this case, we could have used /srom 3. If you use an srom parameter other than 0, be mindful that using the wrong type will result in both garbage in and garbage out.
As digitaldiatribe also points out, if you have more than one Renesas chip on your system, you will also need to add the /address switch to select the right one. Update: As pointed out by Eluder, the latest firmwares from Renesas no longer come with the W200FW35.exe commandline utility. It can still be obtained from the "3.0.2.8.0.8 (uPD720200) & 4.0.1.5.0.2 (uPD720200a)" archive available on the link provided above.

2011-08-22

Writing a Linux device driver module for kernels 2.6 or later with udev

This is a short tutorial for a sample character device module aimed at Linux kernels 2.6 and later (including 3.x and 4.x) using udev.
I thought might as well produce my own tutorial, after finding that most of the ones floating around the net are either lacking in terms of features, or not as up to date as could be with regards to the latest kernel advancements.

Some of the features demonstrated with this sample are:
  • creation of a character device as a module
  • use of udev, for the device instantiation
  • use of a kernel FIFO, for data queuing
  • use of a mutex, to prevent concurrent access to the same device
  • use of sysfs, as a means to add data to the queue
  • detection of access mode on open, to ensure read-only access
  • provision of module options during insmod, to enable debug output
Not bad for a short sample, eh?

Of course, this example is just there to help get you started, and does not replace a proper guide to Linux driver development, such as the freely available and strongly recommended "Linux Device Drivers - Third Edition" from O'Reilly.


What's the deal with udev?

If you have some basic knowledge of device drivers on UNIX systems, you probably are already aware that devices are instantiated into the dev/ directory through the use of a (major, minor) pair of integers. UNIX veterans will no doubt recall how they used to call on mknod to create their devices, and some UNIX systems still require that. Well into the 2.6 kernels, Linux was also more or less using the statically allocated major/minor approach, through its devfs implementation. At one stage however, kernel developers got slightly annoyed about having to coordinate the allocation of major/minor numbers, not being able to keep /dev as clean and as abstracted as they liked and also feared that Linux would run out of numbers to assign. Because of that, they developed a new approach to device instantiation called udev, which led them to eventually ditch devfs altogether in kernel 2.6.18. For more information about udev vs devfs, you may want to read Greg Kroah-Hartman's The final word from December 2003.

How this matters to you as a driver writer is that a lot of existing tutorials you will find rely on devfs or static allocation of a major and minor number, which doesn't apply to recent kernels. Thus, you want a tutorial that demonstrates how to use udev, such as the one provided here.
For the record, this sample driver was tested with kernel 3.0.3 as well as 2.6.39.


Device description

The origin of this device driver sample mostly stems from wanting to craft a virtual device to simulate an HDMI-CEC enabled TV, in order to facilitate the development of libcec/cecd. Basically, with such a driver, a libcec application can connect to a virtual device that plays a pre-recorder set of data, which avoids the need to have an actual HDMI-CEC device plugged in or even a development platform that has an HDMI port.

Since the driver above is fairly straightforward to write, it is a good candidate to serve as a base for a sample. In order to turn it into something that is generic enough for a tutorial, we did remove anything that was HDMI-CEC related and turned the driver into one that can store sequences of character data and then repeat them in the order they were received, pretty much as a FIFO for text string. Given this feature description, it makes a lot of sense to call our little device 'parrot'.

For this 'parrot' device then, what we need first is a buffer to store data. The smart approach here, rather than reinvent the wheel is to use a Kernel FIFO, as this is a feature provided by the kernel and specifically aimed at device drivers. This will greatly simplify our design and of course, if you use this sample as a base for a driver that does actual hardware I/O, knowing how to use a Kernel FIFO will no doubt come handy.

Now, while we obviously are going to use read accesses to the actual device (cat /dev/parrot) to read FIFO messages, we are not going to use write accesses to the device to fill the FIFO. Instead we will use sysfs, and make our device read-only.
This may sound counter-intuitive, aside from wanting to demonstrate the use of sysfs, but the reason is, in a typical virtual repeater device scenario, you would use separate FIFOs for read and write, or a FIFO for read and a sink for write, since you don't want data written by your application to the device to interfere the pre-defined scenario you have for readout. Of course, if you want to add a device write endpoint to our sample device, it is a simple matter of copy/pasting the read procedures, and modify them for write access.

Finally, because this is a sample, we will add some debug output which we will toggle using a module parameter on insmod as it is a good example to demonstrate how driver load options can be handled.


Source Walkthrough

The open()/close() calls should be fairly straightforward. The open() call demonstrates how one can use the struct file f_flags attribute to find out whether the caller is trying to use the device for read-only, write-only or read/write access. As indicated, we forcibly prevent write access, and return an error if the device is open write-only or read/write. Then we use the non-blocking mutex_trylock() and mutex_unlock(), along with a global device mutex, to prevent more than one process from accessing the device at any one time. This way, we won't have to worry about concurrent read accesses to our Kernel FIFO.

The device_read() call uses the kfifo_to_user() to copy data from the Kernel FIFO into userspace. One thing you must be mindful of, when writing device drivers, is that kernelspace and userspace are segregated so you can't just pass a kernel buffer pointer to a user application and expect it to work. Data must be duplicated to or from userspace. Hence the _to_user call. Now, because we store multiple messages in a flat FIFO, and don't use specific message terminators such as NUL, we need a table to tell us the length of each message. This is what the parrot_msg_len[] (circular) table provides. For this to work, we also need 2 indexes into it: one to point to the length of the next message to be read (parrot_msg_idx_rd), and one to point to the next available length entry for a write operation (parrot_msg_idx_wr). We could also have used _enqueue and _dequeue for the suffixes, as this is what you will generally find in a driver source.

The one_shot and message_read variables are used to alleviate a problem you will face if you use cat to read from the device. The problem with cat is that it will continue to issue read requests, until the device indicates that there is no more data to be read (with a read size of zero). Thus, if left unchecked, cat will deplete the whole FIFO, rather than return a single message. To avoid that, and because we know that cat will issue an open() call before attempting to read the device, we use the boolean message_read to tell us if this is our first read request since open or not. If it isn't the first request, and the one_shot module parameter is enabled (which is the default), we just return zero for all subsequent read attempts. This way, each time you issue a cat, you will at most read one message only.

With our open(), close() and read() calls properly defined, we can set the file operations structure up, which we do in the static struct file_operations fops. For more on the file operations which you can use for character devices, please see Chapter 3 of the Linux Device Drivers guide.

All the above takes care of our basic driver workhorse. Yet we still haven't provided any means for the user to populate the FIFO. This is done through the next subroutine called sys_add_to_fifo().

The first thing we do there is check for overflows on either the FIFO or the message length table. Then, we use kfifo_in() to copy the data provided by the user to the sysfs endpoint (see below). Note that, in this case, the sysfs mechanisms have already copied the data provided by the user into kernelspace, therefore using kfifo_in() just works, and there is not need to call kfifo_from_user(), as we would have had to do if we were writing a regular driver write procedure.

The other sysfs endpoint we provide is a FIFO reset facility, implemented in sys_reset, which is very straightforward.

With our sysfs functions defined, we can use the DEVICE_ATTR() macros to define the structure which we'll have to pass to the system during the module initialization. The S_IWUSR parameter indicates that we want users to have write access to these endpoints.

Now, we come to the last part of our driver implementation, with the actual module initialization and destruction. In module_init() the first thing we do, and this is the part where we rely on udev, is to dynamically allocate a major, through register_chrdev(). This call takes 3 parameters: The first one is either a specific major number, or 0 if you want the system to allocate one for you (our case), the second is a string identifier (which can be different from the name you will use in /dev) and the last parameter is a pointer to a file operations structure.

Next, we have a choice of creating a device associated with a class or a bus. For a simple sample such as ours, and given that our device is not tied to any hardware or protocol, creating a bus would be an overkill, so we will go with a class. For laymen, the difference between using a bus and a class is that you will get your devices listed in /sys/devices/<classname> in sysfs, instead of /sys/devices/virtual, and you will also have them under <classname> in /dev (though it is possible to achieve the same if you add a "/" in your device name when calling device_create()).

With a class established, we can then call device_create() which performs the actual instantiation of our device. The description of the device_create() parameters can be found here. We use the MKDEV() macro to create a dev_t out of the major we previously obtained, using 0 as the minor as we only have one device. It is at this stage that a "parrot_device" gets created in /dev, using the parameter provided for the name.

The next calls to device_create_file() are used to create the two sysfs endpoints, fifo and reset, out of the structures previously defined with DEVICE_ATTR(). Because we use a class, this results in the /sys/devices/virtual/parrot/parrot_device/fifo and /sys/devices/virtual/parrot/parrot_device/reset endpoints being created.

Finally we end the module_init() call with the remainder of the device initialization (FIFO, mutex, indexes).

The remainder of the code should be fairly explicit so I won't comment on it.


Source:

parrot_driver.h:
/*
 * Linux 2.6 and later 'parrot' sample device driver
 *
 * Copyright (c) 2011-2015, Pete Batard <pete@akeo.ie>
 *
 *
 * This program is free software; you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation; either version 2 of the License, or
 * (at your option) any later version.
 *
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
 * GNU General Public License for more details.
 *
 * You should have received a copy of the GNU General Public License
 * along with this program. If not, see <http://www.gnu.org/licenses/>.
 */

#define DEVICE_NAME "device"
#define CLASS_NAME "parrot"
#define PARROT_MSG_FIFO_SIZE 1024
#define PARROT_MSG_FIFO_MAX  128

#define AUTHOR "Pete Batard <pete@akeo.ie>"
#define DESCRIPTION "'parrot' sample device driver"
#define VERSION "1.2"

/* We'll use our own macros for printk */
#define dbg(format, arg...) do { if (debug) pr_info(CLASS_NAME ": %s: " format, __FUNCTION__, ## arg); } while (0)
#define err(format, arg...) pr_err(CLASS_NAME ": " format, ## arg)
#define info(format, arg...) pr_info(CLASS_NAME ": " format, ## arg)
#define warn(format, arg...) pr_warn(CLASS_NAME ": " format, ## arg)

parrot_driver.c:
/*
 * Linux 2.6 and later 'parrot' sample device driver
 *
 * Copyright (c) 2011-2015, Pete Batard <pete@akeo.ie>
 *
 *
 * This program is free software; you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation; either version 2 of the License, or
 * (at your option) any later version.
 *
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
 * GNU General Public License for more details.
 *
 * You should have received a copy of the GNU General Public License
 * along with this program. If not, see <http://www.gnu.org/licenses/>.
 */

#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/init.h>
#include <linux/slab.h>
#include <linux/fs.h>
#include <linux/device.h>
#include <linux/types.h>
#include <linux/mutex.h>
#include <linux/kfifo.h>
#include "parrot_driver.h"

/* Module information */
MODULE_AUTHOR(AUTHOR);
MODULE_DESCRIPTION(DESCRIPTION);
MODULE_VERSION(VERSION);
MODULE_LICENSE("GPL");

/* Device variables */
static struct class* parrot_class = NULL;
static struct device* parrot_device = NULL;
static int parrot_major;
/* Flag used with the one_shot mode */
static bool message_read;
/* A mutex will ensure that only one process accesses our device */
static DEFINE_MUTEX(parrot_device_mutex);
/* Use a Kernel FIFO for read operations */
static DECLARE_KFIFO(parrot_msg_fifo, char, PARROT_MSG_FIFO_SIZE);
/* This table keeps track of each message length in the FIFO */
static unsigned int parrot_msg_len[PARROT_MSG_FIFO_MAX];
/* Read and write index for the table above */
static int parrot_msg_idx_rd, parrot_msg_idx_wr;

/* Module parameters that can be provided on insmod */
static bool debug = false; /* print extra debug info */
module_param(debug, bool, S_IRUGO | S_IWUSR);
MODULE_PARM_DESC(debug, "enable debug info (default: false)");
static bool one_shot = true; /* only read a single message after open() */
module_param(one_shot, bool, S_IRUGO | S_IWUSR);
MODULE_PARM_DESC(one_shot, "disable the readout of multiple messages at once (default: true)");


static int parrot_device_open(struct inode* inode, struct file* filp)
{
 dbg("");

 /* Our sample device does not allow write access */
 if ( ((filp->f_flags & O_ACCMODE) == O_WRONLY)
   || ((filp->f_flags & O_ACCMODE) == O_RDWR) ) {
  warn("write access is prohibited\n");
  return -EACCES;
 }

 /* Ensure that only one process has access to our device at any one time
  * For more info on concurrent accesses, see http://lwn.net/images/pdf/LDD3/ch05.pdf */
 if (!mutex_trylock(&parrot_device_mutex)) {
  warn("another process is accessing the device\n");
  return -EBUSY;
 }

 message_read = false;
 return 0;
}

static int parrot_device_close(struct inode* inode, struct file* filp)
{
 dbg("");
 mutex_unlock(&parrot_device_mutex);
 return 0;
}

static ssize_t parrot_device_read(struct file* filp, char __user *buffer, size_t length, loff_t* offset)
{
 int retval;
 unsigned int copied;

 /* The default from 'cat' is to issue multiple reads until the FIFO is depleted
  * one_shot avoids that */
 if (one_shot && message_read) return 0;
 dbg("");

 if (kfifo_is_empty(&parrot_msg_fifo)) {
  dbg("no message in fifo\n");
  return 0;
 }

 retval = kfifo_to_user(&parrot_msg_fifo, buffer, parrot_msg_len[parrot_msg_idx_rd], &copied);
 /* Ignore short reads (but warn about them) */
 if (parrot_msg_len[parrot_msg_idx_rd] != copied) {
  warn("short read detected\n");
 }
 /* loop into the message length table */
 parrot_msg_idx_rd = (parrot_msg_idx_rd+1)%PARROT_MSG_FIFO_MAX;
 message_read = true;

 return retval ? retval : copied;
}

/* The file_operation scructure tells the kernel which device operations are handled.
 * For a list of available file operations, see http://lwn.net/images/pdf/LDD3/ch03.pdf */
static struct file_operations fops = {
 .read = parrot_device_read,
 .open = parrot_device_open,
 .release = parrot_device_close
};

/* Placing data into the read FIFO is done through sysfs */
static ssize_t sys_add_to_fifo(struct device* dev, struct device_attribute* attr, const char* buf, size_t count)
{
 unsigned int copied;

 dbg("");
 if (kfifo_avail(&parrot_msg_fifo) < count) {
  warn("not enough space left on fifo\n");
  return -ENOSPC;
 }
 if ((parrot_msg_idx_wr+1)%PARROT_MSG_FIFO_MAX == parrot_msg_idx_rd) {
  /* We've looped into our message length table */
  warn("message length table is full\n");
  return -ENOSPC;
 }

 /* The buffer is already in kernel space, so no need for ..._from_user() */
 copied = kfifo_in(&parrot_msg_fifo, buf, count);
 parrot_msg_len[parrot_msg_idx_wr] = copied;
 if (copied != count) {
  warn("short write detected\n");
 }
 parrot_msg_idx_wr = (parrot_msg_idx_wr+1)%PARROT_MSG_FIFO_MAX;

 return copied;
}

/* This sysfs entry resets the FIFO */
static ssize_t sys_reset(struct device* dev, struct device_attribute* attr, const char* buf, size_t count)
{
 dbg("");

 /* Ideally, we would have a mutex around the FIFO, to ensure that we don't reset while in use.
  * To keep this sample simple, and because this is a sysfs operation, we don't do that */
 kfifo_reset(&parrot_msg_fifo);
 parrot_msg_idx_rd = parrot_msg_idx_wr = 0;

 return count;
}

/* Declare the sysfs entries. The macros create instances of dev_attr_fifo and dev_attr_reset */
static DEVICE_ATTR(fifo, S_IWUSR, NULL, sys_add_to_fifo);
static DEVICE_ATTR(reset, S_IWUSR, NULL, sys_reset);

/* Module initialization and release */
static int __init parrot_module_init(void)
{
 int retval;
 dbg("");

 /* First, see if we can dynamically allocate a major for our device */
 parrot_major = register_chrdev(0, DEVICE_NAME, &fops);
 if (parrot_major < 0) {
  err("failed to register device: error %d\n", parrot_major);
  retval = parrot_major;
  goto failed_chrdevreg;
 }

 /* We can either tie our device to a bus (existing, or one that we create)
  * or use a "virtual" device class. For this example, we choose the latter */
 parrot_class = class_create(THIS_MODULE, CLASS_NAME);
 if (IS_ERR(parrot_class)) {
  err("failed to register device class '%s'\n", CLASS_NAME);
  retval = PTR_ERR(parrot_class);
  goto failed_classreg;
 }

 /* With a class, the easiest way to instantiate a device is to call device_create() */
 parrot_device = device_create(parrot_class, NULL, MKDEV(parrot_major, 0), NULL, CLASS_NAME "_" DEVICE_NAME);
 if (IS_ERR(parrot_device)) {
  err("failed to create device '%s_%s'\n", CLASS_NAME, DEVICE_NAME);
  retval = PTR_ERR(parrot_device);
  goto failed_devreg;
 }

 /* Now we can create the sysfs endpoints (don't care about errors).
  * dev_attr_fifo and dev_attr_reset come from the DEVICE_ATTR(...) earlier */
 retval = device_create_file(parrot_device, &dev_attr_fifo);
 if (retval < 0) {
  warn("failed to create write /sys endpoint - continuing without\n");
 }
 retval = device_create_file(parrot_device, &dev_attr_reset);
 if (retval < 0) {
  warn("failed to create reset /sys endpoint - continuing without\n");
 }

 mutex_init(&parrot_device_mutex);
 /* This device uses a Kernel FIFO for its read operation */
 INIT_KFIFO(parrot_msg_fifo);
 parrot_msg_idx_rd = parrot_msg_idx_wr = 0;

 return 0;

failed_devreg:
 class_destroy(parrot_class);
failed_classreg:
 unregister_chrdev(parrot_major, DEVICE_NAME);
failed_chrdevreg:
 return -1;
}

static void __exit parrot_module_exit(void)
{
 dbg("");
 device_remove_file(parrot_device, &dev_attr_fifo);
 device_remove_file(parrot_device, &dev_attr_reset);
 device_destroy(parrot_class, MKDEV(parrot_major, 0));
 class_destroy(parrot_class);
 unregister_chrdev(parrot_major, DEVICE_NAME);
}

/* Let the kernel know the calls for module init and exit */
module_init(parrot_module_init);
module_exit(parrot_module_exit);
The relevant Makefile is provided in the parrot source archive (link below).

Testing:
root@linux:/ext/src/parrot# make
make -C /usr/src/linux SUBDIRS=/mnt/hd/src/parrot modules
make[1]: Entering directory `/mnt/hd/src/linux-3.0.3'
  CC [M]  /mnt/hd/src/parrot/parrot_driver.o
  Building modules, stage 2.
  MODPOST 1 modules
  CC      /mnt/hd/src/parrot/parrot_driver.mod.o
  LD [M]  /mnt/hd/src/parrot/parrot_driver.ko
make[1]: Leaving directory `/mnt/hd/src/linux-3.0.3'
root@linux:/ext/src/parrot# insmod parrot_driver.ko debug=1
root@linux:/ext/src/parrot# lsmod
Module                  Size  Used by
parrot_driver           4393  0
root@linux:/ext/src/parrot# ls -lF /dev/parrot_device
crw------- 1 root root 251, 0 2011-08-22 22:04 /dev/parrot_device
root@linux:/ext/src/parrot# ls -lF /sys/devices/virtual/parrot/parrot_device/
total 0
-r--r--r-- 1 root root 4096 2011-08-22 22:05 dev
--w------- 1 root root 4096 2011-08-22 22:05 fifo
drwxr-xr-x 2 root root    0 2011-08-22 22:05 power/
--w------- 1 root root 4096 2011-08-22 22:05 reset
lrwxrwxrwx 1 root root    0 2011-08-22 22:04 subsystem -> ../../../../class/parrot/
-rw-r--r-- 1 root root 4096 2011-08-22 22:04 uevent
root@linux:/ext/src/parrot# echo "this should generate an error" > /dev/parrot_device
-bash: /dev/parrot_device: Permission denied
root@linux:/ext/src/parrot# echo "Yabba Dabba Doo" > /sys/devices/virtual/parrot/parrot_device/fifo
root@linux:/ext/src/parrot# echo "Yabba Dabba Daa" > /sys/devices/virtual/parrot/parrot_device/fifo
root@linux:/ext/src/parrot# cat /dev/parrot_device
Yabba Dabba Doo
root@linux:/ext/src/parrot# cat /dev/parrot_device
Yabba Dabba Daa
root@linux:/ext/src/parrot# echo "test" > /sys/devices/virtual/parrot/parrot_device/fifo
root@linux:/ext/src/parrot# echo "test" > /sys/devices/virtual/parrot/parrot_device/fifo
root@linux:/ext/src/parrot# echo 1 > /sys/devices/virtual/parrot/parrot_device/reset
root@linux:/ext/src/parrot# cat /dev/parrot_device
root@linux:/ext/src/parrot#

Links:

2011-08-12

Which controllers support USB 3.0 Debug?

As defined per section 7.6 of the xHCI specifications?

Unfortunately, none of the current ones from Renesas, VIA Labs or Fresco Logic do, and neither of these manufacturers are planning to add the capability on their existing controllers, through a firmware update for instance... Instead, they only have plans to provide it on their next generation controllers, which won't be available for some time (yes, I am aware that the new Renesas chips have been announced since 2011.03, but I have yet to see a PCI-E expansion card with one of those -- If you actually managed to get your hands on one, I'd like to hear from you!  
UPDATE 2011.08.29: Apparently those guys did... but as a world exclusive, since these cards won't be available till later on this year. Darn!).
UPDATE 2012.01.10: At long friggin' last, uPD720201 based PCIE cards, with debug support, are starting to become available, one such being the Buffalo IFC-PCIE4U3S. Only seems to be available in Asia and Australia at the moment however, but even then, orders may not be satisfied before February because of lack of availability...

So this means, if you have a NEC/Renesas uPD720200/uPD720200A, a VIA Labs VL800/801 or a Fresco Logic FL1000/FL1009 based USB 3.0 controller, you're out of luck with regards to USB 3.0 debug.
This doesn't bode too well for USB 3.0 as replacement for legacy RS232 ports...
Oh and the other disappointing part of the current crop of PCIE-USB3 controllers is they don't support USB-3.0 boot either. Quite the letdown...

Hint: If you want to display the Extended Capabilities of your controller, you can do so by modifying the xhci_setup_port_arrays() of drivers/usb/host/xhci-mem.c in () on a recent Linux kernel, as it already has some code to scan the extended caps. Using this method, I was able to find that, as of firmware 4015, the only Extended Capabilities provided by an uPD720200 USB 3.0 controller was "USB Legacy Support" (xHCI specs chpater 7.1) and "xHCI Supported Protocol" (7.2). The same applies for a Fresco Logic FL1000 based controller that I just got my hands on, though this one also sports vendor specific Extended Capabilities, with IDs 192 and 193.

Below are the answers from various manufacturers regarding their planned support of USB 3.0 debug. Being mostly interested in addon cards for existing systems, I haven't asked Intel or AMD.

NEC/Renesas:
"The uPD720200 does not support the debug port capability, and there is no plan to add it. But we have already added debug port support to our newer USB3 host controllers, uPD720201 and uPD720202."
VIA Labs:
"VL800/801 doesn't support debug port capability. As I know, the only one support is Renesas 3rd gen. host controller UPD720201_202, we plan to add this feature with VL805 that will MP next year."
Fresco Logic:
"FL1000/FL1009 didn't support USB debug extended capability.
We are doing it (for the next generation of controllers) and the schedule will be 2012/Q1."

2011-08-09

Enabling option ROM and flashing gPXE on an SMC 1211

If you're going to play with PCI option ROMs, you're likely to salvage a PCI Network Interface Card (NIC) with a flash ROM, or at least one that has a socket for it. And given their popularity in the late 90s/early 2ks, you have a fair chance getting your hands on with a RealTek RT8139 based one. One such NIC is is the SMC 1211 (or "Accton Technology Corporation SMC2-1211TX" as reported by lscpi -nn on Linux), which, in its basic configuration, comes with a DIP socket for 5V flash chips up to 128 KB in size. This is actually quite a desirable card to have as it is beautifully supported by flashrom, which has great support for the RT8139 chip, and no extra tweaking is needed to support the maximum flash size of 128 KB, as can be the case with other NICs. Add a W29C011A-15 or compatible, which can be easily obtained or salvaged from an old motherboard and you'll have more than enough space to play with an option ROM.

Only problem of course is that most SMC 1211s don't come with a flash chip by default so the option ROM is disabled and needs to be re-enabled. Luckily, Realtek does provide a tool called rset8139.exe to do just that. Of course, rather than blindly trusting the tool before we go and play with a custom option ROM, we may as well attempt to check that everything is in order so we are first going to flash a proper one, such as gPXE before running the rset8139 tool. Off we go then to etherboot/gPXE's awesome rom-o-matic, fill our options, including the 1113:1211 VID:PID of our SMC card and get our gPXE ROM back then.

First order of the day, since we're using a 128 KB flash chip and the gPXE ROM we got was smaller, is pad our ROM to our target size with:

cat gpxe-1.0.1-11131211.rom /dev/zero | dd bs=1k count=128 > gpxe_128k.rom

Then we flash with:

flashrom -p nicrealtek -w gpxe_128k.rom

So that takes care of having a proper option ROM. Of course, since we haven't enabled it on the card, you will find that no matter the options you select in your BIOS, the SMC option ROM is not executed and this is precisely why we need to run that rset8139 utility. Now, it is possible that Linux or Windows version of this utility exist, but it looks like the most common version is the DOS one, which, thanks to the oh-so-convenient Rufus (see this post), running from a DOS bootable USB stick is no problem at all.

One important thing to note is there exists multiple versions of the utility, ranging from 5.00 to 5.09, and that not all of these appear to detect the SMC 1211 card. Some regression has been introduced by Realtek, which leaves the most recent versions of rset8139 unable to change settings for the 1211. Therefore, the version I recommend using if you have an SMC card is v5.01, which can be picked here. If you don't have an SMC card, then you can try your luck with v5.0.9, which appears to be the most recent, and which is available here.

Once you have created your DOS bootable USB stick and copied rset8193.exe over, you should end up with something similar to the screenshot below (courtesy of desconexão.net), where you will be able to enable and set your ROM size:


After these settings are saved and you reboot, you should find that the gPXE option ROM payload is now executed by your BIOS, and with that you can get cracking on building a custom option ROM using your RT8139 based card.

2011-08-08

USB_DOS: The (once) easiest way to create a DOS bootable USB stick on Windows

(Updated 2011.12.14: this guide is now obsolete and has been superseded. If you want to create an MS-DOS or FreeDOS bootable USB Flash Drive, please visit the Rufus page).

If you ever need to run a DOS utility (eg. to flash a BIOS or a firmware), and want to painlessly create a bootable DOS USB key from Windows, this is for you.

Courtesy of FreeDOS and the HP Bootable USB utility (HPUSBFW), the following archive contains everything you need to create a fully functional DOS bootable USB stick. Just extract the file below (using 7-zip if needed) and follow the instructions:

USB_DOS v1.0

Now, for some additional information, in case this may be of interest to you:
  • The HPUSBFW executable has been modified from the original to request elevation on Vista and later. If you remove the manifest you should find that it matches the official HP one.
  • The FreeDOS command.com and kernel.sys files come from the current version of FreeDOS (1.0). command.com has been patched to prevent it from requesting date and time input from the user at boottime. If you unpack command.com with upx, you should find that the only difference with official is a set of 4 NOPs (0x90).
    Oh, and yes I tried to recompile the latest FreeCOM from SVN, in a FreeDOS VMWare image, but it's been quite a struggle (hint: don't waste your time with OpenWatcom, use Borland Turbo C++ v2.01 instead) and the resulting command.com freezes after a few commands when used bare with kernel.sys, as is the case with HPUSBFW.

2011-08-03

UBRX - L2 cache as instruction RAM (CAiR)

Word of advice: if you want to play with CPU caches, and especially an L2 cache, don't use a Slot 1 Pentium III as your test system: L2 is disabled by default there and, because it's really some fast RAM that was added on the board that also has the CPU chip, initializing it take a little more effort than simply flipping a switch... If you want to know all about how to initialize L2 cache on Slot 1 Pentiums, you may want to have a look at the old freebios code.

But now, with L2 sorted out and as a famous robot once said: "Things are starting to look up..."

Why is L2 cache so important for UBRX you ask? Good question.
You see, our goal is to run binary code that was uploaded by the user, on a system that is considered RAM-less, therefore all we have at our disposal are the caches. Now, CAR (Cache As RAM) has been in use by coreboot for some time, but the problem with this implementation is that it only uses L1 cache. However, if you know your CPU architecture 101, you are aware that there are two L1 caches ondie: one for data and another for instructions, with the instruction one being read-only. Thus, the CAR setup method from coreboot only provides access to the L1 data cache, not the instruction one, so we can't simply upload our code into L1-Data and expect it to run.

On the other hand, the L2 cache is a unified one, which means that it works for both data and instruction. Thus, if we manage to get our code onto L2, and have all the caches in WriteBack mode, we should be able to get the CPU to fetch instructions, which we uploaded, from L2, and we're good. This is called Cache As instruction RAM (CAiR). And with L2 caches being more than 256 KB in size, we could actually run a hefty and quite complex section of code, rather than be limited to the 16 or 32 KB of L1.

To achieve that, simply upload the code you want into L1-Data (which would have been initialized as CAR), then read or write a contiguous section of data, from a different address, that is larger than your L1 cache. As L1-Data gets replaced, your code gets pushed onto L2, where it is not accessible for execution by the CPU.
Neat!

So, how does it work in the UBRX console? Like this:
s/u/r/q> s
$60000010 a             # disable cache
$11e c $0 d $01043531 m # setup L2 for PIII Slot 1
$2ff c $0 d $c00 m      # fixed + var MTRRs
$268 c $06060606 d m    # C0000-C7FFF as WriteBack
! $10 a                 # flush and enable cache
$8000 c $c0000 <        # preload region to L2-Unified
# load our code in L1-Data
$c0000 d $f8ba68b0 z    # 'h'
$c0004 d $65b0ee03 z    # 'e'
$c0008 d $ee03f8ba z
$c000c d $f8ba6cb0 z    # 'l'
$c0010 d $6cb0ee03 z    # 'l'
$c0014 d $ee03f8ba z
$c0018 d $f8ba6fb0 z    # 'o'
$c001c d $0db0ee03 z    # CR
$c0020 d $ee03f8ba z
$c0024 d $f8ba0ab0 z    # LF
$c0028 d $ffcbee03 z
$8000 c $c0030 >        # flush L1-Data onto L2-Unified

$c0800 b $c0000         # stack at C8000, code at C0000
.
s/u/r/q> r
hello
s/u/r/q>
Now, by flashing less than 4KB of your BIOS bootblock, you are able to run ANY code you want using the UBRX the recovery console, even if you don't have any RAM installed, with the added benefit that your code can be as large as your L2 cache. Neat!

All of this and more in UBRX v0.4.

2011-07-15

Creating a bootable UEFI DUET USB stick

Now that one of my patches has made it into the UEFI/EDK2 SVN repository, I'm going to provide a quite guide on the easiest way to create a bootable UEFI USB stick for legacy platforms, on Windows.
In case you are not familiar with EFI/UEFI, it is very much possible to run EFI even on legacy (i.e. non EFI) hardware, through a process called DUET, which provides a complete EFI emulation layer using the underlying BIOS. This is great for testing, without having to modify your hardware in any way.

It is all provided by the DuetPkg of the EDK2. However, the main issue with EDK2 is that it may be a difficult to get working, unless you used the same development tools as the EDK2 developers, which, on Windows would default to Visual Studio 2008, which is not free.

Isn't there simple way to use freely available tools on Windows, to easily compile a DUET USB bootstick?
Glad you asked. There is now, and here is the complete process that goes with it:
  • Download and install the Windows Server 2003 WinDDK, v3790.1830, using this direct link (230 MB ISO download). This is of course not the latest DDK but unless you want to waste time in extra configuration, you might as well go with the supported version. If I find the time to fix EDK2 for the latest, I may send another patch to the EDK2 team, but for now, this will have to do.
  • (Optional) If you plan to use ASL to compile ACPI, which isn't needed for DUET, create a C:\ASL directory, download the Microsoft ASL Assembler/Compiler archive from here, open the .exe with 7-zip. Then open the .msi from the .exe (Open Inside), find the file _C6C6461BF87C49D66033009609D53FA5, extract it to your C:\ASL directory and rename it asl.exe (count on Microsoft to provide an installer... for a mere 55 KB exe). Also note that the ASL compiler is for 32 bit only. If you're running 64 bit, tough luck.
  • Mount or burn the ISO and install the WinDDK, to the default C:\WINDDK\3790.1830 directory.
  • If needed, download and install TortoiseSVN, or any other SVN client.
  • Fetch the latest EDK2 from SVN, using https://edk2.svn.sourceforge.net/svnroot/edk2/trunk/edk2 as the SVN repository.
  • Open a DDK console ("Development Kits" → "Win2k3 Free x64 Build Environment" or "Win2k3 Free x86 Build Environment") and navigate to your edk2 direcvtory
  • Run the command:
    edksetup.bat
    This will initialize your environment by creating the required files in the Conf\ directory. You can safely ignore the warning about cygwin.
  • Open the file Conf\target.txt and edit the TOOL_CHAIN_TAG line to have:
    TOOL_CHAIN_TAG = DDK3790xASL
    You also may want to change MAX_CONCURRENT_THREAD_NUMBER and make sure TARGET_ARCH is properly set to either IA32 or X64 according to your needs, as it avoids having to provide extra arch parameter when invoking the commands below).
  • In the console, run (32 bit):
    build -p DuetPkg\DuetPkgIA32.dsc
    or (64 bit)
    build -p DuetPkg\DuetPkgX64.dsc
    It may take a while but it should complete successfully.
  • cd to DuetPkg\ and run:
    PostBuild.bat
  • Plug in your USB key and check its drive letter. In the following I will assume it is F:, and run:
    CreateBootDisk.bat usb F: FAT16
  • Unplug and replug the key as requested, and run:
    CreateBootDisk.bat usb F: FAT16 step2
You should now have a bootable UEFI USB key. Plug that into a PC, boot it, and you will be welcomed into the wonderful world of EFI.
Note that if you rebuild any of the packages, you only have to perform the last step to update the key.

2011-07-11

Introducing UBRX - an Universal BIOS Recovery Console for x86 PCs

Following up on the previous BIOS generation endeavour, as well as wanting to demonstrate to any naysayer that a universal recovery console (or panic room) in the BIOS bootblock of an x86 system is not something from the realm of fantasy, it is my pleasure to introduce UBRX, the Universal Bootblock Recovery console for X86 systems.

In its current version (0.1), it is mostly a Proof of Concept aimed at the coreboot team, since the crafting of a 'panic room' type bootblock has been on their mind for some time, and my understanding is that their approach would be to make such a bootblock platform specific (the motherboard would first need to be supported by coreboot), whereas I believe that there is a way to be a lot more generic to produce a 'panic room' feature that works on all platforms, even the ones that coreboot doesn't support yet. The current bootblock is less than 2 KB, which ought to leave plenty of room for a Y-modem, CAR, and a full console implementation.

If your aim is to develop a real BIOS from scratch, and you have a PC that is more recent than the year 2000 (most PCs from the late nineties should work too) UBRX might also be of interest to you, since it should provide you with a serial console. In short, UBRX can also be used as a base for BIOS development, regardless of the machine you have.

What UBRX implements then is a safe method for the detection of a PnP Super I/O chip, as well as the detection and initialisation of a 16550 compatible UART there, to then provide on-demand access to a serial console. The emphasis here has to be with the safety of the detection being performed, as it is intended to be executed at every boot, without resulting in advert consequences for non PnP Super I/O hardware residing in the same hardware space as the one we check, or any non UART function residing in the Super I/O. To find more about how we achieve detection safety, I suggest you check the "Detection Primer" section of the UBRX README.

Now, of course, as with "unlimited" broadband, the "universal" applicability of UBRX comes with some understanding that fairness needs to be applied to the claim. As such, non PnP Super I/Os, PnP Super I/Os that require uncommon initialization (I'm looking at you ITE), and platforms using CPUs other than Intel or AMD are not currently supported. Also, due to the lack of datasheets from nVidia, it is very likely that UBRX may not work with motherboards sporting an nVidia chipset. However, if you have an Intel motherboard with an ICH# chipset, or an AMD motherboard with an SB##0 SouthBridge (which probably covers more than 90% of PCs from the last few years), UBRX is expected to work.

Finally, as UBRX is only a Proof of Concept for now, the console is limited to a serial repeater, so there's not much you can do with it. Especially it it missing CAR (Cache As RAM) initialization and Y-modem functionality, to be able to transfer and run bare metal executables to do interesting things, such as flashing the remainder of the BIOS, initializing the full range of RAM according to the hardware, or loading a debugger. Therefore, if you choose to test and flash UBRX, remember that you must have means to reflash your BIOS using an external program, as your PC will become unusable.

Please head to the github project for more info. A source tar archive of the source can also be downloaded here.

2011-07-07

git remote repository

If you're like me, you probably have a small Linux server somewhere, begging to be used as a remote central git repository, be it only for the purpose of backing up your development projects on a second machine. A Linux based wireless router or plug computer is perfect for such a job, and, as with any remote VCS, can be invaluable with helping to recover from mishaps on your development rig. And of course, if you're doing development and testing on different computers at once, being able to push/pull from a central server makes your life a lot easier.

Git daemon

By default, git includes a daemon utility to serve clients using the git:// protocol, so you don't even have to install anything extra. The one thing that may matter to you is that the default git daemon does not provide any authentication mechanism, so anybody on your network will be able to push/pull files onto a project. For a SOHO, this isn't a big deal, but in a corporate environment, you may want to look into something a bit more secure.

Our first order of the day then is to decide the directory we want to serve git project from on the server and start the git daemon. In this example, I will use /usr/src on the server. The command to run git daemon then is:
git daemon --reuseaddr --base-path=/usr/src --export-all --verbose --enable=receive-pack --detach --syslog
The --export-all allows pull/clone to be issued by remote clients, regardless of whether a git-daemon-export-ok file exists in the git directory. The --enable=receive-pack is required if you want to be able to push from a client, as it is disabled by default and you will get the error: 'receive-pack': service not enabled for './.git'. The rest of the options should be fairly explicit (if not, see the git daemon page. As you can see, these settings are as permissive as can be, which is probably what you want if you are starting with using git as a server.

Creating the server repository

With the daemon running (check your syslog or messages to confirm), we can now create the directory we want to be used as the origin on our server. Let's call it myproject. Thus:
root@git-server:~# cd /usr/src/
root@git-server:/usr/src# mkdir myproject
root@git-server:/usr/src# cd myproject/
root@git-server:/usr/src/myproject# git init
Initialized empty Git repository in /usr/src/myproject/.git/
Note that the git people historically recommend creating project directory with a .git extension, but there doesn't seem to be much point to it.

Indiana Git and the Intuitiveness of Doom

At this stage, if you have existing files for your project, you have the choice of transferring them to the server and git add + git commit them, or first clone the empty directory on your client and then add the files there. With the latter being the most likely situation, this is what I will demonstrate, therefore (the following was done on Linux - if you are using Msys-git on Windows, see below):
user@git-client:~$ git clone git://git-server/myproject
Cloning into myproject...
warning: You appear to have cloned an empty repository.
user@git-client:~$ cd myproject/
# in real life you would copy, add and commit existing project files there
user@git-client:~/myproject$ touch README
user@git-client:~/myproject$ git add README
user@git-client:~/myproject$ git commit -m "added README"
[master (root-commit) e3cb915] added README
 0 files changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 README
So far so good, but then:
user@git-client:~/myproject$ git push
No refs in common and none specified; doing nothing.
Perhaps you should specify a branch such as 'master'.
Everything up-to-date
OK, googling around shows you need to add origin master to that first push, so:
user@git-client:~/myproject$ git push origin master
Counting objects: 3, done.
Writing objects: 100% (3/3), 208 bytes, done.
Total 3 (delta 0), reused 0 (delta 0)
remote: error: refusing to update checked out branch: refs/heads/master
remote: error: By default, updating the current branch in a non-bare repository
remote: error: is denied, because it will make the index and work tree inconsistent
remote: error: with what you pushed, and will require 'git reset --hard' to match
remote: error: the work tree to HEAD.
remote: error:
remote: error: You can set 'receive.denyCurrentBranch' configuration variable to
remote: error: 'ignore' or 'warn' in the remote repository to allow pushing into
remote: error: its current branch; however, this is not recommended unless you
remote: error: arranged to update its work tree to match what you pushed in some
remote: error: other way.
remote: error:
remote: error: To squelch this message and still keep the default behaviour, set
remote: error: 'receive.denyCurrentBranch' configuration variable to 'refuse'.
To git://git-server/myproject
 ! [remote rejected] master -> master (branch is currently checked out)
error: failed to push some refs to 'git://git-server/myproject'
Now that's even worse! What the heck?

Long story short, git has a design limitation that prevents it to update both the working directory structure (the user visible files such as 'README', .c/.h, etc.) and the index (the hashed nodes, that contain the same information plus details of the various changes the files went trough) at the same time, on the server. Yes, if they really wanted to, the git developers could easily work around that limitation by providing an option to automatically force a sync on the remote working dir from the index, when index changes have been pushed, but they have decided that, since there exists individual cases where such an option would cause harm, they might as well prevent everybody from doing so altogether, regardless of whether people might be fully aware of what they are doing and accept the consequences.
So this means that, as long as git sees the server git directory being checked out with the 'master' branch, which is the case by default even on an empty directory, as well as the client git repo also using the 'master' branch, which is also the default, it considers that the server repository has precedence (i.e. that there might exist uncommitted changes in the server repo, that have higher priority than any committed client ones), and therefore, will reject remote requests to update the index, such as a push.

To work around that then, you need to make sure that the server doesn't have the 'master' branch checked out locally, when you are issuing a git push from the client. Easiest way to do that is to check into a different branch, away from 'master', on the server, so if we just checkout onto a 'dummy' branch that we create (-b option), git should be happy again. Let's try that:
root@git-server:/usr/src/myproject# git checkout -b dummy
fatal: You are on a branch yet to be born
Right... And people ask me why I'm still not entirely convinced that git is the best invention since sliced bread. Yes, I'll be the first to say that it is usually miles better than the competition, and when it works, it's just brilliant, but quite frankly, there is such thing as intuitive, and as far as intuitive goes, there is still a lot of room for improvement with git.
As the error indicates, the problem this time is that our repo is bare. It doesn't even have a HEAD. Now, because we don't have all day to figure out each of git's numerous intricacies, we just take matters into our own hands and hook into the git index directly with:
root@git-server:/usr/src/myproject# git symbolic-ref HEAD refs/heads/dummy
This should ensure that git no longer sees us as potentially modifying the (still non-existing) 'master' branch on the server and stop bugging us.

Then, if you go back to the client:
user@git-client:~/myproject~ git push origin master
Counting objects: 3, done.
Writing objects: 100% (3/3), 208 bytes, done.
Total 3 (delta 0), reused 0 (delta 0)
To git://git-server/myproject
 * [new branch]      master -> master
At last, success!! From there on you should be able to use a naked git push on the client and roll.
If you later want to edit/modify the repo content on the server (the server working dir will remain empty even after a push, because the working directory is no longer following the 'master' HEAD), you can do so by switching back to the 'master' branch with git checkout master. Then an up to date version of what you have pushed should be in your server working dir. However, remember that, if you are planning to remotely push some more, you need to use either git checkout [-b] dummy (if you don't mind having a dummy branch created) or git symbolic-ref HEAD refs/heads/dummy (if you don't want a branch) on the server when you are done, to prevent git from rejecting the operation. If you use the latter, be mindful that you will get a warning: remote HEAD refers to nonexistent ref, unable to checkout error (yes git, if it results in a failure, it is an error, not a warning), so you may have to go back to your server and set symbolic-ref to 'master' before cloning. And if you use the former, make sure that's you're working on the 'master' branch and not 'dummy' after cloning.

Yay, now we have a working git server, and more importantly, we know how to work around the various counter-intuitive git limitations to create repositories on it... Of course, that is, as long as you don't use msys-git on Windows 7...

What's wrong with git daemon and msys-git?

OK, so we are sorted out on Linux. How about we try the same thing on Windows 7 with msys-git?
user@WINDOWS ~
$ git clone git://git-server/myproject
Cloning into myproject...
remote: Counting objects: 8, done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 8 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (8/8), done.

user@WINDOWS ~
$ cd myproject/

user@WINDOWS ~/myproject (master)
$ echo "test" > README

user@WINDOWS ~/myproject (master)
$ git add README
warning: LF will be replaced by CRLF in README.
The file will have its original line endings in your working directory.

user@WINDOWS ~/myproject (master)
$ git commit -m "edited README"
[master warning: LF will be replaced by CRLF in README.
The file will have its original line endings in your working directory.
a0f27c3] edited README
warning: LF will be replaced by CRLF in README.
The file will have its original line endings in your working directory.
 1 files changed, 1 insertions(+), 1 deletions(-)
So far, so good. But then, if you issue git push the command hangs forever:
user@WINDOWS ~/myproject (master)
$ git push
Counting objects: 5, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3)
On the server, the log doesn't seem to indicate that much is amiss:
Jul  7 16:22:23 git-server git-daemon[7324]: Connection from 1.2.3.4:51238
Jul  7 16:22:23 git-server git-daemon[7324]: Extended attributes (13 bytes) exist <host=git-server>
Jul  7 16:22:23 git-server git-daemon[7324]: Request receive-pack for '/myproject'
Congratulations! You've just encountered msys-git bug #457 for which no patch exists yet. If you want a workaround, you can either switch to using ssh (but then you need to sort out authentication, which, for a single-user internal-only VCS operation is a bit of an overkill) or share your repository with samba, and then add the following in the [remote "origin"] section from .git/config:
pushurl = file:////git-server/src/myproject
As can be inferred the above simply forces git to use Samba rather than the git protocol for push operations. With this workaround in place, you are truly set to use a remote git server.

ADDON: The following script might be used as a very useful first commit for the server repository, as it helps toggle between the ability to commit files on the server and allowing client commits:
#!/bin/sh
git branch | grep -q \*
if [ $? -eq 0 ]; then
  git symbolic-ref HEAD refs/heads/dummy
  echo "Switched to fake branch 'dummy'"
else
  git symbolic-ref HEAD refs/heads/master
  echo "Switched to branch 'master'"
fi