PowerShell script to Convert UTF-8 misinterpreted file names

You'd think that somebody else would have come up with a quick script to do just that on Windows, but it looks like nobody else bothered, so here goes.

Here's the deal: You copied a bunch of files, and somewhere along the way, one of the applications screwed up and did not produce actual Unicode file names but instead misinterpreted the UTF-8 sequences as CodePage 1252, resulting in something dreadful like this:

And now you'd like to have a quick way to convert the 1252-interpreted UTF-8 to actual UTF-8. So you look around thinking that, surely, someone must have done something to sort this annoyance, but the only thing you can find is a UNIX perl script called convmv, which isn't really helpful. Why hasn't anyone crafted a quick PowerShell script to do the same on Windows already?

Well, it turns out that, because of PowerShell's limitations, and Windows' getting in the way of enacting a proper conversion of 1252 to UTF-8, producing such a script is actually a minor pain in the ass. Still, now, someone has produced such a thing:

#region Parameters
param(
 # (Optional) The directory
 [string]$Dir = "."
)
#endregion

# You'll need to have your console set to CP 65001 AND use NSimSun as your
# font if you want any hope of displaying CJK characters in your console...
[Console]::OutputEncoding = [System.Text.Encoding]::UTF8

$files = Get-ChildItem -File -Path $Dir -Recurse -Name

foreach ($f in $files) {
  $bytes = [System.Text.Encoding]::GetEncoding(1252).GetBytes($f)
  $nf = [io.path]::GetFileName([System.Text.Encoding]::UTF8.GetString($bytes))
  Write-Host "$f" → "$nf" # [$hex]
  # Must use -LiteralPath else files that contain '[' or ']' in their name produce an error
  Rename-Item -LiteralPath "$f" -NewName "$nf"
}

# Produce a "Press any key" message when ran with right click
$auxRegKey='\SOFTWARE\Classes\Microsoft.PowerShellScript.1\Shell\0\Command'
$auxRegVal=(get-itemproperty -literalpath HKLM:$auxRegKey).'(default)'
$auxRegCmd=$auxRegVal.Split(' ',3)[2].Replace('%1', $MyInvocation.MyCommand.Definition)
if ("`"$($myinvocation.Line)`"" -eq $auxRegCmd) {
  Write-Host "`nPress any key to exit..."
  $null = $Host.UI.RawUI.ReadKey('NoEcho,IncludeKeyDown')
}

If you save this script to something like utf8_rename.ps1 in the top directory where you have your misconverted files, and then use Run with PowerShell in the explorer's context menu, you should then see some output like this (provided your console is set to codepage 65001, a.k.a. UTF-8 and that you select a font that actually supports CJK characters, such as NSimSun (Microsoft will really have to explain how they have no trouble displaying CJK with NSimSun but still can't seem/want to do it with Lucida Console):

Eventually, your file names should have been converted to their expected value, and all will be well:

That is, until someone who thinks it's okay to not properly support UTF-8 absolutely EVERYWHERE (Hey Microsoft, how about some UTF-8 Win32 APIs already?) screws up and forces people to manually unscrew their codepage handling yet again...

Bonus

By the way if you're using Windows 10 19H1 or later, you should know that Microsoft finally added a setting to set the system codepage to UTF-8, which seems to finally improve on the failed codepage conversions that prompted the above script. Even as it says that it's in Beta, you may want to enable it:

Installing Debian ARM64 on a Raspberry Pi 3 in UEFI mode

That's right baby, we're talking vanilla Debian ARM64 (not Raspbian, not Armbian) in pure UEFI mode, where you'll get a GRUB UEFI prompt allowing you to change options at boot and everything.

At long last, the Raspberry Pi can be used to install vanilla GNU/Linux distributions in the same manner as you can do on a UEFI PC. Isn't that nice?

Not that I don't like Raspbian or Armbian (as a matter of fact I am impressed by the very fine job the Armbian maintainers are doing with their distro), but I have now spent enough time helping with the UEFI Raspberry Pi 3 effort not to push this whole endeavour to its logical conclusion: Install vanilla ARM64 GNU/Linux distros. That's because, in terms of long term support and features, nothing beats a vanilla distro. I mean, what's the point of having an 64-bit CPU if the distro you're going to install forces you to use 32-bit?

Prerequisites

Hardware:

• A micro SD card with sufficient space (16 GB or more recommended). You may also manage with a USB Flash Drive, but this guide is geared primarily towards SD card installation

• A Raspberry Pi 3 (Model B or Model B+) with a proper power source. If you're ever seeing a lightning bolt on the top left of your display during install, please invest into a power supply that can deliver more wattage.

Note that our goal here is to install the system on a SD card, through netinstall, using a single media for the whole installation process.

In other words, there is no need to use an additional USB Flash Drive, as we could do, to boot the Debian installer and then install from USB to SD. This is mostly because it's inconvenient to have to use two drives when one can most certainly do, and also because while USB to SD may look easier on paper (no need to fiddle with the "CD-ROM" device for instance) it's actually more difficult to complete properly.

Thus, while I'll give you some pointers on how to perform a USB based installation in Appendix D, I can also tell you, from experience, that you are better off not trying to use a separate USB as your installation media and instead performing the installation from a single SD, as described in this guide.

Software:

• The latest Debian ARM64 net install ISO.
  
  You can download it from https://cdimage.debian.org/debian-cd/current/arm64/iso-cd/
  (debian-##.#.#-arm64-netinst.iso, 250 MB).

• The latest Raspberry Pi 3 UEFI firmware binary, along with the relevant Broadcom bootloader support files (i.e. bootcode.bin, config.txt, fixup.dat, start.elf).
  
  You can find a ready-to-use archive with all of the above at https://github.com/pftf/RPi3/releases
  (RPi3_UEFI_Firmware_v#.##.zip, 3 MB).
  
  Note that this firmware archive works for both the Raspberry Pi 3 Model B and the Raspberry Pi 3 Model B+ (as the relevant Device Tree is automatically selected during boot).

• (Optional) The non-free WLAN firmware binaries that are needed if you want to use Wifi for the installation.
  Note that, if you picked up the archive above then you don't need to do anything as the WLAN firmware binaries are included in it too.

Preparation

Note: a complete example of how to achieve the first 3 steps below using DISKPART on Windows or fdisk + mkfs on Linux is provided in Appendix A at the end of this post.

• Partition your SD media as MBR and create a single partition of 300 MB of type 0x0e (FAT16 with LBA).
  Do not be tempted to use GPT as the partition scheme or 0xef (ESP) for the partition type, as the ondie Broadcom bootloader does not support any of those. It must be MBR and type 0x0e. You can use the command line utilities fdisk on Linux or DISKPART on Windows to do that.

• Set the partition as active/bootable. This is very important as, otherwise, the Debian partition manager will not automatically detect it as ESP (EFI System Partition) which will create problems that you have to manually resolve (See Appendix C).
  If using fdisk on Linux, you can use a to set the partition as active.
  If using Windows, you can use DISKPART and then type the command active after selecting the relevant disk and partition.

• Format the partition as FAT16. It MUST be FAT16 and not FAT32, as the Debian partition manager will not detect it as ESP otherwise and, again, you will have to perform extra steps to salvage your system before reboot (Appendix C).
  The Linux and Windows base utilities should be smart enough to use FAT16 and not FAT32 for a 300 MB partition, so you should simply be able to use mkfs.vfat /dev/<yourdevice> (Linux) or format fs=fat quick in Windows' DISKPART. The Windows Disk Manager should also be smart enough to use FAT16 instead of FAT32 if you decide to use it to format the partition.

• Extract the UEFI bootloader support files mentioned above to the newly formatted FAT partition. If you downloaded the Raspberry Pi 3 UEFI firmware binary from the link above, you just have to uncompress the zip file onto the root of your media, and everything will be set as it should be.

• Extract the content of the Debian ISO you downloaded to the root of the FAT partition. On Windows you can use a utility such as 7-zip to do just that (or you can mount the ISO in File Explorer then copy the files).

Once you have completed the steps above, eject your SD card, insert it in your Pi 3 and power it up. Make sure no other media is plugged in besides the SD card. Especially, make sure that there aren't any USB Flash Drives or USB HDDs connected.

Initial Boot

Unless you did something wrong, you should see the multicoloured boot screen, which indicates that the Raspberry Pi properly detected your SD media and is loading the low level CPU bootloader from it.

Then you should see the black and white Raspberry logo, which indicates that the Raspberry Pi UEFI firmware is running.

Wait for the GNU GRUB menu to appear (which it should do by default after the Pi logo disappears) and choose *Install (which should already be the default) and let the Debian installer process start.

Debian Installer

Note: In case anything goes wrong during install, remember that you can use Alt-F4 to check the current installation log for details about the error.

• Select your Language, Country and Keyboard and let the installer proceed until it reports that No Common CD-ROM drive was detected.

• At this stage, on Load CD-ROM drivers from removable media select No.

• On Manually select a CD-ROM module and device select Yes.

• On Module needed for accessing the CD-ROM select none.

• On Device file for accessing the CD-ROM type exactly the following:
  
  

  -t vfat -o rw /dev/mmcblk0p1

  
  For the reasons why you need to type this, see Appendix B below.

• With the "CD-ROM" device set, let the installation process proceed and retrieve the base packages from the media until it asks you for the non-free firmware files on the network hardware detection. If you plan to use the wired connection, you can skip the (Optional) step below.

• (Optional) If you plan to use WLAN for the installation, choose Yes for Load missing firmware from removable media. If you created the media from that Raspberry Pi 3 firmware archive linked above, the relevant firmware files will be detected under the firmware/ directory.
  
  Note 1: Because there are multiple files to load, you will be prompted multiple times for different firmware files (look closely at their names, you will see that they are actually different). This is normal. Just select Yes for each new file.
  
  Note 2: Though they are included in the UEFI firmware zip archive we linked above, it is most likely okay not to provide the .clm_blob if you don't have it (the Wifi drivers should work without that file), so don't be afraid to select No here if needed.

• Set up your network as requested by the installer by (optionally) choosing the network interface you want to use for installation and (also optionally) setting up your access point and credentials if you use Wifi.

• Go through the hostname, user/password set up and customize those as you see fit.

• Let the installer continue until you get to the Partition disks screen. There, for Partitioning method select Manual. You should see something like this:
  
  

  MMC/SD card #1 (mmcblk0) - 16.0 GB SD 2WCGO
       #1  primary  314.6 MB  B  K  ESP
           pri/log                  FREE SPACE

  
  If, instead, you see something like this:
  
  

  MMC/SD card #1 (mmcblk0) - 16.0 GB SD 2WCGO
       #1  primary  314.6 MB  B   fat16
           pri/log                FREE SPACE

  
  In other words, if you don't see B K ESP for the first partition, then it means that you didn't partition or format your drive as explained above and you will need to reference Appendix C (Help, I screwed up my partitioning!) to sort you out.

• From there select the FREE SPACE partition and use the partition manager's menu to create two new primary partitions (one for swap and one for the root file system), until you have something like this:
  
  

  MMC/SD card #1 (mmcblk0) - 16.0 GB SD 2WCGO
       #1  primary  314.6 MB  B  K  ESP
       #2  primary    1.0 GB     f  swap    swap
       #3  primary   14.7 GB     f  ext4    /
  

• Select Finish partitioning and write changes to disk and then Yes on Write the changes to disks? and let the installer continue with the base system installation.

• After a while, the installer will produce a big red ominous message that says:
  
  

  [!!] Configure the package manager
    
  apt-configuration problem
  An attempt to configure apt to install additional packages from the CD failed.

  
  This, however, is actually a completely benign message and you can safely ignore it by selecting Continue . That's because, since we are conducting a net install, we couldn't care less about no longer being to access the "CD-ROM" files after install...

• Once you have dimissed the message above, pick the mirror closest to your geographical location and let the installer proceed with some more software installation (this time, the software will be picked from that network mirror rather than the media).
  When prompted for the "package usage survey" pick whichever option you like.

• Finally, at the Software selection screen, select any additional software package you wish to install. Note that the "Debian desktop environment" should work out of the box if you decide to install it (though I have only tested Xfce so far). It's probably a good idea to install at least "SSH server".

• Let the process finalize the software and GRUB bootloader installation and, provided you didn't screw up your partitioning (i.e. you saw B K ESP when you entered the partition manager, otherwise see Appendix C) select Continue to reboot your machine on the Installation complete prompt.

If everything worked properly, your system will now boot into your brand new vanilla Debian ARM64 system. Enjoy!

Post install fixes

Here are a few things that you might want to fix post install:

1. You may find a cdrom0 drive on your desktop, which can't seem to be accessible. This is a leftover from the installer process not knowing how to handle the installation media device. You should edit /etc/fstab to remove it.
    
2. If you installed the cups package, you may get an error while loading modules (systemctl --failed will report that systemd-modules-load.service is in failed state). This is all due to the current cups package trying to load IBM PC kernel modules... on a non PC device. To fix this, simply delete /etc/modules-load.d/cups-filters.conf and reboot.
3. If using UEFI firmware v1.6 or later, you can enable the serial console by editing /etc/default/grub and changing GRUB_CMDLINE_LINUX="" to GRUB_CMDLINE_LINUX="console=ttyS0,115200", and then running update-grub.
   You may also enable serial console access for GRUB by adding the following in the same file:
   GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND="serial --unit=0 --speed=115200 --stop=1" 
   

Appendix A: How to create and format the SD partition for installation

IMPORTANT NOTE 1: Please make sure to select the disk that matches your SD media before issuing any of these commands. Using the wrong disk will irremediably destroy your data!

IMPORTANT NOTE 2: Do not be tempted to "force" FAT32 in DISKPART or mkfs and do not forget to set the bootable/active flag, else you will afoul of the issue described in Appendix C. 

Windows

C:>diskpart

Microsoft DiskPart version 10.0.18362.1

Copyright (C) Microsoft Corporation.
On computer: ########

DISKPART> list disk

  Disk ###  Status         Size     Free     Dyn  Gpt
  --------  -------------  -------  -------  ---  ---
  Disk 0    Online          238 GB      0 B        *
  Disk 1    Online          465 GB  1024 KB        *
  Disk 4    Online         4657 GB  1024 KB        *
  Disk 5    Online         4649 GB      0 B        *
  Disk 6    Online           14 GB    14 GB

DISKPART> select disk 6

Disk 6 is now the selected disk.

DISKPART> clean

DiskPart succeeded in cleaning the disk.

DISKPART> convert mbr

DiskPart successfully converted the selected disk to MBR format.

DISKPART> create partition primary size=300

DiskPart succeeded in creating the specified partition.

DISKPART> active

DiskPart marked the current partition as active.

DISKPART> format fs=fat quick

  100 percent completed

DiskPart successfully formatted the volume.

DISKPART> exit

Leaving DiskPart...

C:>

Note, if needed you can also force a specific partition type (e.g. set id=0e to force FAT16 LBA), but that shouldn't be needed as DISKPART should set the appropriate type accordingly.

Linux

The following assumes /dev/sdf is your SD/MMC device. Change it in all the commands below to use your actual device.

(Optional) If your drive was partitioned as GPT, or if you're not sure, you may want to issue the two following commands first. If it's MBR you can skip this step:

# Delete the primary GPT:
dd if=/dev/zero of=/dev/sdf bs=512 count=34
# Delete the backup GPT.:
dd if=/dev/zero of=/dev/sdf bs=512 count=34 seek=$((`blockdev --getsz /dev/sdf` - 34))

Now use fdisk and mkfs to partition the drive:

root@debian:~# fdisk /dev/sdf

Welcome to fdisk (util-linux 2.34).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0x7d188929.

Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-31291391, default 2048):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-31291391, default 31291391): +300M

Created a new partition 1 of type 'Linux' and of size 300 MiB.

Command (m for help): t
Selected partition 1
Hex code (type L to list all codes): e
Changed type of partition 'Linux' to 'W95 FAT16 (LBA)'.

Command (m for help): a
Selected partition 1
The bootable flag on partition 1 is enabled now.

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

root@debian:~# mkfs.vfat -F 16 /dev/sdf1
mkfs.fat 4.1 (2017-01-24)
root@debian:~#

Appendix B: Why do we need to use -t vfat -o rw /dev/mmcblk0p1 as the CD-ROM device?

• Why this weird device name with options? Because these are actually mount command line parameters and the Debian installer actually calls mount behind the scenes and feeds it exactly what we write here. This means we can hijack the device name field to invoke the additional mount parameters we need.

• Why /dev/mmcblk0p1? That's simply name of the device for the first partition (p1) on the SD/MMC media (mmcblk0) as seen by the Linux kernel on a Raspberry Pi.

• Why -t vfat? Because the Debian installer appends fstype=iso9660 to the mount option which prevents automount and forces us to override the file system type.

• Why -o rw? Because the Debian installer won't be able to use the first partition for /boot/efi otherwise or load the WLAN firmware from the media (you get a device or resource busy when trying to remount the media).

Appendix C: Help I screwed up my partitioning!

Of course you did. You thought you knew better, and now you are paying the price...

The problem in  a nutshell is that:

1. You can't use a regular ESP on a Raspberry Pi, on account that GPT or an MBR partition with type 0xef are not handled by the Broadcom CPU bootloader. And there is nothing you can do about this, because this is a behaviour that's hardcoded in the CPU silicon itself.
    
2. The Debian installer's partition manager is very temperamental about what it will recognize as an ESP. In other words, if you don't use the perfect combination of boot flag, partition type and file system, it will fail to see it as an ESP.

Now the good news is that this is recoverable, but you need to know what you're doing.

• The first thing you should do in the Debian partition manager is set the first partition to be used as ESP. In other words, you will need to edit the first partition until you get this:
  

  MMC/SD card #1 (mmcblk0) - 16.0 GB SD 2WCGO
       #1  primary  314.6 MB  B  K  ESP
           pri/log                  FREE SPACE

• Then you can proceed as the guide describe, but you need to bear in mind that, as soon as you choose to write the partition changes, the partition manager will have changed your first partition type to 0xef, which, as we have seen is ubootable by the CPU. Therefore, DO NOT PROCEED WITH THE SYSTEM REBOOT AT THE END UNTIL YOU HAVE CHANGED THE PARTITION TYPE BACK.

• To do that, once you get to the Installation complete prompt that asks you to select Continue to reboot, you need to press Alt-F2 then Enter to activate a console.

• Then type exactly the following command:
  

  chroot /target fdisk /dev/mmcblk0

  Then press the keys t, 1, e, w
  

• No you can go back to the installer console (Alt-F1) and select Continue to reboot

Appendix D: Installing to a SD from an USB Flash Drive

As I explained above, and though it may seem simpler, I would discourage to use this method to install Debian on a Raspberry Pi. But I can understand that, if you don't have a card reader, you may be constrained to using this method.

For the most part, this should work fine out of the box. As a matter of fact, if you do it this way, you won't have to fiddle with the "CD-ROM" media detection. However, I will now list some of the caveat you'll face if you proceed like:

Caveat 1: If you use guided partitioning your SD/MMC media will be formatted as GPT (because this is a UEFI system after all) which the Broadcom CPU used in the Raspberry Pi can not boot. It has to be MBR. How you are supposed to force MBR over GPT in the Debian partition manager, I'll let you figure out.

Caveat 2: Similarly, you need to go through the 0xef to 0x0e conversion of your ESP, as the Pi won't boot from that partition otherwise.

Caveat 3: Of course you will also need to duplicate all the bootcode.bin, fixup.bat and so on from your USB boot media onto the SD ESP partition if you want it to boot (which is the reason why is is much more convenient to just set the ESP and Debian installer on the SD right of the bat, so you don't risk forgetting to copy a file).

Caveat 4: When I tried USB to SD install, I found that the GRUB installer somehow didn't seem to create an efi/boot/bootaa64.efi, which, if left uncorrected, will prevent the system from booting automatically.