2016-05-16

Help, I lost all networking on my Raspberry Pi!

This happened to me the other day, as I was upgrading a Pi system from Debian Jessie to Sid.

After reboot, I suddenly got the following warning in the boot log:
[FAILED] Failed to start Raise network interfaces.
See 'systemctl status networking.service' for details.
And of course, issuing ifconfig returned the dreaded output with only loopback:
root@pi ~ # ifconfig
lo: flags=73  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10
        loop  txqueuelen 1  (Local Loopback)
        RX packets 4  bytes 240 (240.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4  bytes 240 (240.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
A look at the suggested systemctl status networking.service yielded no better results:
root@pi ~ # systemctl status networking.service
• networking.service - Raise network interfaces
   Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor preset: enabled)
  Drop-In: /run/systemd/generator/networking.service.d
           └─50-insserv.conf-$network.conf
   Active: failed (Result: exit-code) since Mon 2016-05-16 22:05:36 IST; 1min 2s ago
     Docs: man:interfaces(5)
  Process: 296 ExecStart=/sbin/ifup -a --read-environment (code=exited, status=1/FAILURE)
  Process: 288 ExecStartPre=/bin/sh -c [ "$CONFIGURE_INTERFACES" != "no" ] && [ -n "$(ifquery --read-environment --list --exclude=lo)" ] && udevadm settle (code=exited, status=0/SUCCESS)
 Main PID: 296 (code=exited, status=1/FAILURE)

May 16 22:05:36 pi systemd[1]: Starting Raise network interfaces...
May 16 22:05:36 pi ifup[296]: Cannot find device "eth0"
May 16 22:05:36 pi ifup[296]: Failed to bring up eth0.
May 16 22:05:36 pi systemd[1]: networking.service: Main process exited, code=exited, status=1/FAILURE
May 16 22:05:36 pi systemd[1]: Failed to start Raise network interfaces.
May 16 22:05:36 pi systemd[1]: networking.service: Unit entered failed state.
May 16 22:05:36 pi systemd[1]: networking.service: Failed with result 'exit-code'.
Drats! What on earth am I gonna do if I no longer have networking?!?

Well, below is what you can do to get out of this precarious situation:

  1. Issue a networkctl to confirm that your Ethernet interface is still present. At this stage, it will probably only be listed as enxa1b2c3..., where A1B2C3... is your Pi's MAC address:
    root@pi ~ # networkctl
    IDX LINK             TYPE               OPERATIONAL SETUP
      1 lo               loopback           carrier     unmanaged
      2 enxa1b2c3d4e5f6  ether              off         unmanaged
    
    2 links listed.
    
    
    This is actually where the problem lies: The network interface isn't mapped to it's usual eth0, which in turn makes the networking boot scripts go "Huh?"...
  2. Check that you can bring the interface up and down, to confirm that it isn't a hardware or kernel issue with the following:
    root@pi ~ # ifconfig enxa1b2c3d4e5f6 up
    [  190.272495] smsc95xx 1-1.1:1.0 enxa1b2c3d4e5f6: hardware isn't capable of remote wakeup
    [  190.285729] IPv6: ADDRCONF(NETDEV_UP): enxa1b2c3d4e5f6: link is not ready
    [  191.851700] IPv6: ADDRCONF(NETDEV_CHANGE): enxa1b2c3d4e5f6: link becomes ready
    [  191.864838] smsc95xx 1-1.1:1.0 enxa1b2c3d4e5f6: link up, 100Mbps, full-duplex, lpa 0xCDE1
    root@pi ~ # networkctl
    IDX LINK             TYPE               OPERATIONAL SETUP
      1 lo               loopback           carrier     unmanaged
      2 enxa1b2c3d4e5f6  ether              routable    unmanaged
    
    2 links listed.
    root@pi ~ # ifconfig enxa1b2c3d4e5f6 down
    [  199.3354400] smsc95xx 1-1.1:1.0 enxa1b2c3d4e5f6: hardware isn't capable of remote wakeup
    NB: Make sure you leave the interface down for the next steps.
  3. Now, we should be able to get eth0 going again by issuing this:
    root@pi ~ # ip link set enxa1b2c3d4e5f6 name eth0
    [  277.211063] smsc95xx 1-1.1:1.0 eth0: renamed from enxa1b2c3d4e5f6
    root@pi ~ # systemctl restart networking
    [  300.952068] smsc95xx 1-1.1:1.0 eth0: hardware isn't capable of remote wakeup
    [  300.959844] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
    [  302.475405] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
    [  302.484821] smsc95xx 1-1.1:1.0 eth0: link up, 100Mbps, full-duplex, lpa 0xCDE1
  4. A quick check with ifconfig should confirm that we're rolling again. However this is just a temporary solution, which won't persist after reboot. So we need something a bit more permanent, which is to create a /etc/udev/rules.d/70-persistent-net.rules (which is probably the one file that got screwed when you lost your network) and which should contain something like:
    SUBSYSTEM=="net", ACTION=="add", DRIVERS=="smsc95xx", ATTR{address}=="*", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

    If you add this file and reboot, you should find that everything's back in order again. Pfew, another crisis averted!

7 comments:

  1. THANK YOU! I've just spent a night pulling my hair out over this and could not find any viable solutions that were specific to the RPI. As a bit of a linux novice in some areas, this post was a complete life saver that saved me the time of reflashing (which I was really close to doing!)

    ReplyDelete
  2. On a 16.04 installation, you need to leave out the KERNEL attribute. Otherwise it wont work.

    ReplyDelete
  3. Urgh, this is so close, but not 100% there! The symptoms all match, the manual steps work (meaning I can get working eth0 back), but it looks like /etc/udev/rules.d/70-persistent-net.rules is ignored. Nothing I put in there has any effect.

    What does work is to change the /etc/network/interfaces.d/50-cloud-init.cfg to refer to "enxb827eb0620ea" instead of eth0 - then I have working interface named "enxb827eb0620ea".

    Which I just KNOW will clutch out the moment I (for example) spin up docker on this device.

    Anything else one can try?

    ReplyDelete
  4. I cannot restart networking! I get an error: "The name org.freedeaktop.policyKit1 was not provided by any service files."

    ReplyDelete
  5. Man! Today I was using Rufus and just came by to tell THANK YOU for all your GREAT software! What even makes you to code so perfectly?

    ReplyDelete
  6. Just to say that I did a system update today (March 8 2019) and lost my network after the reboot. The first part of your solution worked a treat and many thanks for that. BUT the persistent rules part didn't work.

    I have another pi which is running sitting next to the failed one and it doesn't have the rules file in it. There is a set of rules files in /lib/udev/rules.d but the failed pi seems to have the same ones as that.

    ReplyDelete