power outage catch-22

Throughout my lifetime, unexpected power outages would lead Arch to do a disk check during repowering. Following this, a reboot would lead to a customary level 3 login.

The latest way Linux denies a person access to their own system is much more devious and maddening thanks, as always, to systemd. Unless a person has pre-configured a massive, nuanced, and installed (via mkinitcpio), etc/systemd/system/rescue.service and/or etc/systemd/system/emergency.service, they are stuck in a perfectly maddening Catch-22: systemd locks down root. This is an easy problem to fix... if a person can enter commands. However, since systemd has locked root, I can't access the root console. So I can't fix/unlock root.

It starts normally, in emergency mode:

You are in emergency mode. After logging in, type "journalctl -xb" to view system logs, "systemctl reboot" to reboot, or "exit" to continue bootup.

"No problem", one thinks, "this is an easy fix via the emergency command line in a (VGA) console." Wrong. Because next...

Cannot open access to console, the root account is locked. See sulogin(8) man page for more details

Thus we're not in emergency mode and we don't have console access.

There's a lot of emergency console karate I moved to the bottom of this post. Because the only sane way out is a USB rescue stick, and advanced use of it. Eg, chrooting to build a new mkinitcpio.

the horrible requirements

A VGA cable, a monitor with a VGA connection, a box/laptop with an unlocked system, an expendable USB. If no internet connection, a saved arch iso somewhere. Hours or days of free time.

the horrible steps pt 1 (usb portion)

• Find a VGA console and cable. Yes you heard me. HDMI will not work until Linux initializes and does DMCA handshaking garbage. But we're going to be strictly console. Connect the VGA stuff to the locked system
• Locked system: power it up and exit into BIOS. This is typically F2. Sometimes it's F2, pause a second, then F10, pause, F2, F10 etc. Change the boot order to look for a USB first, save, exit, and power down.
• On an unlocked system (yes you will need that too) download an Arch ISO and verify its checksum. Having an old saved arch ISO avoids the download, but you'll still have to find it and know its hard path.
• Unlocked system: insert a USB stick, the one to be booted on the locked system, and find its /dev designator.

  $ lsblk

  Let's say this one is /dev/sda, for our purposes here.
• Unlocked system: wipe the USB file system

  # wipefs --all /dev/sda

• Unlocked system: burn the bootable USB, and be sure it is complete before removing it.

  # dd bs=4M if=[hardpath/isoname.iso] of=/dev/sda conv=fsync oflag=direct status=progress

the horrible steps pt 2 (unlocking root)

# mount -rw /dev/sda1 /mnt
# genfstab -p /mnt >> /mnt/etc/fstab
# arch-chroot /mnt
# passwd
# mkinitcpio -p linux
# grub-mkconfig -o /boot/grub/grub.cfg
# grub-install /dev/sda [no partition]
# exit

---

partial fixes and related (useless)

linux command appends

Requires accessing the linux kernel load line during GRUB's install. Pick one of the linux kernels in the GRUB menu, and use the "e" key. This brings us to the kernel parameters, which we will need to edit. After adding the suffix (see below), "F10" to load the kernel with the added parameters.

partial fix 01

This stack exchange post provided me with a kernel parameter which successfully got me as far as the emergency access prompt.

SYSTEMD_SULOGIN_FORCE=1 init=/sbin/sulogin

The solution is only partial, because necessary commands to unlock root eg, passwd, mkinitcpio, nano, etc, were not available at this prompt. This emergency prompt was running entirely within initramfs.

At reboot I was right back where I started, at a locked console, unless as noted at top, I had a configured emergency.service already installed into systemd.

creation of emergency.service

helpful, but not a fix

Linuxconfig.org. Setting kernel boot parameters.

nope 1

Kali solution. Well documented kernel parameter addition. GRUB ignores.

single init=/bin/bash

nope 2

Well documented kernel parameter addition. GRUB ignores.

break