power outage catch-22

Throughout my lifetime, unexpected power outages would lead Arch to do a disk check during repowering. Following this, a reboot would lead to a customary level 3 login.

The latest way Linux denies a person access to their own system is via systemd. Unless a person has pre-configured and installed (via mkinitcpio) an elaborate etc/systemd/system/rescue.service and/or etc/systemd/system/emergency.service, systemd locks root and withholds error information. Locked root, minus error information, produces perhaps the worst imaginable Catch-22. Initially, a person believes it's easy to fix. It looks normal:

You are in emergency mode. After logging in, type "journalctl -xb" to view system logs, "systemctl reboot" to reboot, or "exit" to continue bootup.

"No problem", one thinks, "this is an easy fix via the emergency command line in a (VGA) console." Wrong. Because next...

Cannot open access to console, the root account is locked. See sulogin(8) man page for more details

Thus we can't enter commands, unlike typical emergency mode. What is the solution?

The disk must be fsck'd. Trial and error took days until I recalled fsck might be needed.

At the bottom of this post, solutions from Reddit, Archwiki, StackExchange, etc, are given, in case they inform someone else. I found them all to be useless and misleading. All involved chrooting and then, eg mkinitcpio, fstab updates, and even grub reinstallation. Some of these may work if a person can run "fsck" at the emergency command line. But this was never suggested.

some other horrible requirements

A VGA cable, a monitor with a VGA connection, a box/laptop with an unlocked system, and an expendable USB. If no internet connection, a saved arch iso somewhere. You need a VGA cable and monitor if your box doesn't have native HDMI on its graphic card. This is because emergency mode only has initramfs, with a limited set of commands -- it doesn't include HDMI handshaking.

1. setting-up for the unlock

• Connect the VGA stuff to the locked system, if your system doesn't have native HDMI
• Power it up and exit into BIOS, typically "F2". Otherwise, F2, pause a second, then F10, pause, F2, F10 etc.
• Inside BIOS, change the boot order to USB first, and save/apply the change.
• Exit BIOS, and power down the locked system.

2. create boot USB

• On an unlocked system (yes you will need that too) download an Arch ISO and verify its checksum. Having an old saved arch ISO avoids the download.
• Insert a USB stick into the unlocked system, and find its /dev designator.

  $ lsblk

  Let's say this one is /dev/sda, for our purposes here.
• Wipe the USB file system

  # wipefs --all /dev/sda

• Burn the ISO onto the bootable USB. Be certain it has completed burning before removing it.

  # dd bs=4M if=[hardpath/isoname.iso] of=/dev/sda conv=fsync oflag=direct status=progress

3. unlocking root

• insert the install USB in the locked system and power it up
• once linux loads from the USB, you'll be running a root prompt. You can find the disk designator for the locked disk drive using lsblk, then just fsck it. Let's say ours was sda.

# lsblk
# fsck /dev/sda
[answer "y" to any repair requests]
# exit

• power-down the locked system and remove the USB
• power up the locked system

The system should boot normally now. If desired, re-enter BIOS and remove the USB boot option.

partial fixes and related (useless)

linux command appends

Requires accessing the linux kernel load line during GRUB's install. Pick one of the linux kernels in the GRUB menu, and use the "e" key. This brings us to the kernel parameters, which we will need to edit. After adding the suffix (see below), "F10" to load the kernel with the added parameters.

partial fix 01

This stack exchange post provided me with a kernel parameter which successfully got me as far as the emergency access prompt.

SYSTEMD_SULOGIN_FORCE=1 init=/sbin/sulogin

The solution is only partial, because necessary commands to unlock root eg, passwd, mkinitcpio, nano, etc, were not available at this prompt. This emergency prompt was running entirely within initramfs.

At reboot I was right back where I started, at a locked console, unless as noted at top, I had a configured emergency.service already installed into systemd.

creation of emergency.service

helpful, but not a fix

Linuxconfig.org. Setting kernel boot parameters.

nope 1

Kali solution. Well documented kernel parameter addition. GRUB ignores.

single init=/bin/bash

nope 2

Well documented kernel parameter addition. GRUB ignores.

break

# mount -rw /dev/sda1 /mnt
# genfstab -p /mnt >> /mnt/etc/fstab
# arch-chroot /mnt
# passwd
# mkinitcpio -p linux
# grub-mkconfig -o /boot/grub/grub.cfg
# grub-install /dev/sda [no partition]
# exit