Arch Linux System maintenance

Check for errors

Failed systemd services

Check if any systemd services have failed:

1

$ systemctl --failed

Log files

Look for errors in the log files located in /var/log/, as well as messages logged in the systemd journal:

1

# journalctl -b

Backup

Configuration files

For tracking changes and backing up configuration files I use etckeeper for /etc and chezmoi (really cool piece of software) for my dotfiles. To back up my data I recently switched to restic. Previously I used rsync, but restic is way faster.

Dotfiles

Backing up

For dotfiles (configuration files in the home directory):

1
2
3
4


$ git init --bare ~/code/repos/dotfiles
$ alias dotfiles='/usr/bin/git --git-dir=$HOME/code/repos/dotfiles/ --work-tree=$HOME'
$ dotfiles config status.showUntrackedFiles no
$ echo "alias dotfiles='/usr/bin/git --git-dir=$HOME/code/repos/dotfiles/ --work-tree=$HOME'" >> $HOME/.bashrc

1
2
3


$ dotfiles status
$ dotfiles add .bashrc
$ dotfiles commit -m 'Initial commit with .bashrc'

Create a new repository in GitHub, and then set remote SSH URL

1

$ dotfiles remote set-url origin git@github.com:csyezheng/dotfiles.git

1

$ dotfiles push --set-upstream origin master

automate backups:

1

$ vim /home/ye/bin/backup-dotfiles.sh

1
2
3
4
5
6
7
8
9


#!/bin/zsh

setopt aliases

alias dotfiles='/usr/bin/git --git-dir=$HOME/code/repos/dotfiles/ --work-tree=$HOME'

dotfiles add -u
dotfiles commit -m 'Update dotfiles'
dotfiles push origin master

1

$ systemctl edit --user --force --full backup-dotfiles.service

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


[Unit]
Description=Backup Dotfiles
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
ExecStart=/home/ye/bin/backup-dotfiles.sh

[Install]
WantedBy=default.target

1

$ systemctl edit --user --force --full backup-dotfiles.timer

1
2
3
4
5
6
7
8
9


[Unit]
Description=Run backup dotfiles daily and on boot

[Timer]
OnBootSec=10min
OnUnitActiveSec=1d

[Install]
WantedBy=timers.target

1
2
3


$ systemctl --user start backup-dotfiles.timer
$ systemctl --user enable backup-dotfiles.timer
$ systemctl --user status backup-dotfiles.timer

1
2


$ systemctl --user list-timers
$ journalctl --user -xeu backup-dotfiles.service

Find all dotfiles using

1

find . -maxdepth 3 -type f -name '\.*' -print

Restoring in a New Machine

To set up a new machine to use your version controlled config files, all you need to do is to clone the repository on your new machine telling git that it is a bare repository:

1

git clone --separate-git-dir=$HOME/.dotfiles https://github.com/anandpiyer/.dotfiles.git ~

However, some programs create default config files, so this might fail if git finds an existing config file in your $HOME. In that case, a simple solution is to clone to a temporary directory, and then delete it once you are done:

1
2
3


git clone --separate-git-dir=$HOME/.dotfiles https://github.com/anandpiyer/.dotfiles.git tmpdotfiles
rsync --recursive --verbose --exclude '.git' tmpdotfiles/ $HOME/
rm -r tmpdotfiles

etc files

1
2
3
4


$ sudo su
# pacman -S etckeeper
# cd /etc
# etckeeper init

1

# vim .gitignore

1
2
3


ca-certificates/ 
java11-openjdk/ 
ssl/

1

# etckeeper vcs rm --cached -r ca-certificates/ java11-openjdk/ ssl/

1

# etckeeper vcs add .gitignore

1

# etckeeper commit "first commit"

1
2
3


# etckeeper vcs remote add origin git@github.com:csyezheng/etc.git
# etckeeper vcs branch -M main
# etckeeper vcs push -u origin main

automate backups:

1

# vim /root/bin/backup-etc.sh

1
2
3
4
5
6


#!/bin/zsh

cd /etc
etckeeper vcs add .
etckeeper vcs commit -m 'Update /etc files'
etckeeper vcs push origin main

1

# systemctl edit --force --full backup-etc.service

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


[Unit]
Description=Backup /etc files
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
ExecStart=/root/bin/backup-etc.sh

[Install]
WantedBy=default.target

1

# systemctl edit --force --full backup-etc.timer

1
2
3
4
5
6
7
8
9


[Unit]
Description=Run backup /etc files daily and on boot

[Timer]
OnBootSec=10min
OnUnitActiveSec=1d

[Install]
WantedBy=timers.target

1
2
3


# systemctl start backup-etc.timer
# systemctl enable backup-etc.timer
# systemctl status backup-etc.timer

1
2


# systemctl list-timers
# journalctl -xeu backup-etc.service

List of installed packages

Backup package list

1

$ sudo vim /etc/pacman.d/hooks/export-native-pkgs.hook

1
2
3
4
5
6
7
8
9


[Trigger]
Operation = Install
Operation = Remove
Type = Package
Target = *

[Action]
When = PostTransaction
Exec = /bin/sh -c '/usr/bin/pacman -Qqen > /home/ye/.native-pkg-list.txt'

1

$ sudo vim /etc/pacman.d/hooks/export-foreign-pkgs.hook

1
2
3
4
5
6
7
8
9


[Trigger]
Operation = Install
Operation = Remove
Type = Package
Target = *

[Action]
When = PostTransaction
Exec = /bin/sh -c '/usr/bin/pacman -Qqem > /home/ye/.foreign-pkg-list.txt'

1
2
3


$ cd /etc
$ sudo etckeeper vcs add -f /etc/pacman.d/hooks/export-native-pkgs.hook
$ sudo etckeeper vcs add -f /etc/pacman.d/hooks/export-foreign-pkgs.hook

1

$ dotfiles add /home/ye/.native-pkg-list.txt /home/ye/.foreign-pkg-list.txt

Install packages from a list

1

$ sudo pacman -S --needed - < /home/ye/.native-pkg-list.txt

1

$ sudo yay -S --needed - < /home/ye/.foreign-pkg-list.txt

Back up the pacman database

Backup:

1

$ tar -cjf pacman_database.tar.bz2 /var/lib/pacman/local

Restoring:

1

$ sudo mv pacman_database.tar.bz2 /

1

$ sudo tar -xjvf pacman_database.tar.bz2

System backup

Full system backup with restic

1

$ sudo pacman -S restic

1
2


$ mkdir -p /data/backup/archlinux_system_backup
$ restic init --repo /data/backup/archlinux_system_backup

1
2


enter password for new repository: restic
enter password again: restic

Systemd Service

1

$ sudo vim /etc/systemd/system/restic-backup.service

1
2
3
4
5


[Unit]
Description=Backup system

[Service]
ExecStart=systemd-inhibit /usr/local/bin/restic-backup

Systemd timer

1

$ sudo vim /etc/systemd/system/restic-backup.timer

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


[Unit]
Description=Timer for full system backups

[Timer]
OnBootSec=5min
OnUnitActiveSec=15min
Unit=restic-backup.service

[Install]
WantedBy=timers.target

Backup script

1

$ sudo vim /usr/local/bin/restic-backup

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


#!/bin/bash

if [[ -n $(pgrep 'restic' | grep 'restic backup') ]]; then
  echo 'restic is already running...' 1>&2
  exit 0
fi

set -e
set -v

export RESTIC_REPOSITORY='/data/backup/archlinux_system_backup'
export RESTIC_PASSWORD_COMMAND='/usr/local/bin/get-restic-password'
export RESTIC_COMPRESSION='off'
export RESTIC_CACHE_DIR=~/.cache/restic

mkdir -p "${RESTIC_CACHE_DIR}"

restic unlock 
restic backup / --exclude-file=/etc/restic/excludes.txt --tag scheduled 
restic check --with-cache --read-data-subset=5G
restic forget --prune --keep-hourly 24 --keep-daily 30 --keep-monthly 6 --keep-weekly 4 --keep-yearly 3

1

$ sudo vim /usr/local/bin/get-restic-password

1
2
3


#!/bin/bash

echo restic

1
2


$ sudo chmod 744 /usr/local/bin/restic-backup
$ sudo chmod 700 /usr/local/bin/get-restic-password

Excludes list

1
2


$ sudo mkdir /etc/restic
$ sudo vim /etc/restic/excludes.txt

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


/data/**
/dev/**
/home/*/**/*.pyc
/home/*/**/__pycache__/**
/home/*/**/node_modules/**
/home/*/.cache/**
/home/*/.local/lib/python*/site-packages/**
/home/*/.mozilla/firefox/*/Cache/**
/lost+found/**
/media/**
/mnt/**
/proc/**
/root/**
/run/**
/swapfile
/sys/**
/tmp/**
/var/cache/**
/var/cache/pacman/pkg/**
/var/lib/docker/**
/var/lib/libvirt/**
/var/lock/**
/var/log/**
/var/run/**

Enable

1
2


$ systemctl start restic-backup.timer
$ systemctl enable restic-backup.timer

Backup /boot partition

1

$ sudo vim /etc/pacman.d/hooks/95-bootbackup.hook

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


[Trigger]
Operation = Upgrade
Operation = Install
Operation = Remove
Type = Path
Target = usr/lib/modules/*/vmlinuz

[Action]
Depends = rsync
Description = Backing up /boot...
When = PostTransaction
Exec = /usr/bin/rsync -a --delete /boot /.bootbackup

Upgrading the system

If experiencing trouble after an update, double-check pacman’s output by looking at /var/log/pacman.log.

Downgrading packages

If a package was installed at an earlier stage, and the pacman cache was not cleaned, install an earlier version from /var/cache/pacman/pkg/.

1

# pacman -U file:///var/cache/pacman/pkg/package-old_version.pkg.tar.type

Once the package is reverted, temporarily add it to the IgnorePkg section of pacman.conf, until the difficulty with the updated package is resolved.

Downgrading the kernel

If you are unable to boot after a kernel update, Boot using an Arch Linux USB flash installation media and mount the partition where your system is installed to /mnt. If you have /boot or /var on separate partitions, also mount them to /mnt (e.g. mount /dev/sdc3 /mnt/boot). Then chroot into the system using:

1

# arch-chroot /mnt

Now you can go into the directory /var/cache/pacman/pkg and downgrade at least linux, linux-headers and any kernel modules.

1

# pacman -U file://linux-4.15.8-1-x86_64.pkg.tar.xz file://linux-headers-4.15.8-1-x86_64.pkg.tar.xz file://virtualbox-host-modules-arch-5.2.8-4-x86_64.pkg.tar.xz

Once done, exit the chroot (with exit) and reboot.

Rebuild the package

If the package is unavailable, find the correct PKGBUILD and rebuild it with makepkg.

For packages from the official repositories, retrieve the PKGBUILD with ABS and change the software version. Alternatively, find the package on the Packages website, click “View Changes”, and navigate to the desired version. The necessary files can then be downloaded from the directory so that the package can be rebuilt.

Check for orphans and dropped packages

1

$ sudo vim /etc/pacman.d/hooks/check-orphans-pkgs.hook

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


[Trigger]
Operation=Install
Operation=Upgrade
Operation=Remove
Type=Package
Target=*

[Action]
Description=Log Orphan Packages
When=PostTransaction
Exec=/bin/bash -c 'pkgs="$(pacman -Qtdq)"; if [[ ! -z "$pkgs" ]]; then echo -e "The following packages are installed but not required (anymore):\n$pkgs\nYou can mark them as explicitly installed with '\''pacman -D --asexplicit <pkg>'\'' or remove them all using '\''pacman -Qtdq | pacman -Rns -'\''"; fi'

Troubleshooting

file exists in filesystem

force pacman to overwrite files that match glob.

1

# pacman -S --overwrite glob package

unable to lock database

1

# rm /var/lib/pacman/db.lck

“Unable to find root device” error after rebooting

There are two options; first, try the Fallback entry.

In case you removed the Fallback entry, you can always press the Tab key when the boot loader menu shows up (for Syslinux) or e (for GRUB or systemd-boot), rename it initramfs-linux-fallback.img or initramfs-linux-lts-fallback.img and press Enter or b (depending on your boot loader) to boot with the new parameters.

Once the system starts, run this command (for the stock linux kernel) either from the console or from a terminal to rebuild the initramfs image:
1 2 3

# mkinitcpio -p linux or # mkinitcpio -p linux-lts
If that does not work, from a current Arch release (CD/DVD or USB stick), mount your root and boot partitions to /mnt and /mnt/boot, respectively. Then chroot using arch-chroot:
1 2

# arch-chroot /mnt # pacman -Syu mkinitcpio systemd linux
Reinstalling the kernel (the linux package) will automatically re-generate the initramfs image with mkinitcpio -p linux. There is no need to do this separately.

Afterwards, it is recommended that you run exit, umount /mnt/{boot,} and reboot.
- If pacman fails with Could not resolve host, please check your internet connection.
- If you cannot enter the arch-chroot or chroot environment but need to re-install packages, you can use the command pacman --sysroot /mnt -Syu foo bar to use pacman on your root partition.

Proxy settings

Make sure that the relevant environment variables ($http_proxy, $ftp_proxy etc.) are set up. If you use pacman with sudo, you need to configure sudo to pass these environment variables to pacman.

1

$ sudo -E pacman -Syu

The recommended way of preserving environment variables is to append them to env_keep:

1

$ vim /etc/sudoers

1

Defaults env_keep += "ftp_proxy http_proxy https_proxy no_proxy"

Use the package manager to install software

Update the mirrorlist

To get an even more up-to-date list of mirrors, use the Pacman Mirrorlist Generator page.

To enable mirrors, edit /etc/pacman.d/mirrorlist and locate your geographic region.

1

$ sudo su

1

# pacman -S pacman-contrib

1

# cp /etc/pacman.d/mirrorlist /etc/pacman.d/mirrorlist.backup

1

# curl -s "https://archlinux.org/mirrorlist/?country=CN&protocol=https&use_mirror_status=on" | sed -e 's/^#Server/Server/' -e '/^#/d' | rankmirrors -n 10 - > /etc/pacman.d/mirrorlist

1

# cat /etc/pacman.d/mirrorlist

Clean the filesystem

Cleaning the package cache

1

# paccache -r

1
2


# systemctl start paccache.timer
# systemctl enable paccache.timer

Removing unused packages (orphans)

1

# pacman -Qtdq | pacman -Rns -

Tips and tricks

Avoid any use of the testing repository, even individual packages from testing. Similarly, avoid packages which are built directly from upstream development sources. These are usually found in the AUR, with names including things like: “dev”, “devel”, “svn”, “cvs”, “git”, etc.
Install the linux-lts package. To make it available as a boot option, you will need to update your boot loader’s configuration file to use the LTS kernel and ram disk: vmlinuz-linux-lts and initramfs-linux-lts.img.

Reference

System maintenance

Table of Contents