Synchronization and backup

Backup plan

File type	Frequency	Tools and Storage	Sync / Copy	Encryption
dotfiles	daily	git + github	Sync	N
/etc files	daily	etckeeper + git + github	Sync	N
scripts	daily	git + github	Sync	N
documents	daily	rclone + onedrive / google drive	Sync	N
pictures	weekly	rclone + dropbox / pcloud	Copy	Y
private/sensitive data	weekly	rclone + dropbox / pcloud	Copy	Y
learning materials (courses + languages)	weekly	rclone + alist + baidunetdisk / aliyundrive	Copy	N
ebooks	weekly	rclone + alist + baidunetdisk / aliyundrive	Copy	N
music	weekly	rclone + alist + baidunetdisk / aliyundrive	Copy	N
movie	weekly	rclone + alist + baidunetdisk / aliyundrive	Copy	N
applications	monthly	rclone + alist + baidunetdisk / aliyundrive	Copy	N
system backup	monthly	restic + Hard disk drive	Sync	Y
*	–	syncthing	Sync	N

Configuration files

For tracking changes and backing up configuration files I use etckeeper for /etc and chezmoi (really cool piece of software) for my dotfiles. To back up my data I recently switched to restic. Previously I used rsync, but restic is way faster.

Dotfiles

Initial

For dotfiles (configuration files in the home directory):

1
2
3
4


$ git init --bare $HOME/.dotfiles
$ alias dotfiles='/usr/bin/git --git-dir=$HOME/.dotfiles/ --work-tree=$HOME'
$ dotfiles config status.showUntrackedFiles no
$ echo "alias dotfiles='/usr/bin/git --git-dir=$HOME/.dotfiles/ --work-tree=$HOME'" >> $HOME/.zshrc

1
2
3


$ dotfiles status
$ dotfiles add .bashrc
$ dotfiles commit -m 'Initial commit with .bashrc'

Create a new repository in GitHub, and then set remote SSH URL

1

$ dotfiles remote set-url origin git@github.com:csyezheng/dotfiles.git

1

$ dotfiles push --set-upstream origin master

Backup Script

1

$ vim /home/ye/bin/backup-dotfiles.sh

1
2
3
4
5
6
7
8
9


#!/bin/zsh

setopt aliases

alias dotfiles='/usr/bin/git --git-dir=$HOME/code/personal/dotfiles/ --work-tree=$HOME'

dotfiles add -u
dotfiles commit -m 'Update dotfiles'
dotfiles push origin master

Systemd Service

1

$ systemctl edit --user --force --full backup-dotfiles.service

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


[Unit]
Description=Backup Dotfiles
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
ExecStart=/home/ye/bin/backup-dotfiles.sh

[Install]
WantedBy=default.target

Systemd Timers

1

$ systemctl edit --user --force --full backup-dotfiles.timer

1
2
3
4
5
6
7
8
9


[Unit]
Description=Run backup dotfiles daily and on boot

[Timer]
OnBootSec=10min
OnUnitActiveSec=1d

[Install]
WantedBy=timers.target

1
2
3


$ systemctl --user start backup-dotfiles.timer
$ systemctl --user enable backup-dotfiles.timer
$ systemctl --user status backup-dotfiles.timer

1
2


$ systemctl --user list-timers
$ journalctl --user -xeu backup-dotfiles.service

Find all dotfiles using

1

find . -maxdepth 3 -type f -name '\.*' -print

Restoring

To set up a new machine to use your version controlled config files, all you need to do is to clone the repository on your new machine telling git that it is a bare repository:

1

git clone --separate-git-dir=$HOME/.dotfiles https://github.com/csyezheng/dotfiles.git $HOME

However, some programs create default config files, so this might fail if git finds an existing config file in your $HOME. In that case, a simple solution is to clone to a temporary directory, and then delete it once you are done:

1
2
3


git clone --separate-git-dir=$HOME/.dotfiles https://github.com/csyezheng/dotfiles.git $HOME/tmp-dotfiles
rsync --recursive --verbose --exclude '.git' $HOME/tmp-dotfiles/ $HOME/
rm -r $HOME/tmp-dotfiles/

/etc files

Initial

1
2
3
4


$ sudo su
# pacman -S etckeeper
# cd /etc
# etckeeper init

1

# vim .gitignore

1
2
3


ca-certificates/ 
java11-openjdk/ 
ssl/

1

# etckeeper vcs rm --cached -r ca-certificates/ java11-openjdk/ ssl/

1

# etckeeper vcs add .gitignore

1

# etckeeper commit "first commit"

1
2
3


# etckeeper vcs remote add origin git@github.com:csyezheng/etc.git
# etckeeper vcs branch -M main
# etckeeper vcs push -u origin main

Backup Script

1

# vim /root/bin/backup-etc.sh

1
2
3
4
5
6


#!/bin/zsh

cd /etc
etckeeper vcs add .
etckeeper vcs commit -m 'Update /etc files'
etckeeper vcs push origin main

Systemd Service

1

# systemctl edit --force --full backup-etc.service

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


[Unit]
Description=Backup /etc files
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
ExecStart=/root/bin/backup-etc.sh

[Install]
WantedBy=default.target

Systemd Timers

1

# systemctl edit --force --full backup-etc.timer

1
2
3
4
5
6
7
8
9


[Unit]
Description=Run backup /etc files daily and on boot

[Timer]
OnBootSec=10min
OnUnitActiveSec=1d

[Install]
WantedBy=timers.target

1
2
3


# systemctl start backup-etc.timer
# systemctl enable backup-etc.timer
# systemctl status backup-etc.timer

1
2


# systemctl list-timers
# journalctl -xeu backup-etc.service

Restoring

List of installed packages

hook

1

$ sudo vim /etc/pacman.d/hooks/export-native-pkgs.hook

1
2
3
4
5
6
7
8
9


[Trigger]
Operation = Install
Operation = Remove
Type = Package
Target = *

[Action]
When = PostTransaction
Exec = /bin/sh -c '/usr/bin/pacman -Qqen > /home/ye/.native-pkg-list.txt'

1

$ sudo vim /etc/pacman.d/hooks/export-foreign-pkgs.hook

1
2
3
4
5
6
7
8
9


[Trigger]
Operation = Install
Operation = Remove
Type = Package
Target = *

[Action]
When = PostTransaction
Exec = /bin/sh -c '/usr/bin/pacman -Qqem > /home/ye/.foreign-pkg-list.txt'

1
2
3


$ cd /etc
$ sudo etckeeper vcs add -f /etc/pacman.d/hooks/export-native-pkgs.hook
$ sudo etckeeper vcs add -f /etc/pacman.d/hooks/export-foreign-pkgs.hook

1

$ dotfiles add /home/ye/.native-pkg-list.txt /home/ye/.foreign-pkg-list.txt

Reinstalling

1

$ sudo pacman -S --needed - < /home/ye/.native-pkg-list.txt

1

$ sudo yay -S --needed - < /home/ye/.foreign-pkg-list.txt

Pacman database

1

$ mkdir -p /data/backup/pacman_database

Systemd Service

1

$ systemctl edit --user --force --full backup-pacman-database.service

1
2
3
4
5
6
7
8
9


[Unit]
Description=Backup pacman database

[Service]
Type=oneshot
ExecStart=tar -cjf /data/backup/pacman_database/pacman_database.tar.bz2 /var/lib/pacman/local

[Install]
WantedBy=default.target

Systemd Timers

1

$ systemctl edit --user --force --full backup-pacman-database.timer

1
2
3
4
5
6
7
8
9


[Unit]
Description=Run backup pacman database one week and on boot

[Timer]
OnBootSec=10min
OnUnitActiveSec=1w

[Install]
WantedBy=timers.target

1

$ systemctl enable --user backup-pacman-database.timer

Restoring

1

$ sudo mv /data/backup/pacman_database/pacman_database.tar.bz2 /

1

$ sudo tar -xjvf pacman_database.tar.bz2

List of enabled systemd units

Systemd Service

1

$ systemctl edit --user --force --full backup-enabled-systemd-units.service

1
2
3
4
5
6
7
8
9


[Unit]
Description=Backup enabled systemd units list

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'systemctl list-unit-files --state enabled --type timer --type service | tail -n +2 | head -n -2 | cut -d\  -f1 > /home/ye/.enabled-units-list.txt && systemctl list-unit-files --state enabled --type timer --type service --user | tail -n +2 | head -n -2 | cut -d\  -f1 > /home/ye/.enabled-user-units-list.txt'

[Install]
WantedBy=default.target

Systemd Timers

1

$ systemctl edit --user --force --full backup-enabled-systemd-units.timer

1
2
3
4
5
6
7
8
9


[Unit]
Description=Run backup enabled systemd units list daily and on boot

[Timer]
OnBootSec=10min
OnUnitActiveSec=1d

[Install]
WantedBy=timers.target

1

$ systemctl enable --user backup-enabled-systemd-units.timer

Restoring

1

$ sudo systemctl enable $(< $HOME/.enabled-units-list.txt)

1

$ systemctl enable --user $(< $HOME/.enabled-user-units-list.txt)

User data

Synchronize user data to OneDrive Using rclone

1

$ sudo pacman -S rclone

Creating Client ID for OneDrive Personal

To create your own Client ID, please follow these steps:

Open https://portal.azure.com/#blade/Microsoft_AAD_RegisteredApps/ApplicationsListBlade and then click New registration.
Enter a name for your app, choose account type Accounts in any organizational directory (Any Azure AD directory - Multitenant) and personal Microsoft accounts (e.g. Skype, Xbox), select Web in Redirect URI, then type (do not copy and paste) http://localhost:53682/ and click Register. Copy and keep the Application (client) ID under the app name for later use.
Under manage select Certificates & secrets, click New client secret. Enter a description (can be anything) and set Expires to 24 months. Copy and keep that secret Value for later use (you won’t be able to see this value afterwards).
Under manage select API permissions, click Add a permission and select Microsoft Graph then select delegated permissions.
Search and select the following permissions: Files.Read, Files.ReadWrite, Files.Read.All, Files.ReadWrite.All, offline_access, User.Read and Sites.Read.All (if custom access scopes are configured, select the permissions accordingly). Once selected click Add permissions at the bottom.

Now the application is complete. To configure a remote storage, simply run the following command:

1

$ rclone config

Listing a remote:

1

$ rclone ls remote:path

Scripts

1

$ vim ~/bin/rclone_onedrive.sh

1
2
3
4
5
6
7
8


#!/bin/zsh

export http_proxy=socks5://127.0.0.1:20170
export https_proxy=$http_proxy
export HTTP_PROXY=$http_proxy
export HTTPS_PROXY=$http_proxy

rclone sync --progress /data/bin onedrive:/bin

1

$ chmod +x ~/bin/rclone_onedrive.sh

Systemd Service

1

$ systemctl edit --user --force --full rclone-onedrive.service

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


[Unit]
Description=Backup to OneDrive using rclone
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
ExecStart=/home/ye/bin/rclone_onedrive.sh

[Install]
WantedBy=default.target

Systemd Timers

1

$ systemctl edit --user --force --full rclone-onedrive.timer

1
2
3
4
5
6
7
8
9


[Unit]
Description=Backup to OneDrive using rclone

[Timer]
OnBootSec=15min
OnUnitActiveSec=1d

[Install]
WantedBy=timers.target

1

$ systemctl enable --user rclone-onedrive.timer

TODO: send email when error occuring

Mounting OneDrive in the file manager (Optional)

1

$ sudo pacman -S fuse3

1

$ mkdir ~/clouds/OneDrive

1

$ rclone mount onedrive:/ ~/clouds/OneDrive --vfs-cache-mode off --vfs-cache-max-age 20m --copy-links --no-gzip-encoding --no-check-certificate --allow-other --allow-non-empty --umask 000 --daemon

This command will mount one drive in the given location and will continue to run in the terminal. When you stop the process with,ctrl + c the one drive will be unmounted.

stop the mount manually:

1

$ fusermount -u ~/clouds/OneDrive

Synchronize user data to 139Yun Using rclone and alist

Setup alist

install alist:

1

# curl -fsSL "https://alist.nn.ci/v3.sh" | bash -s install

1

# systemctl enable alist

Manually set a password

1
2


cd /opt/alist/
./alist admin set password

Now open http://127.0.0.1:5244 You can see the login page. Click the Login button at the bottom to log in and then click Manage button to enter the background page, click storages button and add storage on the storage page. Choose 139Yun drive.

Root folder id can be found in getdisk request payload
Type select Personal Cloud
Authorization can be found in getdisk request header
Set mount point as /139yun

139Yun config

rclone config webdav

create a new config:

1

$ rclone config

1
2
3
4
5
6


name: webdav
type: webdav
url: http://127.0.0.1:5244/dav/
vendor: Other
user: admin
password: password

List directories in top level of your WebDAV:

1

$ rclone lsd webdav:/139Yun

Scripts

1

$ vim ~/bin/rclone_139yun.sh

1
2
3
4
5
6
7
8


#!/bin/zsh

rclone copy --progress /data/languages webdav:/139yun/applications
rclone copy --progress /data/courses webdav:/139yun/courses
rclone copy --progress /data/datasets webdav:/139yun/datasets
rclone copy --progress /data/ebooks webdav:/139yun/ebooks
rclone copy --progress /data/languages webdav:/139yun/languages
rclone copy --progress /data/music webdav:/139yun/music

1

$ chmod +x ~/bin/rclone_139yun.sh

Systemd Service

1

$ systemctl edit --user --force --full rclone-139yun.service

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


[Unit]
Description=Backup to 139yun using rclone
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
ExecStart=/home/ye/bin/rclone_139yun.sh

[Install]
WantedBy=default.target

Systemd Timers

1

$ systemctl edit --user --force --full rclone-139yun.timer

1
2
3
4
5
6
7
8
9


[Unit]
Description=Backup to 139yun using rclone

[Timer]
OnBootSec=30min
OnUnitActiveSec=1w

[Install]
WantedBy=timers.target

1

$ systemctl enable --user rclone-139yun.timer

Mounting OneDrive in the file manager (Optional)

1

$ mkdir ~/clouds/139yun

1

$ rclone mount webdav:/139yun ~/clouds/139yun --vfs-cache-mode off --vfs-cache-max-age 20m --copy-links --no-gzip-encoding --no-check-certificate --allow-other --allow-non-empty --umask 000 --daemon

stop the mount manually:

1

$ fusermount -u ~/clouds/139yun

Synchronize user data to 189Cloud Using rclone and alist

Setup alist

install alist:

1

# curl -fsSL "https://alist.nn.ci/v3.sh" | bash -s install

1

# systemctl enable alist

Manually set a password

1
2


cd cd /opt/alist/
./alist admin set password

Root folder id: -11
username
password
cookie
Set mount point as /189cloud

189Cloud config

rclone config webdav

create a new config:

1

$ rclone config

1
2
3
4
5
6


name: webdav
type: webdav
url: http://127.0.0.1:5244/dav/
vendor: Other
user: admin
password: password

List directories in top level of your WebDAV:

1

$ rclone lsd webdav:/189cloud

Scripts

1

$ vim ~/bin/rclone_189cloud.sh

1
2
3
4
5
6
7
8


#!/bin/zsh

rclone copy --progress /data/languages webdav:/189cloud/applications
rclone copy --progress /data/courses webdav:/189cloud/courses
rclone copy --progress /data/datasets webdav:/189cloud/datasets
rclone copy --progress /data/ebooks webdav:/189cloud/ebooks
rclone copy --progress /data/languages webdav:/189cloud/languages
rclone copy --progress /data/music webdav:/189cloud/music

1

$ chmod +x ~/bin/rclone_189cloud.sh

Systemd Service

1

$ systemctl edit --user --force --full rclone-189cloud.service

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


[Unit]
Description=Backup to 189cloud using rclone
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
ExecStart=/home/ye/bin/rclone_189cloud.sh

[Install]
WantedBy=default.target

Systemd Timers

1

$ systemctl edit --user --force --full rclone-189cloud.timer

1
2
3
4
5
6
7
8
9


[Unit]
Description=Backup to 189cloud using rclone

[Timer]
OnBootSec=30min
OnUnitActiveSec=1w

[Install]
WantedBy=timers.target

1

$ systemctl enable --user rclone-189cloud.timer

Mounting OneDrive in the file manager (Optional)

1

$ mkdir ~/clouds/139yun

1

$ rclone mount webdav:/139yun ~/clouds/139yun --vfs-cache-mode off --vfs-cache-max-age 20m --copy-links --no-gzip-encoding --no-check-certificate --allow-other --allow-non-empty --umask 000 --daemon

stop the mount manually:

1

$ fusermount -u ~/clouds/139yun

Synchronize user data to Baidu Netdisk Using rclone and alist

Setup alist

install alist:

1

# curl -fsSL "https://alist.nn.ci/v3.sh" | bash -s install

1

# systemctl enable alist

Manually set a password

1
2


cd cd /opt/alist/
./alist admin set password

Set mount point as /BaiduNetdisk

Baidu Netdisk config

rclone config webdav

create a new config:

1

$ rclone config

1
2
3
4
5
6


name: webdav
type: webdav
url: http://127.0.0.1:5244/dav/
vendor: Other
user: admin
password: password

List directories in top level of your WebDAV:

1

$ rclone lsd webdav:/BaiduNetdisk

Mounting OneDrive in the file manager (Optional)

1

$ mkdir ~/clouds/139yun

1

$ rclone mount webdav:/BaiduNetdisk ~/clouds/BaiduNetdisk --vfs-cache-mode off --vfs-cache-max-age 20m --copy-links --no-gzip-encoding --no-check-certificate --allow-other --allow-non-empty --umask 000 --daemon

stop the mount manually:

1

$ fusermount -u ~/clouds/BaiduNetdisk

EFI partition

1

$ mkdir -p /data/backup/archlinux_efi

Systemd Service

1

# systemctl edit --force --full backup-efi.service

1
2
3
4
5
6
7
8
9


[Unit]
Description=Backing up EFI partition

[Service]
Type=oneshot
ExecStart=/usr/bin/rsync -a --delete /efi /data/backup/archlinux_efi

[Install]
WantedBy=default.target

Systemd Timers

1

# systemctl edit --force --full backup-efi.timer

1
2
3
4
5
6
7
8
9


[Unit]
Description=Run backing up EFI partition one week and on boot

[Timer]
OnBootSec=15min
OnUnitActiveSec=1w

[Install]
WantedBy=timers.target

1

# systemctl enable backup-efi.timer

/boot partition

1

$ mkdir -p /data/backup/archlinux_boot

1

$ vim /etc/pacman.d/hooks/95-bootbackup.hook

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


[Trigger]
Operation = Upgrade
Operation = Install
Operation = Remove
Type = Path
Target = usr/lib/modules/*/vmlinuz

[Action]
Depends = rsync
Description = Backing up /boot...
When = PostTransaction
Exec = /usr/bin/rsync -a --delete /boot /data/backup/archlinux_boot

System data

Full system backup using restic

Initial

1

$ sudo pacman -S restic

1

$ mkdir -p /data/backup/restic

1

$ restic init --repo /data/backup/restic

1
2


enter password for new repository: restic
enter password again: restic

Systemd Service

1

$ vim /etc/systemd/system/restic-backup.service

1
2
3
4
5


[Unit]
Description=Backup system

[Service]
ExecStart=systemd-inhibit /usr/local/bin/restic-backup

1

$ vim /etc/systemd/system/restic-backup.timer

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


[Unit]
Description=Timer for full system backups

[Timer]
OnBootSec=5min
OnUnitActiveSec=15min
Unit=restic-backup.service

[Install]
WantedBy=timers.target

Backup script

1

$ vim /usr/local/bin/restic-backup

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


#!/bin/bash

if [[ -n $(pgrep 'restic' | grep 'restic backup') ]]; then
  echo 'restic is already running...' 1>&2
  exit 0
fi

set -e
set -v

export RESTIC_REPOSITORY='/mnt/restic'
export RESTIC_PASSWORD_COMMAND='/usr/local/bin/get-restic-password'
export RESTIC_COMPRESSION='off'
export RESTIC_CACHE_DIR=~/.cache/restic

mkdir -p "${RESTIC_CACHE_DIR}"

restic unlock 
restic backup / --exclude-file=/etc/restic/excludes.txt --tag scheduled 
restic check --with-cache --read-data-subset=5G

# Snapshot retention
# Adjust the values if wanted.
restic forget --prune --keep-hourly 24 --keep-daily 30 --keep-monthly 6 --keep-weekly 4 --keep-yearly 3

1

$ vim /usr/local/bin/get-restic-password

1
2


#!/bin/bash
echo restic

1
2


# chmod 744 /usr/local/bin/restic-backup
# chmod 700 /usr/local/bin/get-restic-password

Excludes list

1

$ vim /etc/restic/excludes.txt

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


/data/**
/dev/**
/home/*/**/*.pyc
/home/*/**/__pycache__/**
/home/*/**/node_modules/**
/home/*/.cache/**
/home/*/.local/lib/python*/site-packages/**
/home/*/.mozilla/firefox/*/Cache/**
/lost+found/**
/media/**
/mnt/**
/proc/**
/root/**
/run/**
/swapfile
/sys/**
/tmp/**
/var/cache/**
/var/cache/pacman/pkg/**
/var/lib/docker/**
/var/lib/libvirt/**
/var/lock/**
/var/log/**
/var/run/**

Enable timer

1

# systemctl enable restic-backup.timer

Reference

Tracking_dotfiles

etckeeper

List of installed packages

Back up the pacman database

rclone

rclone subcommands

Restic

Table of Contents

Backup plan

Configuration files

Dotfiles

Initial

Backup Script

Systemd Service

Systemd Timers

Restoring

/etc files

Initial

Backup Script

Systemd Service

Systemd Timers

Restoring

List of installed packages

hook

Reinstalling

Pacman database

Systemd Service

Systemd Timers

Restoring

List of enabled systemd units

Systemd Service

Systemd Timers

Restoring

User data

Synchronize user data to OneDrive Using rclone

Creating Client ID for OneDrive Personal

Scripts

Systemd Service

Systemd Timers

TODO: send email when error occuring

Mounting OneDrive in the file manager (Optional)

Synchronize user data to 139Yun Using rclone and alist

Setup alist

rclone config webdav

Scripts

Systemd Service

Systemd Timers

Mounting OneDrive in the file manager (Optional)

Synchronize user data to 189Cloud Using rclone and alist

Setup alist

rclone config webdav

Scripts

Systemd Service

Systemd Timers

Mounting OneDrive in the file manager (Optional)

Synchronize user data to Baidu Netdisk Using rclone and alist

Setup alist

rclone config webdav

Mounting OneDrive in the file manager (Optional)

EFI partition

Systemd Service

Systemd Timers

/boot partition

System data

Full system backup using restic

Initial

Systemd Service

Backup script

Excludes list

Enable timer

Reference