If you’re like me, you’re now relying on multiple Raspberry Pi devices across your home to perform various home automation tasks. I’m pretty good about backing up my devices using rsync, but even having file backups doesn’t replace a full OS-level backup. I recently lost 2 SD cards at the same time on different devices and spent hours recreating the content on each - in both situations, the reisntalled OS was newer than before so packages didnt match one for one, and my git repos had newer content, thus not everything worked as it did before. After replacing both SD cards, reinstalling and reconfiguring both systems, I wanted to make sure I didn’t have to go through that again. This post will hopefully explain how you can proactively prevent SD failure on your RPi devices by keeping a local, bootable and restorable clone of your device around - and in most cases you can use some spare USB flash drives to accomplish this.
What you need to get started
- rpi-clone - open source tool that allows you to clone your running RPi device to another device.
- Spare USB flash drive OR Spare SD Card + USB reader - If you have a RPi 2B or newer, you can use a USB flash drive whose size is slightly greater than the amount of space you have used on your existing SD card.
My search for a good backup solution took me to rpi-clone. This script simply allows you to clone your running RPi to USB flash drive or another SD card (connected to a USB reader). I chose the USB option as my devices are all 2B or newer and because I have a bunch of spare USB flash drives. USB booting is only available on Pi 2B v1.2, 3A+, 3B, and 3B via this method and Pi 4 via this method. The change, depending on your model, needs to happen before your devices can be booted from USB, thus you need to do this prior to your SD card failing. Follow the directions above to adjust the boot configuration or update the EEPROM of your device.
If your device is too old to support USB boot, I would recommend you use a USB to SD card adapter and an SD card in each RPi device you own. This will allow you to keep a backup available. Remember, you don’t need a USB device or SD card of the same size as you have on your device - it only has to be large enough to hold the the contents you’re using on your RPi - so you can likely re-purpose some smaller USB or SD cards to support your device backups.
Setting it up
Once you’ve pulled down rpi-clone, copy both rpi-clone and rpi-clone-setup to /usr/local/bin. Insert your USB drive or SD card in a USB reader into your Rpi and determine what disk it is. This will be sda in most cases. If you have other drives connected, you should make sure you have the right one as this process will destroy data on whatever device you specify. I use dmesg to see kernel messages and help me locate the right partition. Your drive may not look exactly like the below drive, but should once rpi-clone has been run once.
$ dmesg | grep sd
[ 2.412296] sd 0:0:0:0: [sda] 31266816 512-byte logical blocks: (16.0 GB/14.9 GiB)
[ 2.413584] sd 0:0:0:0: [sda] Write Protect is off
[ 2.413601] sd 0:0:0:0: [sda] Mode Sense: 43 00 00 00
[ 2.414303] sd 0:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 2.424847] sda: sda1 sda2
[ 2.428996] sd 0:0:0:0: [sda] Attached SCSI removable disk
Once you determine the correct drive, run rpi-clone via sudo passing it the raw device of your backup drive.
$ sudo /usr/local/bin/rpi-clone sda
Booted disk: mmcblk0 31.9GB Destination disk: sda 8.0GB
---------------------------------------------------------------------------
Part Size FS Label Part Size FS Label
1 /boot 256.0M fat32 -- 1 256.0M fat32 --
2 root 29.5G ext4 rootfs 2 7.2G ext4 --
---------------------------------------------------------------------------
== SYNC mmcblk0 file systems to sda ==
/boot (53.0M used) : SYNC to sda1 (256.0M size)
/ (2.3G used) : SYNC to sda2 (7.2G size)
---------------------------------------------------------------------------
Run setup script : no.
Verbose mode : no.
-----------------------:
Ok to proceed with the clone? (yes/no): y
Syncing file systems (can take a long time)
Syncing mounted partitions:
Mounting /dev/sda2 on /mnt/clone
=> rsync // /mnt/clone with-root-excludes ...
Mounting /dev/sda1 on /mnt/clone/boot
=> rsync /boot/ /mnt/clone/boot ...
Editing /mnt/clone/boot/cmdline.txt PARTUUID to use da13a66a
Editing /mnt/clone/etc/fstab PARTUUID to use da13a66a
===============================
Done with clone to /dev/sda
Start - 12:58:20 End - 12:59:39 Elapsed Time - 1:19
Cloned partitions are mounted on /mnt/clone for inspection or customizing.
Hit Enter when ready to unmount the /dev/sda partitions ...
unmounting /mnt/clone/boot
unmounting /mnt/clone
===============================
You’ll notice the booted SD card is 32GB and my backup drive is only 8GB - this is OK. If my data on my primary drive grows beyond 8GB, my automated backup job below will complain. After you’ve run this once, you can then run rpi-clone in “quite” mode. This mode will only report errors - it will also not run on top of a device that hasn’t already been cloned once - so there is no chance that you will corrupt a drive that wasn’t previously cloned to.
Next, add a cron entry to backup your drive weekly - you can probably do daily since rpi-clone is smart enough to only copy changed files - however I’d rather not wear out drives prematurely. I also have another backup process that syncs data off nightly, so I could easily re-construct a weeks worth of backups.
$ sudo crontab -e
0 23 * * 7 /usr/local/bin/rpi-clone sda -q
That’s it. My only additional caution is to make sure that your mail works on your system, otherwise you might not see any warnings/alerts. This is a good guide to do this - however I’d setup a gmail account specifically purposed to send mail from - do not use your primary mailbox as your password will be in the clear on your system.
Update 1: Booting from SSD
I recently decided to speed up my drive access times on my Raspberry Pi by moving to a bootable SSD drive. Luckily, this tool also lets you do that. Another benefit is that SSD have longer lifetimes than SD cards, so hopefully this will increase the longevity of the data on the drive. One thing changes with regards to cloning when you’re planning to boot using a USB SSD as your boot device - you run the same commands above to create your USB clone, but once you’ve done that, you’ll need to adjust your backup routine to backup your SSD drive to your old SD card. In order to do this, you’ll need to adjust your cron entry to be the command below:
$ sudo rpi-clone -l mmcblk0
This will now clone your USB SSD to your SD card, but will now instruct your SD card to continue to force the USB device to be the boot (and root) device. It does this by writing /boot/cmdline.txt to your SD card instructing it to use the USB device. The thing to remember here is if you need to actually boot from your SD card (i.e. say your SSD device is toast), then you’ll need to connect your SD card to a device where you can cp /boot/cmdline.boot /boot/cmdline.txt. rpi-clone keeps a copy of the real boot arguments for SD card in this file on the SD card in case they are needed. Here is the section in rpi-clone docs that cover this:
The -l option causes the SD card cmdline.txt to be backed up to cmdline.boot and the destination USB disk cmdline.txt to be copied to the SD card. Since the USB cmdline.txt was edited to reference the USB disk, the next Pi boot will start with the SD card /boot partition, but will redirect to using the USB root partition. Since the USB fstab was edited to reference the USB disk, the Pi will boot with the USB partition 1 mounted on /boot. The SD card /boot partition that initiated the boot process is no longer in use but can remain in place for subsequent SD card to USB boots. To make the SD card standalone bootable again, its cmdline.boot can be moved back to cmdline.txt.
Update 2: Nothing is easy - dealing with errors when attempting to boot via SSD
One of the items I ran into shortly after moving over to booting via SSD, was the fact that I was seeing errors like this when booting up (or running dmesg). I believe this issue is specific to Raspberry Pi 4 devices because of UAS, but don’t quote me on that:
[ 573.203294] sd 6:0:0:0: [sda] tag#29 uas_eh_abort_handler 0 uas-tag 9 inflight: CMD OUT
[ 573.203302] sd 6:0:0:0: [sda] tag#29 CDB: Write(10) 2a 00 00 4f a0 00 00 04 00 00
[ 573.205063] sd 6:0:0:0: [sda] tag#28 uas_eh_abort_handler 0 uas-tag 10 inflight: CMD OUT
[ 573.205070] sd 6:0:0:0: [sda] tag#28 CDB: Write(10) 2a 00 00 4f a4 00 00 04 00 00
[ 573.208537] sd 6:0:0:0: [sda] tag#27 uas_eh_abort_handler 0 uas-tag 6 inflight: CMD OUT
This is a sign that you need to apply some additional boot options to your system in order to support the USB host controller of your SSD drive. To do this, follow the instructions here:
Locate the idVendor and idProduct of the USB controller via dmesg, update /boot/cmdline.txt to add these parameters to the beginning of the line, add the text usb-storage.quirks=aaaa:bbbb:u where aaaa is the idVendor for your device and bbbb is the idProduct. So, with the device above the string will be usb-storage.quirks=2109:0715:u.
You may also need to adjust your voltage settings as per this guide by adding over_voltage=1 to /boot/config.txt. This will make sure your USB device is receiving the most voltage you can send it. If you still have issues, you may need a better/bigger power supply.