Let’s say you installed proxmox on a dedicated server to which you have physical access, but now you need to replace the boot disk. If you use a similar medium (replacing a SSD with a SSD or a usb key with a usb key) with a larger or equal size, then you can simply copy the full disk using dd for instance.

But what if you installed proxmox using LVM, and want to switch to a raid install ? Or you need to change some other parameter of the install, such as enabling encryption, or changing the size of the /boot partition ? Then you need to perform a fresh install, and setup the server again. If you’re just importing a few VMs then this could be trivial, but what if you have a lot of scripts and configuration on the server ? Then you need to backup the configuration, run a fresh install, then it back. This it what we did for our server.

This procedure can be adapted to many scenarios, for instance :

• Replacing a failing boot drive, when dd throws too many errors but you can still access the filesystem.
• Changing install parameters such as partitions, switching to and from ZFS, or between ZFS modes.
• Repairing a proxmox install where the package management got broken, or some important system files were deleted by mistake.

However, you may expect more complications if :

• You are changing the hardware configuration of the server as the same time (changing data drives, changing the server, the networking).
• The old boot medium is so degraded that the system won’t boot at all.
• The proxmox server is part of a cluster.

In our case, we had a NAS server with a ZFS pool of 6 HDDs, and a single USB key as a boot drive, set up using LVM. The USB drive started throwing errors so we wanted to replace it with two USB keys in a ZFS RAID1 configuration (mirroring) to allow easy replacement and better error detection.

Backup the main boot partition

The first, as always is to take an image of the failing drive, and check that all the required data is accessible from the image. This can be done directly from the running server, but if possible it’s better to do it offline by either moving the drive to a computer or booting from a live USB drive.

Since the original system is still running, you can save a bit of time by dumping the list of installed packages using dpkg -i and making a copy of the /etc/pve folder (more on that later). You should also note the version of proxmox used, since we will want to reinstall the same version.

Then check that you can open the image and access the files. ZFS and LVM add extra complexity here because they don’t usually allow pools or volumes with duplicate names, so it may be easier to open these images on systems that are using a different technology (for instance, mounting a zfs pool in a computer that uses LVM for boot).

Another strategy for LVM is to image the logical volume directly (or image the partition dev/sdaX for a more typical ext4 install for instance) instead of the physical drive /dev/sda.

Here a few links on how to mount various image types:

• Mounting an ext partition from a full drive image.
• Mounting a partition from a partition image is trivial : mount -o ro partition.iso /mnt
• Mounting a ZFS pool from a disk image is more involved, especially if another pool with the same name already exists on the local system. (archive link)
• The process for mounting a LVM volume is similar, but collisions can be a bigger issue.

Reinstall proxmox

Then you can reinstall proxmox using a USB installation drive. Make sure the original drive is unplugged if it is not needed for the installation, since it will provide an extra copy of the data just in case, and it avoids booting on the wrong medium.

Make sure to use the same version of proxmox as the original.

The next step is to boot into the new system, and restore the configuration.

Restore the configuration

The crux of this procedure is that proxmox (and debian in general) is extremely forgiving when it comes to overwriting its configuration, to the point where you can even do it while the system is running.

Backup the new configuration files (mostly /etc/fstab)

What we did is plug in the old boot drive, mount it read-only to /mnt (so in the end we did not use the image at all in our case), and :

cp -afv /mnt/{root,home,etc} /
reboot

One caveat is that /etc/pve is mounted using a fuse device, it actually is a sqlite database located in /var/lib/pve-cluster.

systemctl stop pve-cluster
cp -afv /mnt/var/lib/pve-cluster /var/lib/
systemctl start pve-cluster

This will restore all the cluster configuration such as storage, VMs, …

Here are the things you may want to check manually:

• Previously-installed packages will need to be reinstalled
• If you changed the storage layout, you may need to tweak /etc/fstab
• Crontabs are stored in /var/spool/cron/crontabs

Otherwise you can simply reboot and the system will be fully restored! In particular our main storage pool was imported without issues, and the ssh host key did not change which avoided a lot of painful maintenance. Usernames and passwords were also migrated since they are all stored in /etc. File permissions were also all correct since we copied both the user ids which are stored in /etc/passwd and files with the -a flag of cp which preserves file permissions.

One last thing we had to do when switching to a ZFS-based boot pool is to add rootdelay=10 to GRUB_CMDLINE_LINUX_DEFAULT and update-grub to fix an issue during boot.

---

Check out our other articles if you want to know how we setup our proxmox infrastructure on bare-metal servers or how we used ZFS send/recv to automate our VM backups.