Guillaume Matheron

Data scientist, PhD in computer science

Reinstall proxmox while keeping all the configuration and VMs intact

Let’s say you installed proxmox on a dedicated server to which you have physical access, but now you need to replace the boot disk. If you use a similar medium (replacing a SSD with a SSD or a usb key with a usb key) with a larger or equal size, then you can simply copy the full disk using dd for instance.

But what if you installed proxmox using LVM, and want to switch to a raid install ? Or you need to change some other parameter of the install, such as enabling encryption, or changing the size of the /boot partition ? Then you need to perform a fresh install, and setup the server again. If you’re just importing a few VMs then this could be trivial, but what if you have a lot of scripts and configuration on the server ? Then you need to backup the configuration, run a fresh install, then it back. This it what we did for our server.

This procedure can be adapted to many scenarios, for instance :

  • Replacing a failing boot drive, when dd throws too many errors but you can still access the filesystem.
  • Changing install parameters such as partitions, switching to and from ZFS, or between ZFS modes.
  • Repairing a proxmox install where the package management got broken, or some important system files were deleted by mistake.

However, you may expect more complications if :

  • You are changing the hardware configuration of the server as the same time (changing data drives, changing the server, the networking).
  • The old boot medium is so degraded that the system won’t boot at all.
  • The proxmox server is part of a cluster.

In our case, we had a NAS server with a ZFS pool of 6 HDDs, and a single USB key as a boot drive, set up using LVM. The USB drive started throwing errors so we wanted to replace it with two USB keys in a ZFS RAID1 configuration (mirroring) to allow easy replacement and better error detection.

Backup the main boot partition

The first, as always is to take an image of the failing drive, and check that all the required data is accessible from the image. This can be done directly from the running server, but if possible it’s better to do it offline by either moving the drive to a computer or booting from a live USB drive.

Since the original system is still running, you can save a bit of time by dumping the list of installed packages using dpkg -i and making a copy of the /etc/pve folder (more on that later). You should also note the version of proxmox used, since we will want to reinstall the same version.

Then check that you can open the image and access the files. ZFS and LVM add extra complexity here because they don’t usually allow pools or volumes with duplicate names, so it may be easier to open these images on systems that are using a different technology (for instance, mounting a zfs pool in a computer that uses LVM for boot).

Another strategy for LVM is to image the logical volume directly (or image the partition dev/sdaX for a more typical ext4 install for instance) instead of the physical drive /dev/sda.

Here a few links on how to mount various image types:

Reinstall proxmox

Then you can reinstall proxmox using a USB installation drive. Make sure the original drive is unplugged if it is not needed for the installation, since it will provide an extra copy of the data just in case, and it avoids booting on the wrong medium.

Make sure to use the same version of proxmox as the original.

The next step is to boot into the new system, and restore the configuration.

Restore the configuration

The crux of this procedure is that proxmox (and debian in general) is extremely forgiving when it comes to overwriting its configuration, to the point where you can even do it while the system is running.

Backup the new configuration files (mostly /etc/fstab)

What we did is plug in the old boot drive, mount it read-only to /mnt (so in the end we did not use the image at all in our case), and :

cp -afv /mnt/{root,home,etc} /

One caveat is that /etc/pve is mounted using a fuse device, it actually is a sqlite database located in /var/lib/pve-cluster.

systemctl stop pve-cluster
cp -afv /mnt/var/lib/pve-cluster /var/lib/
systemctl start pve-cluster

This will restore all the cluster configuration such as storage, VMs, …

Here are the things you may want to check manually:

  • Previously-installed packages will need to be reinstalled
  • If you changed the storage layout, you may need to tweak /etc/fstab
  • Crontabs are stored in /var/spool/cron/crontabs

Otherwise you can simply reboot and the system will be fully restored! In particular our main storage pool was imported without issues, and the ssh host key did not change which avoided a lot of painful maintenance. Usernames and passwords were also migrated since they are all stored in /etc. File permissions were also all correct since we copied both the user ids which are stored in /etc/passwd and files with the -a flag of cp which preserves file permissions.

One last thing we had to do when switching to a ZFS-based boot pool is to add rootdelay=10 to GRUB_CMDLINE_LINUX_DEFAULT and update-grub to fix an issue during boot.

Check out our other articles if you want to know how we setup our proxmox infrastructure on bare-metal servers or how we used ZFS send/recv to automate our VM backups.

2 responses to “Reinstall proxmox while keeping all the configuration and VMs intact”

  1. Hi Guillaume,

    I’m running Proxmox latest version on single lvm disk ext4 FS, can you provide some guidance how to backup either boot disk or a way to copy important files/directories so when the disc fails I can install same ver of proxmox and restore the files.
    I just started using Proxmox so I’m still learning. Currently, I have installed it on a single SATA SSD which I have cloned to another same size SSD disk using the dual sata docking station which has a hardware clone feature and it works great. Due to limited ports on my system I prefer to use the NVME M.2 2230 disk for boot so if you can provide some guidance.

    • Hi! My first question would be whether you store the VM drives on this same disk? If yes, do you want to also backup the VMs?

      If it’s only the proxmox config you’re worried about, you can just write down the version you use, and backup `/mnt/{root,home,etc,pve}` with `cp -a` to a (USB?) external drive. Make sure the drive has an ext4 filesystem so you don’t lose the permissions. This will be much faster than creating a disk image.

      VMs require a bit more finesse to backup while running, you could look at proxmox-backup-server or

      Also remember that RAID is not a backup!

Leave a Reply

Your email address will not be published. Required fields are marked *