2 of 3

Recovering File Systems and LVMLesson 2 of 3

Recovering LVM: labels, metadata and missing disks

Restore a lost PV label and VG metadata from backup, handle a missing disk, and fix devices-file problems.

  • pvs
  • vgs
  • lvs
  • pvs -o +pv_uuid
  • vgcfgbackup
  • vgcfgrestore
  • vgcfgrestore --list
  • pvcreate --uuid --restorefile
  • vgchange -ay
  • vgreduce --removemissing
  • lvmdevices
  • pvscan
  • pvmove
  • lvm vgck

55 min3 exercises

After this lesson you can

  • Explain where LVM keeps its metadata and what /etc/lvm/backup and /etc/lvm/archive contain.
  • Restore a physical volume label and a volume group from a metadata backup, using the right UUID.
  • Respond to a missing physical volume without making the damage worse, and explain the devices file.

The idea

LVM stores the description of its own layout on the disks themselves. Each physical volume carries a label near the start (so tools can recognise it), and a metadata area that holds a text description of the whole volume group: its PVs, LVs, extent maps and UUIDs. LVM also copies that text to /etc/lvm/backup/VGNAME after every change and keeps older versions in /etc/lvm/archive/.

That arrangement allows two kinds of recovery:

  1. The data is intact but the label or metadata was overwritten. Rebuild them from the text backup.
  2. A whole disk is gone. Bring up what can be brought up, then take the dead disk out of the description.

Everything in LVM is identified by UUIDs, not by device names. A restore therefore must give the new label the same UUID that the metadata expects.

Where things are

ItemLocation
PV label and metadata areaFirst sectors of each PV
Latest metadata text/etc/lvm/backup/VGNAME (update after each LVM command)
Older versions/etc/lvm/archive/VGNAME_NNNNN-*.vg
Configuration/etc/lvm/lvm.conf
Devices file (RHEL 9 and later)/etc/lvm/devices/system.devices

Reading the damage

pvs -o +pv_uuid       # the PVs LVM sees, with UUIDs
vgs                   # VG attribute column: 'p' = partial (a PV is missing)
lvs -a -o +devices    # which PV each LV uses
vgcfgrestore --list VG   # metadata backups with times and descriptions
pvscan; vgscan        # rescan if a device was just attached

A missing PV shows as [unknown] in pvs, plus a warning Couldn't find device with uuid .... The warning gives the UUID you will need.

Try it

These are real outputs from LVM on a loop device with a freshly created volume group vgt. (Two sandbox-specific lines about libdevmapper and kernel driver version were removed.)

The label is lost

The first two sectors of the PV were zeroed, as if overwritten:

pvs; vgs
(no output: LVM no longer sees the disk)

First attempt, the wrong UUID. The top id = in the backup file is the VG’s UUID:

pvcreate --uuid GqMQHd-1PO3-1b3K-sbEd-BlIo-AOYo-vYraFJ --restorefile /etc/lvm/backup/vgt /dev/loop0
  WARNING: Couldn't find device with uuid JDlu5K-brp9-8yp8-Nxhy-Y0oO-4xTz-GuCNOq.
  Can't find uuid GqMQHd-1PO3-1b3K-sbEd-BlIo-AOYo-vYraFJ in backup file /etc/lvm/backup/vgt
  Run `pvcreate --help' for more information.

LVM tells you the right one in the warning: JDlu5K-... is the PV UUID. It is also in the pv0 { id = ... } block of the backup. Second attempt:

pvcreate -ff -y --uuid JDlu5K-brp9-8yp8-Nxhy-Y0oO-4xTz-GuCNOq --restorefile /etc/lvm/backup/vgt /dev/loop0
vgcfgrestore vgt
vgs; pvs
  WARNING: Couldn't find device with uuid JDlu5K-brp9-8yp8-Nxhy-Y0oO-4xTz-GuCNOq.
  Physical volume "/dev/loop0" successfully created.
  Restored volume group vgt.
  VG  #PV #LV #SN Attr   VSize   VFree  
  vgt   1   0   0 wz--n- 196.00m 196.00m
  PV         VG  Fmt  Attr PSize   PFree  
  /dev/loop0 vgt lvm2 a--  196.00m 196.00m

The VG came back with the same name, size and UUIDs. If there had been LVs, they would be back too: the data was never touched, only the label and metadata were rewritten. Mount the LVs and check the file systems.

Order matters: pvcreate --uuid ... --restorefile first (so a PV with that identity exists), then vgcfgrestore. The -ff forces overwriting and is why this is a recovery tool, not an everyday one.

A disk is missing

When a PV has gone (failed, or not attached), vgs shows the VG as partial and pvs shows [unknown]:

vgs
pvs
  WARNING: Couldn't find device with uuid 8cT4Hv-xxxx-xxxx-xxxx-xxxx-xxxx-9aLmQp.
  VG    #PV #LV #SN Attr   VSize   VFree 
  vgapp   2   2   0 wz-pn- <19.99g <9.99g
  PV         VG    Fmt  Attr PSize  PFree
  /dev/vdb   vgapp lvm2 a--  <10.00g 0
  [unknown]  vgapp lvm2 a-m  <10.00g <9.99g

(Representative output.) The p in wz-pn- and the m in a-m mean partial and missing. What you do depends on where the lost extents were.

SituationAction
The disk is attached but not scannedDevices file: lvmdevices --adddev; a rescan: pvscan --cache
Disk temporarily gone, returns laterFix the path, then vgchange -ay normally
LV entirely on healthy PVsvgchange -ay --activationmode degraded VG brings it up
LV partly on the missing PV--activationmode partial activates it with holes; reads of missing parts fail. Copy off what you can
Disk is never coming backlvremove the LVs that used it, then vgreduce --removemissing VG
Replacement disk, same data (RAID/mirror LVs)pvcreate --uuid ... --restorefile, vgcfgrestore, lvconvert --repair

Going further

Backups are the recovery. The text backups in /etc/lvm/backup live on the same machine as the disks they describe. A VG whose root LV has failed took its own backup with it. Copy /etc/lvm off the host regularly, or rely on the metadata areas on the disks, which vgcfgrestore -f and the older vgcfgbackup -f FILE can write and read as plain files. Beyond the exam: RAID LVs (lvcreate --type raid1), thin pools and snapshots have their own repair commands (lvconvert --repair), and a thin pool that fills up is a separate failure mode; see the monitoring lesson for how to spot it.

Practice

0 of 3 done

Do these on your own system. They get harder. Check each result, then tick it off.

  1. Warm-up

    Map the LVM stack and its backups

    Show every PV with its UUID, every VG with its free space, and every LV with its path and size. Then list the automatic metadata backups for one VG and the date of the newest.

    Check your result
    • sudo pvs -o +pv_uuid prints a UUID column; sudo vgs shows VFree; sudo lvs -o +lv_path shows the paths.
    • ls -l /etc/lvm/backup/ lists a file named after the VG; sudo vgcfgrestore --list VGNAME shows entries with a time.
    Hint

    LVM writes /etc/lvm/backup/VGNAME after every change and keeps older versions in /etc/lvm/archive/. vgcfgrestore --list reads them.

    Solution
    sudo pvs -o +pv_uuid
sudo vgs
sudo lvs -o +lv_path
ls -l /etc/lvm/backup /etc/lvm/archive
sudo vgcfgrestore --list vgdata | head
  2. Practice

    Restore a wiped PV label

    Setup (on a scratch disk or image file, via a loop device /dev/loopX, never real data):

    truncate -s 200M /root/pv1.img; L=$(sudo losetup -f --show /root/pv1.img)
sudo pvcreate $L && sudo vgcreate vgscratch $L
sudo vgcfgbackup vgscratch
sudo dd if=/dev/zero of=$L bs=512 count=2 conv=notrunc

    pvs and vgs no longer show the disk. Restore the label with the original PV UUID, then restore the VG metadata, and prove the VG is back.

    Check your result
    • sudo vgs vgscratch shows 1 PV and the original size.
    • Your command used the UUID from the pv0 { id = ... } block of /etc/lvm/backup/vgscratch, not the id at the top of the file (that is the VG UUID).
    Hint

    pvcreate --uuid PVUUID --restorefile /etc/lvm/backup/VG /dev/loopX writes a new label with the old identity. Only then does vgcfgrestore VG accept the disk. The first id = in the backup file is the volume group, the one inside physical_volumes { pv0 { ... } } is the PV.

    Solution
    sudo grep -A3 'pv0 {' /etc/lvm/backup/vgscratch
sudo pvcreate -ff -y --uuid PV-UUID-FROM-PV0 --restorefile /etc/lvm/backup/vgscratch $L
sudo vgcfgrestore vgscratch
sudo vgs; sudo pvs
  3. Challenge

    A disk disappeared from a volume group

    A volume group vgapp spans two disks, /dev/vdb and /dev/vdc. The data LV lvapp lives on vdb; lvtmp lives on vdc. Disk vdc died. Write the sequence of commands and checks you would use to (1) see exactly what LVM thinks, (2) get lvapp mounted and usable without touching vdc, (3) remove the dead disk's remains from the VG, and (4) prevent the problem repeating. Explain why you must not run vgreduce --removemissing --force first.

    Check your result
    • Evidence first: pvs/vgs show [unknown] or a missing PV and an attribute p (partial) on the VG.
    • Activation with vgchange -ay --activationmode degraded (or partial if lvapp spans the missing disk, accepting that data is lost).
    • vgreduce --removemissing vgapp only after you know which LVs are affected; --force deletes LVs that used the missing disk.
    • Prevention: mirror/RAID LVs (lvconvert --type raid1), monitoring, backups.
    Hint

    Partial activation is a recovery mode, not a repair: it lets the intact LVs come up. Decide what to do with the broken ones (lvremove, restore from backup) only after you have read lvs -a -o +devices.

    Solution
    sudo pvs; sudo vgs; sudo lvs -a -o +devices vgapp
sudo vgchange -ay --activationmode degraded vgapp     # or: partial
sudo mount /dev/vgapp/lvapp /app
sudo lvremove vgapp/lvtmp                             # LV that lived on the dead disk
sudo vgreduce --removemissing vgapp
sudo vgs
# prevention
sudo lvconvert --type raid1 -m1 vgapp/lvapp           # needs free space on another PV