LVM device-mapper: reload ioctl failed: Invalid argument

LVM2 (Logical Volume Management) is pretty amazing, but when something goes wrong, it’s not easy to troubleshoot. This is not the fault of the tools, but a reflection that LVM is relatively new in Linux, and not widely understood.

What I Tried to Do

I tried to increase the size of a logical volume with the lvextend command:

lvextend --extents 100%FREE /dev/VolumeGroup1/var

This form of the command is supposed to use all of the free space in the volume group.

What Failed

The command responded with an error message that contained this text:

device-mapper: reload ioctl failed: Invalid argument

Running lvdisplay showed the following status for the volume:

LV Status    suspended


LVM could not activate the volume with the vgchange command:

vgchange -ay

Attempts to run fsck.ext4 resulted in an error message about a bad superblock. The volume also became unresponsive, and the system would not run certain commands (even though the volume was not mounted, the device mapper process was probably corrupted). I also saw kernel messages like this in the system log:

kernel: device-mapper: device 8:16 too small for target

How I Undid the Damage

When you have an LVM problem, don’t panic. There is built-in undo/revert/restore feature in LVM2! Every time you change something, a file is automatically saved to /etc/lvm/archive (at least on an Ubuntu 14 system). I have NO IDEA why this feature is not more widely known! The command is vgcfgrestore:

vgcfgrestore --file /etc/lvm/archive/name_of_archive_file VolumeGroupName

NOTE: if you are running LVM commands when your system is booted from a recovery CD, the undo files are saved to the /etc directory on the RAM disk, which is wiped out when the system is rebooted. Be sure to copy all files from /etc/lvm/archive to another drive, such as a USB drive, so they will be available if you have to reboot.

How I Resized the Volume

Rodrigo Tamada figured out why lvextend sometimes fails when it attempts to automatically determine the amount of free space on the volume. The problem is that sometimes LVM does not correctly calculate the amount of free space on the physical volume. LVM then extends the logical volume beyond the extents of the physical volume. I followed the procedure suggested by Rodrigo, and it worked. First, look at the output of the lvdisplay command to find the name of the physical volume that the logical volume resides on. Then, run the pvresize command, no size specified, to change the physical volume size to match the size of the device:

pvresize /dev/mapper/your_volume_name

Run pvdisplay to verify that the physical volume size has changed. Now, you should be able to extend a volume to use all the available space.

5 thoughts on “LVM device-mapper: reload ioctl failed: Invalid argument

  1. Luis

    vgcfgrestore saved my ass from a HUGE problem.. thanks!!! But I have 3 servers using the same PV (it’s a NAS) and I need to specify how much the LV should grow on each server. I can’t just let it take all available space. Any ideas?

    Reply

Leave a Reply