What to do when /vz partition goes read-only

[firew@ll]

Note : Be Careful while using these steps.

A few times now we have seen problems on the Linux virtuozzo servers where the /vz partion errors out and the kernel forces it to go into a read-only state. The solution is to manually run a filesystem check (FSCK) with the -fy arguments on the /vz partition. It is possible on a vz server to do this without a support ticket at Gnax however you must be careful with a few steps.

1. Stop the vz service (service vz stop)

2. Disable the vz service from restarting on boot (chkconfig --level 2345 vz off)

3. Comment out the /vz partition inside the /etc/fstab file to prevent the /vz partition being automaticlyt checked on boot.

4. Reboot the server

5. Once the server reboots and you can SSH back in you can begin the filesystem check.

6. Determine which physical partition is /vz/ You can do this with the command (fdisk -l /dev/sda). This will show you all the partitions.

Usually the /vz partition is the last one listed and will have the larges amount of blocks with a "ID" of 83 and "System" label of Linux:

EXAMPLE
# fdisk -l /dev/sda

Disk /dev/sda: 299.9 GB, 299978719232 bytes
255 heads, 63 sectors/track, 36470 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 1305 10482381 83 Linux
/dev/sda2 1306 2610 10482412+ 83 Linux
/dev/sda3 2611 3915 10482412+ 83 Linux
/dev/sda4 3916 36470 261498037+ 5 Extended
/dev/sda5 3916 4437 4192933+ 82 Linux swap
/dev/sda6 4438 4568 1052226 83 Linux
/dev/sda7 4569 36470 256252783+ 83 Linux

In the above example you can see that /dev/sda7 have the most blocks and has the "ID" of 83 and "System" label of Linux:.

Therefore we would issue the command (fsck -fy /dev/sda7).

7. Once the file system check is complete you can uncomment the /vz parition in /etc/fstab

8. re-enable the vz service to start on boot (chkconfig --level 2345 vz on)

9. start up the vz service (service vz start)

All should then be well. You can monitor the logfile /var/log/vzctl.log as VE's are brought online.

If you are uncomfortable doing any of this then please contact a senior tech or management and we can help out. Please remember it is very dangerous to FSCK a mounted partition to always ensure the target partition os unmounted before beginning. Generally you cannot umout a /vz partition after stopping the VZ service so I recommend the reboot with /vz partition disabled and the vz service disabled.

-Firew@ll