one canceled VPS was terminated from this node yesterday and OpenVZ tried to delete data of that VPS from its /vz/private as well as /vz/tmp
It managed to move its data from /vz/private to /vz/tmp but "rm -rf" command started giving problems while openvz started deleting data from /vz/tmp. This command had maxed out CPU load and 100% resources were assigned to this command. no other command could be executed on the node as rm -rf had taken 100% resources. Soft reboot didnt work as server could not assign any resources to reboot command as well.
We had to make a decision to power cycle server to get rid of this "rm -rf" command but after first power cycle same command started once again. I managed to move rm command to get rid of it after next reboot but on next power cycle server didnt come online. Console was giving errors with /vz partition so /vz was removed from /etc/fstab and then the server came online. We had to then mount /vz again and server was rebooted ( soft reboot ) but this time when it came online there was no data in /etc
once data from /etc is lost then there's nothing much left on the server to be done and we had to start OS reload process on other drive. This server has got capacity for 2 drives only and as we had RAID1 configured so first we had to remove RAID configuration then one drive was used as secondary and a fresh drive was used as primary.
RAID is once again being built now and hopefully the node will come online in next couple of hours.
__________________
UK Web Hosting || Business Hosting || eUKhost Knowledgebase
Toll Free : 0808 262 0255 || MSN : mark @ eukhost.com || AIM : eukmark
A bunch of Sheep led by a Lion is better than a bunch of Lions led by a Sheep.
__________________________________________________
Great Opportunity :: Join our Affiliate Program for FREE and earn 20% commission on each referral.
|