Cloud Performance Observations

[ejsolutions]

Houston we have a problem: I can scale back the CPU to 2, from 24, but not 1.
"Error : Parameter Value Error."

[eUK-Rishi]

Your reading of those stats is obviously different from mine. What I see is a vindication of my suggestion that swap is set too large and a part consequence is that buffers have been poorly allocated. This is having a detrimental effect on disc I/O. Reduce the swap size and see if some swap gets used along with RAM being allocated to disc buffers. I (notionally) bet the I/O wait drops.

Aye aye captain !! The swap reduce is underway. This should decrease iowait. We will be making 512M but users are free to modify thereafter. Also checking if online resize is harmful and it seems it is because swap topped by LVM
I was checking another guys %iowait and it was 60-80% , checked the dstat and it was 50-70M read per sec and again RAM to 512M :sigh: .

Sorted the other error Please give a try !

[ejsolutions]

Aye aye captain !!

I've been called Scottie but never Captain before!

Sorted the other error Please give a try !

That was quick! Yep, back to 1, thanks.

[ejsolutions]

A change can't come quick enough:
load average: 11.64, 22.92, 16.22

[eUK-Rishi]

Hello,

Please follow below instructions to decrease the SWAP or you may create a ticket to our support to do it for you

First you will need to turn off the swap to reduce it and avoid any data corruption

# swapoff -a

It may take a while for 512M RAM Server as the page flush will be slow on it.

To confirm the swap is off

# free -m
             total       used       free     shared    buffers     cached
Mem:          1024        932         91          0          9        352
-/+ buffers/cache:        570        453
Swap:            0          0          0

Now, to reduce the swap LVM partition

# lvreduce -L 512M /dev/VolGroup00/LogVol01
  WARNING: Reducing active logical volume to 512.00 MB
  THIS MAY DESTROY YOUR DATA (filesystem etc.)
Do you really want to reduce LogVol01? [y/n]: y
  Reducing logical volume LogVol01 to 512.00 MB
  Logical volume LogVol01 successfully resized

Format the reduced volume with swapfs

# mkswap /dev/VolGroup00/LogVol01

Turn on the swap and verify

# swapon -a
# free -m
             total       used       free     shared    buffers     cached
Mem:          1024        933         90          0          9        353
-/+ buffers/cache:        570        453
Swap:          511          0        511

Lets see if this lessens your %iowait.

[ejsolutions]

I think I'm going to have to kill my VM before being able to implement.

load average: 70.96, 63.54, 48.26

!!!
Now why didn't I think of doing the lvreduce?

[ejsolutions]

I'm fighting a losing battle at the moment, to bring load down to usable levels, by stopping/killing processes.

load average: 47.59, 29.19, 24.51

At this rate, I think I'm looking at a VM rebuild
I have noticed a few things, including clamd hanging during startup - I'm sure exim should be started much later during init. The iSCSI daemon appeared to hang for some time, finally reporting that no targets could be found - is this actually used for eNlight?

Also, your HyperV Client server was hammering iptables - added it to csf.allow and csf.ignore.
I'm getting a series of cphulkd processes running and wonder if I'm getting a DOS attack on the server.

More later...

[eUK-Rishi]

I have noticed a few things, including clamd hanging during startup - I'm sure exim should be started much later during init. The iSCSI daemon appeared to hang for some time, finally reporting that no targets could be found - is this actually used for eNlight?

iSCSI is not used by eNlight. You can safely disable it.
Have you reduced the swap or still facing the load ?

[ejsolutions]

...Have you reduced the swap or still facing the load ?

After what seems to me like a VERY long day already, I have managed to disable sufficient processes to flush the caches, enough to do a swapoff. Hurrah!
Now set to 512Mb but still have underlying issues - at this moment I suspect it's clamd causing all the issues, so time to reinstall it, as it is affecting exim.

top - 15:51:43 up 48 min,  1 user,  load average: 4.73, 8.17, 12.22
Tasks:  88 total,   2 running,  86 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.4%sy,  0.0%ni, 88.0%id, 11.2%wa,  0.0%hi,  0.0%si,  0.4%st
Mem:    524288k total,   468620k used,    55668k free,     1872k buffers
Swap:   524280k total,   281736k used,   242544k free,    26380k cached

I daren't come out of 'top' - might never get back in.
CSF is running overtime, blocking stuff.

[eUKhost.com]

Hi Jon,

I hope its all going in right direction. You guys made me read for 10 minutes, but i am still trying to find out if everything's alright.

[ejsolutions]

Hi Jon,

I hope its all going in right direction. You guys made me read for 10 minutes, but i am still trying to find out if everything's alright.

Executive Summary
----------------------
In a word, no. Work in progress.

[eUKhost.com]

Executive Summary
----------------------
In a word, no. Work in progress.

yeah

I had a word with the Technical team directly and managed to learn few things. sometimes talking is better than reading.

[danyboyj]

I was checking another guys %iowait and it was 60-80% , checked the dstat and it was 50-70M read per sec and again RAM to 512M :sigh:

I'm guessing that was my VM - I'll admit I'm quite inexperienced when it comes to Linux - I'm a developer not a sysadmin. But if 512Mb is too little memory for a VM - why allow that setting in the control panel? Anyways.... it's at 2Gb now.

Sorry to thread hijack but - I'm wondering if my VM issues are related to any work that might be going on? I run a small site and when it's running - it runs flawlessly. But I go to bed - wake up next morning and try to access it - it's dead (or asleep)

HTTP / SSH / FTP is all inaccessible - then a few minutes later it will be responsive - then die again. All the while load is near 0 and uptime reports its been running for hours.

[John]

I'm guessing that was my VM - I'll admit I'm quite inexperienced when it comes to Linux - I'm a developer not a sysadmin. But if 512Mb is too little memory for a VM - why allow that setting in the control panel? Anyways.... it's at 2Gb now.

Sorry to thread hijack but - I'm wondering if my VM issues are related to any work that might be going on? I run a small site and when it's running - it runs flawlessly. But I go to bed - wake up next morning and try to access it - it's dead (or asleep)

HTTP / SSH / FTP is all inaccessible - then a few minutes later it will be responsive - then die again. All the while load is near 0 and uptime reports its been running for hours.

Hi Danyboyj,

If load is 0, then no, it isn't an issue with infrastructure. Time to pop a ticket into tech support to have them investigate deeper for you.

[danyboyj]

Yup - there's a ticket in the system