Cloud Performance Observations

[ejsolutions]

Some things that I've noticed may be of interest.

Cloud 1: vmware - euKHost
Since the recent outage of Cluster #3, I've at long last noticed a return to the original great levels of performance. Following the major disruption a couple of months back, with storage array(s), the performance dropped considerably and I mentioned this at the time. A few tweaks were done by support but it was never the same as before. I can happily report that things are back to normal.
I/O wait is now averaging about 5% again, with a resultant drop in CPU usage and overall load.

Cloud 2: zenenterprisepv - customer resizing (like eNlight) competitor supplier
Small Cloud with light load - displays weird rise and fall in applications memory usage. Apart from lousy slow response to support issues, generally it suits the purpose. It is a responsive system.
I/O wait averages about 5%, CPU low, Load low.

Cloud 3: zenpv - eNlight - eukHost
Small Cloud (same resources as Cloud 2) with a few domains migrated from Cloud 2. Lightly loaded in terms of traffic. Takes what an appears to be an age (3 to 8 secs. approx.) to "wake up" when a site request is made, then runs OK.
I/O wait averages about 10% and never drops below approx. 8%, CPU is relatively high and Load climbs at little provocation (just like Cloud 1 was).

Cloud 4: virtuozzo - USA provider
Big traffic, so unfair to make any comparisons, other than I prefer vmware and its logical use of vm/swap space.

Has eNlight been crippled by a slower storage medium compared to vmware Cloud, is the obvious question? Wouldn't be surprising but the knock on effect of possibly increasing the monthly running cost, due to Load is of concern, at this stage.

EJ

[DPS Computing]

Thank you for your detailed feedback . I am sure that this will be as useful to other customers as it is for me .

I'm sure someone from eUK will be along shortly to answer your questions .

[John]

Some things that I've noticed may be of interest.

Cloud 1: vmware - euKHost
Since the recent outage of Cluster #3, I've at long last noticed a return to the original great levels of performance. Following the major disruption a couple of months back, with storage array(s), the performance dropped considerably and I mentioned this at the time. A few tweaks were done by support but it was never the same as before. I can happily report that things are back to normal.
I/O wait is now averaging about 5% again, with a resultant drop in CPU usage and overall load.

Cloud 2: zenenterprisepv - customer resizing (like eNlight) competitor supplier
Small Cloud with light load - displays weird rise and fall in applications memory usage. Apart from lousy slow response to support issues, generally it suits the purpose. It is a responsive system.
I/O wait averages about 5%, CPU low, Load low.

Cloud 3: zenpv - eNlight - eukHost
Small Cloud (same resources as Cloud 2) with a few domains migrated from Cloud 2. Lightly loaded in terms of traffic. Takes what an appears to be an age (3 to 8 secs. approx.) to "wake up" when a site request is made, then runs OK.
I/O wait averages about 10% and never drops below approx. 8%, CPU is relatively high and Load climbs at little provocation (just like Cloud 1 was).

Cloud 4: virtuozzo - USA provider
Big traffic, so unfair to make any comparisons, other than I prefer vmware and its logical use of vm/swap space.

Has eNlight been crippled by a slower storage medium compared to vmware Cloud, is the obvious question? Wouldn't be surprising but the knock on effect of possibly increasing the monthly running cost, due to Load is of concern, at this stage.

EJ

Hello,

After the slow down last week, we moved quite a few VMs to a new storage platform to restore full service while faulty components were replaced. The problem last week was caused by faulty cables going to a controller, which meant the other controller was taking all the load. After we moved VMs to get the load "normal", the faulty cables were identified and replaced as well. This brought the other controller back online, which meant the VMs still hosted on it had the use of two controllers and had considerably less VMs on. Your VM wasn't moved, so it is benefiting from a considerably less busy storage device.

In regards to eNlight, it is on our latest storage platform. Looking at the stats, it isn't remotely close to hitting any limit. On your VMware Cloud you have 1024 MB RAM. On your eNlight it you have limited it to 512 MB, and it is also set in economy mode. These things combined and perhaps differences in technologies would likely explain your performance difference. On the eNlight platform, as with our latest VMware and HyperV clusters, faster switches and more connections are used compared to our older Clusters.

FYI, the storage system your VMware VM is on will soon be decomissioned, as the hardware is going end of life. You will be migrated (no downtime) to a new system over the weekend or early next week. After which you can compare performance again. I do like these types of threads

[ejsolutions]

Fantastic reply, John.

Coming from both a hardware and software background, I can appreciate the difficulty in diagnosing, let alone isolating, a faulty cable! I hope someone got at least a 'pat on the back'.

I have realised that I can't compare "my" two eUKhost Clouds as like-for-like, due to a number of differences, hence the inclusion of 'Cloud #2'. For me, it's very rare to be able to compare four different flavours of Cloud (as a consumer). The zenenterprisepv VM, as mentioned, is nigh on directly comparable to eNlight, running within 512Mb on the same release of Centos/cPanel. Its resources are capped, with no burst, and so I assume is similar to running eNlight in 'economy mode'. The back-end control panel is different, so there's likely to be hidden differences though.

It leads me to wonder if there is a fundamental difference between zenenterprisepv and zenpv, or whether there is much more to this than meets the eye. I doubt that the zenenterprisepv VM has nearly as good (poor word) an infrastructure to run from as eNlight: it is after all a cheap unmanaged service. This is what makes the 5% difference in I/O wait surprising - maybe I'm just fortunate to be on a quiet cluster on 'Cloud #2'.

If further interested/appropriate, I can attach a couple of Munin charts to illustrate the discussion.

EJ

[ejsolutions]

FYI, the storage system your VMware VM is on will soon be decomissioned, as the hardware is going end of life. You will be migrated (no downtime) to a new system over the weekend or early next week.

Thanks for the 'heads up' - much appreciated. One hopes that the cPanel backup window doesn't produce Loads over 8 again.

[John]

Hello,

Your performance will only get better, of that I'm sure

In regards to eNlight, you might want to reboot it into Performance Mode if you are limiting it to 512 MB RAM anyway. That way there is no cache manipulation, which could impact I/O in certain situations. Also might be worth comparing OS versions (CentOS Version) and if one is 64 bit or 32 bit.

[eUKhost.com]

Hi Jon

I would like to get some of our R&D team members involved to see if we can make some changes on our side to give similar performance in economy mode and high performance mode.

We can make some changes in the specification of your Cloud VM and see if you experience improved performance.

[John]

Hello,

Rishi is following the thread Changing it would defeat the purpose, but in this case as scaling isn't being used anyway, it makes no sense to have it in Economy Mode. Economy Mode keeps scaling of RAM under control with a small sacrifice in performance in certain situations, where as Performance mode lets the OS do as it pleases, giving an obvious boost. In the current setup there is no benefit and only downsides to keeping a limit of 512 MB RAM and then also setting it in economy.

[ejsolutions]

As suggested, I've switched to "performance mode".

Below are the differences between the two Cloud VMs, noting the allocation of buffers to be substantially different. Interesting artefact in the way memory is report, though note the actual allocation is the same. eNlight should have the edge in raw performance due to better spec. CPU but the loading on both VMs is so low as to be of negligible significance.

[Domain names obscured but you guys will likely guess/know ]

Total processors: 1
    Intel(R) Xeon(R) CPU X3440 @ 2.53GHz
    2533.330 MHz
    Cache 8192 KB

Memory: 501248k/524288k available (2530k kernel code, 22404k reserved, 1736k data, 196k init)
Linux secure-1.cloudxxxx.com 2.6.18-238.19.1.el5xen #1 SMP Fri Jul 15 08:16:59 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

Current Memory Usage
             total       used       free     shared    buffers     cached
Mem:        524288     521180       3108          0       2496     108108
-/+ buffers/cache:     410576     113712
Swap:       522104      75928     446176
Total:     1046392     597108     449284

Total processors: 1
   Intel(R) Xeon(R) CPU X5660 @ 2.80GHz
   2792.998 MHz
   Cache 12288 KB

Memory: 187896k/16777216k available (2535k kernel code, 335844k reserved, 1748k data, 196k init)
Linux secure-1.xxxxcloud.net 2.6.18-274.12.1.el5xen #1 SMP Tue Nov 29 14:18:21 EST 2011 x86_64 x86_64 x86_64 GNU/Linux

Current Memory Usage
             total       used       free     shared    buffers     cached
Mem:        524288     468972      55316          0        240      18336
-/+ buffers/cache:     450396      73892
Swap:      2129912     277912    1852000
Total:     2654200     746884    1907316

(In my experience, in non-vm environments, a swap of more than twice RAM can be detrimental. In modern times, I usually restrict to match RAM.)

[ejsolutions]

A couple of pics added, 'cos only words can get boring.

[ejsolutions]

For other (potential) customers of eUKhost, this is the level of interest shown by the staff, that sets them apart from the competition. Yes, I have my "moments" with them but overall you're going to be hard pressed to find a better hosting provider. As for USA-based ones: they don't even come close!

[eUKhost.com]

For other (potential) customers of eUKhost, this is the level of interest shown by the staff, that sets them apart from the competition. Yes, I have my "moments" with them but overall you're going to be hard pressed to find a better hosting provider. As for USA-based ones: they don't even come close!

Thanks Jon

Our eNlight team is reading this thread. I am waiting for them to respond here and make the thread more interesting.

[ejsolutions]

A bit slow off the mark with this one..

Rishi is following the thread ...

Is this the Guru that lives under the suspended floor of the machine room, wearing an antistatic wrist strap and tin foil hat?


Meanwhile, back to comparing PHP code. :'(

[John]

A bit slow off the mark with this one..


Is this the Guru that lives under the suspended floor of the machine room, wearing an antistatic wrist strap and tin foil hat?

He is one of the main developers of eNlight, so is best placed to advise on performance differences

[eUK-Rishi]

Hi ejsolutions,

Thanks for starting a wonderful thread and we expect more to come.

Theoretically there are multiple factors on performance scale, in sorted order
* SAN vs Local Disk
* Kernel
* CPU family
* Size of RAM and cache pressure
* LVM vs non LVM
* FileSystem ext3 vs ext4
* OS type 64 vs 32

eNlight Cloud runs same storage platform as vmware so the difference is on the Hypervisors and VM OS.

From your reply the factors which I see is

* Both have 64bit OS
* Kernel 2.6.18
* But a big Buffer and Cache difference, buffer and cache are yet to fill in enlight VM which will reduce your iowait.

You can do

sync ; echo 3 > /proc/sys/vm/drop_caches

on both the server and wait for 1 day to see the graphs.
Above command will drop cache and buffer to disk and will put both VMs on standard disk load.

I would like to have results of this. If you can PM me your enlight server details, I can take a deeper look into it. Also it depends on the app of both server