Centos memory allocation with eNlight

[ejsolutions]

In a vain attempt to improve the performance of my low-end VM, I've been trying to reduce the amount of memory swapping and thus disc I/O activity.

CentOS release 5.8 (Final) 32-bit

[email protected] [~]# sysctl vm.swappiness
vm.swappiness = 0

Yet..

[email protected] [~]# free -m
             total       used       free     shared    buffers     cached
Mem:           512        184        327          0          0         14
-/+ buffers/cache:        168        343
Swap:         1023        147        876

Yes, setting the same thing in /etc/sysctl.conf and rebooting does absolutely nothing too. Has anyone else noticed that this directive is being ignored?

To really illustrate that something is amiss..

[email protected] [~]# swapoff -a
swapoff: /dev/mapper/VolGroup00-LogVol01: Cannot allocate memory
[email protected] [~]# swapon -s
Filename				Type		Size	Used	Priority
/dev/mapper/VolGroup00-LogVol01         partition	1048568	157000	-1

[email protected] [~]# sync; echo 3 > /proc/sys/vm/drop_caches
[email protected] [~]# free -m
             total       used       free     shared    buffers     cached
Mem:           512        141        370          0          0          7
-/+ buffers/cache:        133        378
Swap:         1023        195        828

EJ

[ejsolutions]

Re: Centos memory allocation with eNlight

Note: I'm spending too much time on this stuff, so trashing this particular VM. Does anyone other than me run a low-end cPanel eNlight Cloud??

[THOMAS-UK]

Re: Centos memory allocation with eNlight

Setting swapiness to 0 only means that it will avoid swap usage, it doesn't mean that it won't swap. That you can't turn swap off simply means that there is not enough memory available to write back to memory what is in swap right now. As you dropped the caches I don't know how much you had available before but it is always going to be tricky with only 512M RAM.

If you're looking to improve I/O, don't drop caches. Disable the swap partition and reboot. No swap will benefit your server but a memory shortage does mean that it will kill processes and the server may become unstable. The only good answer I can give you is assign more memory to the server, disable SWAP and have a close look every now as to how it's performing.

[ejsolutions]

Re: Centos memory allocation with eNlight

Thanks for your reply, though you've not added anything that I didn't know already. I beg to differ with regards to 512Mb RAM - I have no intention of running with no swap and I have had other servers (Xen) running fine for months with this amount. The key point is that there is ample RAM available and swap is being used excessively. This appears to be specific to the eNlight environment and without knowing the internals, I can only surmise that the hypervisor has something to do with it.

EJ

(Dropping the caches was in a vain attempt to free RAM and was not intended to be viewed as a periodic task - I didn't even infer that. It transpired that part of the swap is made up of the LVM assigned swapfile, though reducing the size of that didn't improve matters either.)

[THOMAS-UK]

Re: Centos memory allocation with eNlight

You said the swapiness setting was ignored, but it's not. I'm not sure if the path you're taking here is the right one too. I assume you have gone over all the relevant Apache and MySQL settings already? Do you know what is causing the swap usage, if it's a particular process? Instead of affecting how it swaps, it's more important why it swaps (and that is not as simple as 'there is not enough memory available). That said, the internals of Enlight are similar to Xen, but comparing systems will always be difficult. The hypervisor itself is not often the problem.

There is a reason why we don't sell 512MB servers any more by the way.

[ejsolutions]

Re: Centos memory allocation with eNlight

On the premise that I'm sick argoung my point and a picture tells a thousand words, here's two...

Note the chunk of unused RAM (pale yellow/green) - this is not Windoze - nearly all RAM should be utilised for something.

Note the possible resulting I/O Wait (dark blue), which isn't good for overall performance nor Load.

My eNlight balance is nearing the bottom, so it may be time to revert to a shared VPS for the few small sites that run on this low-end eNlight. :'( I'll hazard a guess that the site performance will be better - time will tell.

[John]

Re: Centos memory allocation with eNlight

Hello EJ,

A few observations. Your VM is currently set to be in Economy Mode which will artificially play with how much memory your server uses. As you have limited your RAM to 512 MB anyway, I'd recommend you set this to performance mode. Economy mode was designed to stop servers using lots of cache after a peak, basically.

The scaling range you have selected also might be a little too small to properly work. You might want to keep that in mind.

[ejsolutions]

Re: Centos memory allocation with eNlight

As you have limited your RAM to 512 MB anyway, I'd recommend you set this to performance mode. Economy mode was designed to stop servers using lots of cache after a peak, basically.

The scaling range you have selected also might be a little too small to properly work..

Thanks for the mode explanation - now switched to Performance.
Not sure how to address the second point, as 512Mb RAM is the upper limit, to keep costs in check. I've actually raised the minimum from 192 to 256, to monitor the effect.

EJ

[John]

Re: Centos memory allocation with eNlight

Did you reboot after doing it? I believe it's required.

[ejsolutions]

Re: Centos memory allocation with eNlight

Did you reboot after doing it? I believe it's required.

In a word, No. I wasn't aware that was required, though fortunately this isn't a "critical" VM and I should be able to reboot quite readily. Checking server activity now...

[ejsolutions]

Re: Centos memory allocation with eNlight

As a matter of interest, I turned off the swap partition and created a (far smaller) 256Mb swapfile. After instructing fstab to use this instead and a reboot later...

[root /]# free -m
total used free shared buffers cached
Mem: 512 255 256 0 3 31
-/+ buffers/cache: 220 291
Swap: 255 208 47

There appears to be an artificial upper limit being placed on RAM availability. I would create an even smaller 128Mb swapfile but suspect that the server will simply go OOM, instead of using the available free RAM. Shall I be brave? Hmm, the eNlight console doesn't want to co-operate so I'm not convinced I could recover from a crashed server.

EJ

[ejsolutions]

Re: Centos memory allocation with eNlight

Just for completeness.
Continuing performance issues with eNlight has forced me to move the sites on it, to a different Cloud solution (along with an excellent pricing offer). It's not ideal, in that it's a US-based server but something had to be done and the feedback so far from users is very encouraging.

On eNlight:

[email protected] [~]# free -m
             total       used       free     shared    buffers     cached
Mem:           512        184        327          0          0         14
-/+ buffers/cache:        168        343
Swap:         1023        147        876

On the new platform:

[email protected] [~]# free -m
             total       used       free     shared    buffers     cached
Mem:           489        458         31          0         10         90
-/+ buffers/cache:        356        133
Swap:         1023        126        897

Memory is being used up as expected and as a consequence, the overall improvement to performance is dramatic. Unfortunately, this means "bye bye" to eNlight until someone can convince me that this issue is resolved.

[eUK-Rishi]

Re: Centos memory allocation with eNlight

Hi EJ,

I'm really sorry I got little late here. It is really unfortunate to see you away from eNlight.

In your last post, looking at the "total" RAM I believe that it is a CentOS6 VM. Below is a random 512MB VM #free -m
Its 64bit CentOS5. I tried to search 32bit one but not many opt for it.

[email protected] [~]# free -m
             total       used       free     shared    buffers     cached
Mem:           512        457         54          0          1         32
-/+ buffers/cache:        423         88
Swap:         1023        175        848

I would not agree to RAM limit on eNlight because all the VMs can fully utilize their allocated RAM. Its more dependent on OS.
eNlight is a Hypervisor which runs all Windows, CentOS, Ubuntu etc. and behaves same for everyone.

If you get some free time to explore more, do let me know

[ejsolutions]

Re: Centos memory allocation with eNlight

Hi,
A good deal of time and "messing about" had been done, trying both C5.8 and C6, including significant downtime and some service replication issues (Apache rebuilds). At the time, nothing appeared to make a significant improvement and I have/had 'moaned about it' at some length.
From previous VMs on different platforms, plus the newly established VM, the only real conclusion that could be drawn was indeed an issue with the Hypervisor and/or OS template. (The current low-end VM was built from a bare C6, with cPanel/WHM manually installed.)

I have some dregs of credit left on Enlight and assuming a 'trial' cPanel install, I may find some time to 'play'. Unfortunately, I'm a tad busy with ecommerce software at the moment, though do I need something to break the monotony, unless I power up the Raspberry Pi again.

Regards,
EJ

[Enjoying my speedy eUKHost vmware, as usual.]

[eUK-Rishi]

Re: Centos memory allocation with eNlight

Hi EJ,

I can understand your concern but there is nothing special done to the "VMs" of CentOS.
Its the same default CentOS which most of us have/use.

Let me know when would you like to give a try and I'll arrange credits so that you can test it to full scale & duration till your satisfaction.
Lets power up the Raspberry Pi

Cheers,
Rishi