High server load

[jamie07051975]

Hi.

We currently have a number of cloud based servers with EUKHost and over the past few months one of them is getting gradually worse, suffering from high load.

Over the past few weeks I've transferred some of the bigger and more used web sites of it to try to easy the burden (most of the sites are test sites so clients can see the site before putting it live and so get very little traffic).

Over the past few days been trying to see why the server is struggling so did a top this morning:

top - 08:24:08 up 1 day, 14:24, 1 user, load average: 2.15, 3.34, 3.14
Tasks: 198 total, 1 running, 192 sleeping, 0 stopped, 5 zombie
Cpu(s): 0.1%us, 0.1%sy, 0.0%ni, 74.9%id, 25.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 5993032k total, 2465552k used, 3527480k free, 91832k buffers
Swap: 4128760k total, 2416k used, 4126344k free, 1079732k cached

To me this reports that the memory is fine, the server is 75% idle but all of the processes on the server are in a waiting state (25%). Have run an iostat -x 2 5 and it shows this:

avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 0.13 24.91 0.00 74.97

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 24.00 1.00 444.00 388.00 33.28 1.31 55.96 39.88 99.70
dm-0 0.00 0.00 24.50 0.00 448.00 0.00 18.29 10.43 574.45 40.71 99.75
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 0.25 43.98 0.00 55.76

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 83.50 36.00 25.00 588.00 864.00 23.80 2.14 34.74 16.36 99.80
dm-0 0.00 0.00 35.50 109.00 576.00 872.00 10.02 7.03 48.35 6.91 99.80
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

avg-cpu: %user %nice %system %iowait %steal %idle
0.12 0.00 0.12 38.58 0.00 61.17

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 4.50 13.50 3.50 228.00 32.00 15.29 9.20 78.00 58.74 99.85
dm-0 0.00 0.00 13.50 27.50 236.00 220.00 11.12 11.10 35.34 24.37 99.90
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

avg-cpu: %user %nice %system %iowait %steal %idle
6.13 0.00 1.25 33.29 0.00 59.32

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 6.00 53.00 25.00 1128.00 284.00 18.10 9.64 224.88 12.75 99.45
dm-0 0.00 0.00 53.50 11.00 1132.00 88.00 18.91 11.11 322.71 15.42 99.45
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

This shows that the bottle neck seems to be the disks as they are pretty much 100% in use constantly.

Having a hard time getting support to help on this, after I posted a support ticket about this I have now been told it is a network issue. Asked if this would affect the disks and they then said no. Arrgghhhh.

Can anyone offer any help?

[squidier]

Re: High server load

I think there's some sort of disk problem on VmwareCloud5 this morning. Not sure what's happening at the moment - I guess it's being worked on?
I found restarting httpd relieved it for a while.
Jane

[jamie07051975]

Re: High server load

I think there's some sort of disk problem on VmwareCloud5 this morning. Not sure what's happening at the moment - I guess it's being worked on?
I found restarting httpd relieved it for a while.
Jane

Jane, you should work in the support department. More sense from that than I've had the past few months!

That issue was raised directly after I sent in a support ticket after having a right load of rubbish from the live chat. Just get really frustrated at times, they have some great people there (like Oliver) but then live chat just seems to be... (leave you fill in the gaps).

I asked if the cloud5 issue would be related to the disk issue I was having and was told no as it was a network issue!

[squidier]

Re: High server load

Well, it may well be a network problem but it's manifesting itself to us as a disk issue.

I agree there are some good people in support, I would just like a little more feedback so I can manage client expectations.

[jamie07051975]

Re: High server load

My disks are pretty much in use 100% which means that everything is waiting on the disks. Server is pretty much idle, loads of memory free. Seems like the same issue!

[squidier]

Re: High server load

I've just rebooted and have a better load now, %wa still looks too high though.
Would do a disk speed test but don't want to overload it until they've finished fixing.

[jamie07051975]

Re: High server load

I've had no response from my original ticket (marked as "emergency") - sent 2.5 hours ago.

If I was told "really sorry, we have a major issue but we are working on it" fair enough, but nothing. Problem is that when speaking with live chat I have to go through the same again each time and then they tell to send a support ticket in (arrggghhhhhh) probably because they have no clue.

[squidier]

Re: High server load

Just been told it's all ok now. Did a disk speed test and got 42.4 MB/s which is better than the 1.7 I had half an hour ago.
Waiting to be told what the problem was.

[jamie07051975]

Re: High server load

Thanks for heads up, not been told anything!

[jamie07051975]

Re: High server load

Thanks for the PM Jane - couldn't reply (not enough posts)

[eUK.Ed]

Re: High server load

We are very sorry about the disruption that this mornings incident on one of our VMware Cloud clusters has caused, the issue should now be resolved. If you are still facing issues please contact our support team as soon as possible as your particular case will need to be looked into in more detail. We are currently compiling a report on today's incident which will be available as soon as possible.

Kind Regards,

Ed Berry

[ejsolutions]

Re: High server load

A network problem to the disc array could look as if the disc is overworked. This has happened in the past (on a different node/array).

I'm not impressed at all with today's 2.5 hour downtime, nor the appalling Live Chat and initial phone call that I made. Only once a member of Customer Support was contacted, did something constructive happen, though with no clues as to what the fault actually was.

[Who said failover, redundancy and responsiveness?]

As a side note the eukstatus web page is an utter waste of space. It remained static with no updates for 2 hours: what is the point?!

It looks as though little has been learned from the debacle just under a year ago, when the transfer to the new data centre went as smooth a 150 grade sandpaper.

EJ

----------------------------------------------------------------------------------------
I'll quote from the sales blurb on the main EUKHost website and just say, caveat emptor.

"Whilst failure of the server hardware or network components would take a standard hosting environment offline, our cloud environment has been engineered to specifically target these issues.."

[jamie07051975]

Re: High server load

The status is page is pointless, its still not updated (looks the same as it did yesterday). It took 12 hours for me to get a reply from your staff after I created a support ticked marked as "Emergency" at around 12:15 midday yesterday.

I'm a web developer not a server technician and I put the above details together in a matter of minutes but your support staff on live chat had no clue what I was talking about.

As I had no reply from you yesterday I had no option but to start moving sites away to another provider. I may only have a handful of servers with you but I'm sure others are feeling pretty shocked with the level of "failover, redundancy and responsiveness" as mentioned by ejsolutions.

I have been sold extras in the past which I have never needed, and only when a member of staff from another department got involved was I told what the real issue was and how it could be fixed. Pro-active monitoring never worked, even after I sent in three support tickets when the server went down and no-one was alerted. The last time I was told it was now working, yet logging I could see myself that it wasn't working.

Had enough now.

[EUKJenna]

Re: High server load

Good Afternoon,

We are sorry to hear that you have all been affected by yesterdays outage.

So that we can address your concerns and SLA claims, please can you contact: [email protected] with your queries.

We are currently awaiting more information and an incident report and we will update all tickets and claims as soon as possible.

Kind Regards,
Jenna Strong
Customer Relations
eUKhost

[squidier]

Re: High server load

Looks like it's gone down again today.