Announcement

Collapse
No announcement yet.

High server load

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • ollybee
    replied
    Re: High server load

    Hi,

    Originally posted by woodpecker View Post
    We know since you collapsed our websites, but we didn't know, do you think we are psychic?
    Fair comment. We didn't mail out all customers since the migrations would be transparent. My formal job title is Enterprise support manager, I had discussed this with some of my customers and many others have come to know since but I accept you may not have done.

    We had been migrating customers to the new cluster for over 2 weeks before these issues came to light and everything was running smoothly up to that point. It's frustrating for us as we had used similar tools to ioping to run a synthetic workloads on the storage from test VM's across the new hosts to simulate the workload from the original VMware cluster, there were no issues during testing either. The system has plenty of capacity as it was designed designed with growth, performance and redundancy in mind.

    It's taking time to try and zero in on where the problems lies as everything is new compared to the old cluster and the issues can't be reproduced on demand.

    Oliver

    Leave a comment:


  • woodpecker
    replied
    Re: High server load

    "As you may know we are migrating our VMware servers to a new cluster"

    We know since you collapsed our websites, but we didn't know, do you think we are psychic?

    Current dire status of VMC5 I/O

    [email protected] [~/ioping-0.6]# ./ioping -RD .

    --- . (ext3 /dev/root) ioping statistics ---
    863 requests completed in 3000.0 ms, 295 iops, 1.2 mb/s
    min/avg/max/mdev = 0.6/3.4/237.9/9.5 ms

    Other cloud platforms are in the region of circa 15000 iops on this test

    Leave a comment:


  • RyanPS
    replied
    Re: High server load

    Our monitoring system suggests that you have disabled the virtual agents required for your system to run effectively. Please speak to support who will be happy to install these virtual agents for you. If other readers are unsure if they have these virtual agents, please also contact the support team who can check whether or not these are installed and install them if not.

    Leave a comment:


  • ollybee
    replied
    Re: High server load

    Hi,

    I am a senior administrator at eUKhost, let me update you on what has been going on.

    As you may know we are migrating our VMware servers to a new cluster. The bottom line is the SAN on the new cluster is not performing as expected or indeed in the same way it did during our testing phase. On Monday the storage required a complete restart and this caused 1.5 - 2hours outage realistically until all VM's where back up and running normally. Since then we have been suffering latency spikes which have been causing the high loads on the guest VM's. We are currently working with our storage vendor to find a solution to these issues. We are also putting together a plan B in case the issues can't be resolved to our satisfaction.

    We have also suffered a denial of service attack on Tuesday morning which will have affected customers on other platforms. DDOS attacks are a fact of life but this one was affected customer more widely due to the way the network was configured to accommodate the VMware live migrations between data centers.

    Let me apologise again for the outages and lack of information. Please be assured we are taking this seriously and working hard on a solution

    Oliver

    Leave a comment:


  • cbjroms
    replied
    Re: High server load

    I am also very disappointed with the lack of response from [email protected] & Tech Support over the last few days. Keep being promised relies asap and the nothing for 24 hours. What is going on?

    Leave a comment:


  • GarryK
    replied
    Re: High server load

    Hi!

    I've asked one of our Senior Administrators to look into this for you, they'll get back to you shortly.

    Regards,
    Garry K.

    Leave a comment:


  • squidier
    replied
    Re: High server load

    Now 4 incidents of high load since 9am this morning....

    Leave a comment:


  • woodpecker
    replied
    Re: High server load

    I find the lack of response by eukhost disgraceful, no response here, by email or by ticket, I'm quite happy to start via social media if you prefer but last time I did that you got teary about it.

    I'm looking at migrating elsewhere, had enough of this nonsense!

    Leave a comment:


  • squidier
    replied
    Re: High server load

    Originally posted by Rock View Post
    I've updated the status page. Please let me know if you still see any sluggishness with the servers.
    My server experienced another high load this morning.

    You have asked us to report back re sluggishness, we have - so now what?

    Going by yesterday's experience, this slowness precedes the server grinding to a halt completely.

    Leave a comment:


  • woodpecker
    replied
    Re: High server load

    Seems I'm just talking to myself here but this tests the latency by writing 1000 small files, again dismal performance vs the cheap VPS

    1.5 seconds to complete on the cloud, 0.005 Seconds on the VPS!

    VMC5

    [email protected] [/etc]# dd if=/dev/zero of=/root/testfile bs=512 count=1000 oflag=direct
    1000+0 records in
    1000+0 records out
    512000 bytes (512 kB) copied, 2.36528 seconds, 216 kB/s

    VPS

    [email protected] [~]# dd if=/dev/zero of=/root/testfile bs=512 count=1000 oflag=direct
    1000+0 records in
    1000+0 records out
    512000 bytes (512 kB) copied, 0.00500516 seconds, 102 MB/s

    Leave a comment:


  • woodpecker
    replied
    Re: High server load

    Anyone from EUKhost around?

    YOU NEED TO FIX THE VMC5 IO PERFORMANCE

    Here is a test of the super expensive cloud with extra money paid for SAS vs an EUKhost el cheapo VPS

    VMC5 3 test runs gave a dismal 10.9 / 18.1 /26 MB/s

    dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct
    1+0 records in
    1+0 records out
    1073741824 bytes (1.1 GB) copied, 98.8213 seconds, 10.9 MB/s

    Cheapo VPS

    3 test runs 163 / 309 / 338 MB/s

    dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct
    1+0 records in
    1+0 records out
    1073741824 bytes (1.1 GB) copied, 3.17711 seconds, 338 MB/s

    Leave a comment:


  • woodpecker
    replied
    Re: High server load

    And the support ticket answer from the clowns in India:-

    "I'm afraid but we're not able to access your server via SSH"

    Correct, and you're not having access, the SSH keys you try and install are deleted and the password and ports changed.

    VMC5 seems to be served from a NodeFour data centre now, is this a recent change?

    So, EUKhost, what EXACTLY have you done here, we pay extra for an SAS disk upgrade as well!

    Leave a comment:


  • woodpecker
    replied
    Re: High server load

    Originally posted by squidier View Post
    Again and again we are promised more updates when these things happen - but nothing changes.

    I believe there may be some sort of hardware upgrades in progress - if that is true, now is the worst time of year for it to go ahead.

    To make matters worse another server I have with Eukhost has now gone down. Not a good day.
    I think there is a serious issue with the io on VMC5, when the webalizer cron jobs ran at 5am the server just locked with a 1 minute load of 94, this is 4 CPU 10GHz cloud!

    Time: Wed Dec 2 05:00:17 2015 +0000
    1 Min Load Avg: 94.72
    5 Min Load Avg: 31.03
    15 Min Load Avg: 12.69

    Also at 2:20 am the server locked solid running an overnight rsync job with crazy loads as below.

    Time: Wed Dec 2 02:21:06 2015 +0000
    1 Min Load Avg: 81.26
    5 Min Load Avg: 70.29
    15 Min Load Avg: 41.24

    It looks to me like we are on a crippled or bandwidth limited SAN

    Leave a comment:


  • squidier
    replied
    Re: High server load

    Originally posted by Rock View Post
    I've updated the status page. Please let me know if you still see any sluggishness with the servers.
    My server does some backups overnight and I noticed that it reached very high loads a couple of times during the backup - this only happened very infrequently in the past, it happened 3 times last night.

    Leave a comment:


  • squidier
    replied
    Re: High server load

    Again and again we are promised more updates when these things happen - but nothing changes.

    I believe there may be some sort of hardware upgrades in progress - if that is true, now is the worst time of year for it to go ahead.

    To make matters worse another server I have with Eukhost has now gone down. Not a good day.


    Originally posted by woodpecker View Post
    I've seen a lot of high io wait times recently, this morning it was noticeably worse, live chat were as useless as ever so I logged a ticket to which no one bothered to reply, shortly after logging the ticket VM5 was dead for over an hour, 3 ecom sites down on one of the busiest weeks of the year!

    All support want to do is log into our VM, that's not going to happen, the issue appeared to be either the network bandwidth to the SAN or the SAN itself, at times we were seeing 99% io wait on 4 cpu cores which threw the load up and jammed the server.

    It seems to have settled now but as usual its too little too late...

    This isn't the first time this has happened, if you can't get your act together we'll have to move to something more robust.

    Leave a comment:

Working...
X