Power Outage

[eUKhost.com]

There was some problem with UPS system in Bluesqaure Building 2 which resulted in power outage for most of our servers in Bluesqaure Building 2.

Technical details are not available as of now as technicians will let us know details of this power outage once investigation part is completed. We are working on getting some of our servers online which failed to boot after power outage.

[jc8654]

After talking to support they hope to have everything back online within a couple of hours. They have to work through the nodes which are down and gradually bring them back online as they need to be restarted.

[Site]

Downtime was minimal here, well done Euk / Bluesquare for getting them back up and running again.

[martynh99]

has it gone again - got a semi dedicated windows server that went down about an hour ago - tech support on the case now.

[dbm44]

We had a semi-dedicated Linux box that was down for over 8 hours. Sorry but that level of downtime is simply unacceptable. Would like a full explanation please.

[martynh99]

Originally Posted by dbm44

We had a semi-dedicated Linux box that was down for over 8 hours. Sorry but that level of downtime is simply unacceptable. Would like a full explanation please.

Whilst I sympathise with you, it's 99.5% uptime - sounds great when you hear it but do you realise that means your service can be down for 43 hours in a year and it's still met that uptime.

Datacenters are built with resilience but problems happen at even the best datacenters - it's how you handle the outage / communicate this to your customers that matters.

In this case eukhost haven't done too bad, they've posted there is a problem - what I would have liked to have seen is a list of master nodes that had suffered hardware failures, that way I would have known that my semi would have taken a little longer to recover rather than wondering has it been spotted and then having to get on online chat going taking away valuable sysadmin resource to chat to me who could have been fixing servers instead.

Time for a live, automated system status page maybe ?

But overall

[dbm44]

I understood the uptime guarantee to be 99.95%, which is 4.32 hours a year - a lot less than the 43 hours you mention.

Oh and as I type the server is down again.

I have an IT management background so I fully acknowledge that stuff goes wrong to everyone, been there done that. Still don't think 8 hours plus downtime is OK

[martynh99]

Originally Posted by dbm44

I understood the uptime guarantee to be 99.95%, which is 4.32 hours a year - a lot less than the 43 hours you mention.

Apologies, after 4 hours down I rechecked and the sales blurb says 99.97 so 2.62 hours !

Oh and as I type the server is down again.
I have an IT management background so I fully acknowledge that stuff goes wrong to everyone, been there done that.

Me too, thats why I pointed out the real stats about % uptime, many people see 99.5 and think it looks great but when you convert it to real time, the figures are shocking
Still don't think 8 hours plus downtime is OK

^^^^ agreed,

So Euk - what is the payback when you break your 99.97 uptime guarantee then ?

[eUK-Nick]

Hello Martyn,

Kindly send an email to feedback{at}eukhost.com. I will check your accounts myself and get back to you on this.

[eUKhost.com]

Originally Posted by martynh99

So Euk - what is the payback when you break your 99.97 uptime guarantee then ?

1.6. Performance Credit ("Performance Credit") occurs when 100% uptime is not met. eUKhost LTD. will refund the customer 5% of the monthly fee for each 30 minutes of downtime (up to 100% of the customer's monthly fee). Network downtime is measured from the time a trouble ticket is opened by a customer to the time the server is once again able to transmit and receive data.

Uptime Guarantee differs on some of our hosting plans so please refer our SLA to know more about compensation that we offer on such outages.

[eUKhost.com]

Originally Posted by dbm44

We had a semi-dedicated Linux box that was down for over 8 hours. Sorry but that level of downtime is simply unacceptable. Would like a full explanation please.

Please let me know IP address of your semi-dedicated server.

[Site]

Is there any comment from Bluesquare regarding the failure of the UPS system?

[dbm44]

Originally Posted by eUKhost.com

Please let me know IP address of your semi-dedicated server.

Have sent you a PM with the IP address

[eUK-Nick]

Originally Posted by dbm44

Have sent you a PM with the IP address

Hello David,

I have sent you an email with full details. Please do have a look

[martynh99]

Probs again ?

Mine was back online at 9:58 but it appears to have gone again in last 10 minutes - support on the case again.

[eUK-Nick]

Originally Posted by martynh99

Probs again ?

Mine was back online at 9:58 but it appears to have gone again in last 10 minutes - support on the case again.

Martyn,

The semi-dedicated will be back up in the next 10 minutes. We have restarted it right now. This is being done (even though it is morning) to correct a few file system errors which crept in due to the outage.

[eUK-Nick]

We would like to give all our customers an update about the power outage that happened on Wednesday 30th April. There were two factors which hampered our response to the short power failure which we have detailed below.

The nature of the UPS failure was unique according to the manufacturer. There are over 3000 systems of the same model installed worldwide, and not one of these has experienced a similar failure. The failure was extremely unusual and unexpected, and the UPS manufacturer is having an engineer from Switzerland over to examine the failed system in more detail. The manufacturer has not yet determined the exact cause of the fault and as such we are not in a position to update you on what action we will be taking.

When the UPS failed a number of capacitors and other electrical components were overloaded and burnt out, this in turn triggered our fire alarm.

The secondary reason that the outage had a wider impact was that our two external DNS resolving name servers failed to come back up on reboot. They booted into the operating system correctly, however the actual DNS daemon did not restart. This meant that for all clients who use our DNS servers to resolve external domains, they were not able to do this for a period until these servers were changed to new ones.

To ensure that the same DNS issue does not happen again we are implementing some changes. We have already taken the DNS daemon out of service that failed to restart automatically and are currently running on an interim solution. Secondly, we are adding a new DNS server in to a separate site that runs off different power systems. This will become the secondary server, and the existing secondary server will become a tertiary server. The third and final change we are making is to use a combination of different DNS daemons on our three DNS servers, to ensure that we are not reliant on a single item of software.

We will keep you all updated with further progress. Thank you all for your patience.

[Site]

Hi Nick,
Many thanks for the update, its nice to see your on the case.

[shoecake]

Nick,
Although I was not affected apart from the initial outage, I just wanted to thank you for the detailed update of what happened. I am very impressed to get this level of detail, which is not normally forthcomming from other providers.

I hope you keep doing this as I find it very usefull for myself and for my customers to know what is going on when there is a problem. FYI this level of openness is one of the main reasons I chose eukhost.

Patrick

[Site]

Originally Posted by shoecake

This level of openness is one of the main reasons I chose eukhost.

Thats very true


Well done Nick... any news from Bluesquare on the Outage?