[eUK-Nick]

We would like to give all our customers an update about the power outage that happened on Wednesday 30th April. There were two factors which hampered our response to the short power failure which we have detailed below.

The nature of the UPS failure was unique according to the manufacturer. There are over 3000 systems of the same model installed worldwide, and not one of these has experienced a similar failure. The failure was extremely unusual and unexpected, and the UPS manufacturer is having an engineer from Switzerland over to examine the failed system in more detail. The manufacturer has not yet determined the exact cause of the fault and as such we are not in a position to update you on what action we will be taking.

When the UPS failed a number of capacitors and other electrical components were overloaded and burnt out, this in turn triggered our fire alarm.

The secondary reason that the outage had a wider impact was that our two external DNS resolving name servers failed to come back up on reboot. They booted into the operating system correctly, however the actual DNS daemon did not restart. This meant that for all clients who use our DNS servers to resolve external domains, they were not able to do this for a period until these servers were changed to new ones.

To ensure that the same DNS issue does not happen again we are implementing some changes. We have already taken the DNS daemon out of service that failed to restart automatically and are currently running on an interim solution. Secondly, we are adding a new DNS server in to a separate site that runs off different power systems. This will become the secondary server, and the existing secondary server will become a tertiary server. The third and final change we are making is to use a combination of different DNS daemons on our three DNS servers, to ensure that we are not reliant on a single item of software.

We will keep you all updated with further progress. Thank you all for your patience.