Network issues

[Fidget]

Is it possible to post to this thread when these 'network issues' become clearer? Its been 90 minutes that we have been compleetely down now.

thanks

[eUKhost.com]

There was routing problem in the datacenter due to which some IPs were not resolving. Problem was sorted in less than 30 minutes but windows servers took some more time.

Your website is up and running right now.

[Fidget]

Sorry to hear of the problems. We aren't on windows, we are on a linux vps. But our websites are all still down

[eUKhost.com]

On 21st March 2007 at 20:27 there was an unscheduled outage which affected some routes into our network.

This occurred during a scheduled maintenance window on a small section of our network. The small number of clients that this would have affected were informed well in advance as per our usual procedures.

The course of events that led to the network fault were as follows:

1) We took a layer 3 OSPF Link down as part of the scheduled maintenance window on a small section of our network. This change should have automatically routed around the rest of the network.
2) Problems were observed with some routes by the onsite network team and these were investigated.
3) The link that was taken down was brought back up in order to return to the original configuration.
4) All network devices were investigated for errors. All links were showing as healthy.
5) The Network Team discovered that the CEF table in a core router had entries that had not been properly updated automatically during the course of taking the OSPF Link down.
6) A Layer 2 interface was taken down because of the corrupt CEF tables which were then forced to be cleared. This resolved the problem and all Network Diagnostics show a clear bill of health for routes in and out.

This was resolved at 21:14

This lasted 47 minutes.

After the situation was resolved the Network Team investigated what would have caused the CEF table to have maintained old entries. It is believed at this stage that it was as a result of a Cisco Software Bug as there was no hardware or link failure or error. We will inform Cisco about this immediately and investigate further in the morning during working hours.

Date: 21st March 2007
Time: 20:27
Duration: 47 minutes

We would like to express our sincere apologies for the inconvenience this will have caused and assure all clients that we will be pursuing and seeking a resolution from Cisco regarding the error.

We would like to apologize for the length of downtime. This error was very unusual, in that there was no hardware or network link failure, and the configuration of all network devices was correct. Our network staff checked these aspects of the network in this order. The problem was actually caused by an undocumented bug in the Cisco IOS software, which prevented the routers forwarding tables from being cleared automatically by a layer 3 routing adjustment. This should happen automatically and transparently. The unique nature of this error was why it took quite some time for our staff to track down and put a workaround in place to resolve the error.

[Fidget]

That's a very detailed response and much appreciated.

But all of our websites are still down (nearly 2 and a half hours). I've requested further assistance via my original support ticket though.

[eUKhost.com]

I've asked our VPS team to look into it. Not sure whats wrong with your VPS as the routing problems got sorted long time back.