Unplanned outage-Exetel core network

Old, inactive threads

Unplanned outage-Exetel core network

Postby garryl on Thu Aug 30, 2012 9:40 am

Event:

We had a service disruption on our core network from 7:20 AM to 9:25am. All services should be restored by now.

Impact:

All services were down during this time.
garryl
Exetel Staff
 
Posts: 2
Joined: Fri Feb 24, 2012 9:06 am
Location: Sydney

Re: Unplanned outage-Exetel core network

Postby James on Thu Aug 30, 2012 10:51 am

A full reason for outage will be posted once we have investigated the root cause of this outage.

Exetel sincerely apologies for any inconvenience caused.
James
Exetel Staff
 
Posts: 3233
Joined: Mon May 09, 2005 10:27 pm

Re: Unplanned outage-Exetel core network

Postby ronzul on Tue Sep 04, 2012 6:16 pm

Hi

Is there any update on this? I see the outage last week as a serious matter. Not only was my own personal ADSL down, the Exetel website was down so there was no information about the outage. And the exetel phone line would either simply be engaged, or after getting through the initial IVR would be engaged.

Somewhat annoying that the newsletter that just came out says how wonderful everything is and makes no mention of a serious event.

Thanks
ronzul
 
Posts: 16
Joined: Mon Feb 07, 2005 11:51 pm

Re: Unplanned outage-Exetel core network

Postby James on Tue Sep 04, 2012 6:19 pm

ronzul wrote:Hi

Is there any update on this? I see the outage last week as a serious matter. Not only was my own personal ADSL down, the Exetel website was down so there was no information about the outage. And the exetel phone line would either simply be engaged, or after getting through the initial IVR would be engaged.

Somewhat annoying that the newsletter that just came out says how wonderful everything is and makes no mention of a serious event.

Thanks


A formal RFO will be posted here asap.
James
Exetel Staff
 
Posts: 3233
Joined: Mon May 09, 2005 10:27 pm

Re: Unplanned outage-Exetel core network

Postby James on Tue Sep 18, 2012 10:01 am

Report Date : 7:20AM 30/08/2012
Network Incident in Summary;

Root Cause of Incident – (Extrinsic or Intrinsic) : Intrinsic
Date / Time of incident Resolved : 09.15AM 30/08/2012
Date / Time of incident Closed : 05:00AM 31/08/2012

Network Incident in Detail:

One of Exetel's major gateway routers failed.
This occured due to the failure of router memory, with initial restoration actions not successful.
An emergency router replacement was made to restore the affected services.

Impact :

Exetel customers were experiencing browsing difficulties due to the DNS failure.
VOIP and email services were not functioning properly.
Exetel Helpdesk and Phone system was not functional.

Restoration Action :

A new permanent router was installed on 31/08/2012.

Root Causes :

Failure of critical hardware.

Issues Identified :

Changes should be made to current network topology to place redundant DNS servers in additional PoPs.

Corrective Future Action :;

Improve redundancy in north Sydney data center, and redundant servers in additional PoPs. Additional servers have now been installed at these PoPs.
James
Exetel Staff
 
Posts: 3233
Joined: Mon May 09, 2005 10:27 pm

Re: Unplanned outage-Exetel core network

Postby ronzul on Tue Sep 18, 2012 10:20 am

Thanks for the update.
It helps to know that exetel is taking steps to minimise the chance of such an outage in future.
ronzul
 
Posts: 16
Joined: Mon Feb 07, 2005 11:51 pm


Return to Archived threads

Who is online

Users browsing this forum: No registered users and 0 guests

cron