Orange has revealed the reasons for the network issues that negatively impacted emergency calls and voice services on 2 June in France.
Stephane Richard, Chairman and CEO of Orange, assigned the Group’s Internal Audit department to conduct an investigation into this crisis to identify the root causes and to provide recommendations.
The investigation confirms that the network issue was due to a software malfunction. Orange revealed the network outage was not due to cyberattack.
On 2 June, between 16h45 and midnight, Orange’s voice services as well as access to certain emergency services were severely disrupted on a national level. During this period, an 11 percent deterioration in the rate of call routing (or approximately 11,800 calls) to emergency services was observed.
This malfunction impacted the interconnection between mobile voice and Voice over IP (VoIP) services on the one hand and those hosted on the PSTN switched network (including most emergency numbers) on the other hand. The incident occurred following a network modernization operation that was initiated in early May to increase capacity in response to growing traffic.
The interconnection architecture is based on a platform of “Call Servers”. A bug in the Call Server software provoked a malfunction of this platform resulting in the disruption to the service. This bug was activated following the application of standard reconnection commands, disrupting the overall operation of the Call Servers, despite their redundancy over six different sites.
This software failure has been identified by the Group’s partner and supplier of the equipment concerned, and a fix has been issued.
Alert process
Orange’s technical teams identified the software malfunction immediately thanks to the internal alert systems. They initiated the first technical operations but were unfortunately unable to restore the service.
The complexity of the malfunction, the variety of technologies involved and the specific network architectures of the emergency services delayed the diagnosis. Once the diagnostic had been established, the problem was resolved within a few hours thanks to the mobilization of around one hundred experts.
Despite the mobilization of the technical teams, the delay in activating the managerial crisis committee resulted in late communication to all stakeholders.
Recommendations
The investigation recognizes the immediate involvement and strong mobilization of the Group’s technical teams from the onset of the disruption, as well as the collaboration with authorities in order to restore normal service.
As Chairman of the GSMA, Stephane Richard will also propose, at a global level, the creation of a team responsible for listing and analyzing sensitive network malfunctions in order to better share feedback between operators.
