Contingency plans in focus after SingTel fire

[SINGAPORE] SingTel's engineers spent the day piecing together fibre optic strands by hand amid charred post-fire debris at the telco's Bukit Panjang facility yesterday.

Finding the cause of Wednesday's fire and the subsequent service disruptions, however, will preoccupy the firm during the coming weeks.

The fire, which had disrupted services for consumers and businesses over the past two days, had burnt 149 fibre optic cables in a chamber at the telco's Bukit Panjang exchange, 81 of which were OpenNet's. The other two chambers in the building were not affected.

Fibre cable repairs for corporate customers were completed last night, while the remaining services were on track for restoration by 7am today.

There is no preliminary estimate for the financial fallout of the fire or the service disruption, SingTel's CEO Consumer Singapore, Yuen Kuan Moon, told the media in a briefing at ComCentre yesterday.

In response to questions about what could have caused the fire and what role, if any, the facility's fire suppression system had played, Mr Yuen said, "We are currently investigating. Our priority is to first ensure that service is being restored."

In the coming weeks, the questions will begin in earnest. Within the telco, an inquiry will be convened to determine the cause of the fire and how its distribution network can be reinforced, Mr Yuen said.

SingTel also defended its business continuity plans yesterday, saying that it has physical location diversity - the Bukit Panjang building is one of nine such locations in Singapore - and electronic diversity, in which some switches are replicated.

"Unfortunately, in the case of the Bukit Panjang office, the damage caused by the fire is physical," the telco said.

The restoration task was made harder because the colour codes used to differentiate one kind of cable from another had been razed off by the fire. Engineers had to either divert cables to an undamaged chamber or cut out burnt portions and splice together the undamaged fibres - each thinner than a human hair - by hand.

Each cable can carry 96, 192 or 288 fibre strands, each strand supporting up to 16 corporate customers or 24 residential ones.

SingTel will not be the only entity asking hard questions of itself. The industry regulator, the Infocomm Development Authority of Singapore (IDA), will carry out a "thorough study" of the incident, said Minister for Communications and Information Yaacob Ibrahim yesterday. "We have to wait for the police report . . . and then we will start our investigation," he said, speaking at SingTel's Bukit Panjang building yesterday.

An IDA investigation has the potential to result in mammoth fines. Just last week, M1 was fined a history-making $1.5 million for the outage of its 2G and 3G mobile phone services in January.

Dr Yaacob called what happened on Wednesday a "major incident that we are concerned about". Telephone exchanges are considered critical pieces of infrastructure, he noted.

Other quarters, too, will look askance at SingTel. DBS, which saw some of its branches and ATMs affected on Wednesday, told The Business Times that it has a "diverse network contingency plan that ensures minimal disruption to our businesses in the event of an incident such as the fire at SingTel's Bukit Panjang exchange".

DBS is "working with SingTel to understand why the network contingency plan was not effective for the small proportion of DBS/POSB branches that were impacted," the bank's spokeswoman added.

Two of UOB's branches and 11 of OCBC's ATMs were also affected by SingTel's fire on Wednesday. All three banks saw services resume either that night itself or early yesterday morning before the start of business.

"Our branches were not impacted as our network was designed to be able to be supported by a secondary exchange if the primary one fails," an OCBC spokesman said.

BT understands that a bank's ATM might be able to shrug off a service disruption if it has a backup network line, as long as both the main and backup line are not connected to the same exchange.

Yesterday, Dr Yaacob said the IDA had always been concerned about the need for backup systems, even before the SingTel fire happened. "We've been looking at it closely. Every incident will be a learning point for us . . . clearly, there are things (that) we will begin to learn because the systems are becoming very complex," he said.

By yesterday, the collateral damage from the fire appeared to have been contained. By 6pm, OpenNet had restored service to 25 per cent of M1's 1,000 affected fibre broadband customers.

StarHub, which leases optical fibre capacity from SingTel, said that cable TV, cable broadband and digital voice services for affected customers were fully restored yesterday afternoon, while 36 per cent of its fibre broadband services had been restored by yesterday evening.