Contractor error caused server outage behind DBS, Citi banking disruption: Equinix

EQUINIX said on Friday (Oct 20) that a mistake made by its vendor contractor during a planned control system upgrade led to the outage at one of its data centres last Saturday.

The mistake created a technical issue with its chilled water system and caused some of the data centre’s halls to overheat. This impacted its tenants’ operations, including those of Citibank and DBS.

“A vendor contractor incorrectly sent a signal to close the valves from the chilled water buffer tanks, affecting the flow of chilled water to the cooling system,” Equinix said.

It added that it is reviewing its processes, and that it is putting updated policies and procedures in place, as well as additional levels of reviews and audits to prevent similar incidents from happening during future upgrades.

Market watchers said that it may not have been wise for Equinix to embark on such system upgrades in the day, when digital services are likely to be depended on.

IDC Financial Insights research director Ashish Kakar said that he was surprised that such maintenance was done in the day instead of in the middle of the night.

“You never do it at a time when many people are actually using (the services),” he said.

“It is always at a time when you think people are sleeping, fewer people are accessing the system. Then, you go ahead and do an upgrade.”

Technical director of Asia-Pacific at Chint Global Lim Say Leong noted that data-centre providers typically decide when they would like to carry out maintenance at their data centres. Maintenance can also be arranged during the day if adequate provisions are made.

Still, he noted, the choice to schedule maintenance in the day instead of at night may have been related to costs: “If they’re outsourcing the service, I think the cost is definitely very attractive.”