Disk failure root cause of SGX's July 14 outage

A DISK failure and an application that did not detect the problem were the root cause of the securities market shutdown on July 14 (Thursday), Singapore Exchange (SGX) said on Tuesday.

"Investigations found that Thursday's event was caused by a disk failure and an application that did not detect the problem. The disruption was prolonged due to challenges in the orders and trade reconciliation process," SGX said.

The faulty disk had been replaced on Thursday and complete health checks conducted, it said.

"We are working with our vendor to review the application which sends out clearing confirmation messages, and will implement the necessary changes to ensure detection by the application of specific hardware problems," it said.

SGX promised to improve its process in data generation and fine-tune the data files to better enable members in their reconciliation processes.

"We will work with members to review their order and trade reconciliation process, to improve overall recovery and market resumption, in the event of a similar recurrence," it said.

The number of business continuity planning scenarios which require industry-wide participation for reconciliation and recovery will be increased.

Detailing Thursday's disruption, SGX said at 9.38am it had detected input/output errors on a disk that runs the application to send out clearing confirmation messages to members.

"As the application did not detect the disk failure, which it should have, it did not automatically cutover to SGX's backup secondary system. SGX initiated a manual cutover from the primary to secondary systems at 1012 hours."

As a result of the disk failure, some clearing confirmation messages were not generated. Following the cutover, the application attempted to resend all missed messages, which resulted in some messages being duplicated, the exchange explained.

Trading ceased at 11.38am, and from 11.51am, members could reconcile and manage their orders including those which were traded. Trading was orderly throughout the time the market was open.

"While there was a problem with the clearing confirmation messaging application, there was no problem with the trade confirmation messaging application, which continued to work as normal," SGX said.

"To reconcile their orders and trades, members can rely either on trade confirmation messages or clearing confirmation messages. Those using the latter in this case experienced problems with reconciling orders and trades," it said.

SGX said order and trade reconciliation required the exchange and its members to match their trade or clearing confirmation data files.

"The entire process took longer than expected, as members use different systems, and either the trade confirmation or clearing confirmation messages, or both," it said.