You are here
Hard disk and software failure root cause of SGX outage: CEO
A COMBINATION of a hard disk failure and an application that failed to detect the problem was the root cause of the Singapore Exchange's (SGX) securities market shutdown last Thursday - the longest trading halt in the bourse's history that has raised concerns about its reputation.
The disk supplied by HP had failed. This was followed by the failure of the software supplied by Nasdaq which meant that there could not be a switch to a secondary system, SGX chief executive Loh Boon Chye told reporters on Tuesday.
The disruption was prolonged due to challenges in the orders and trade reconciliation process, he said, adding that the faulty disk had been replaced last Thursday and complete health checks conducted.
He said SGX is working with its vendor to review the application which sends out clearing confirmation messages, and will implement the necessary changes to ensure detection by the application of specific hardware problems.
Mr Loh also promised to improve its process in data generation and fine-tune the data files to better enable the reconciliation processes for members.
"We now know what happened. We need to do the forensics on the software. We also know now that one-off specific reports need to be generated quicker. We will work with the industry on what these reports may be and also come up with a catalogue or library of data files that need to be generated, simulate those scenarios and work with the industry to come up with a plan," he said.
SGX said besides working with members to review their order and trade reconciliation process to improve overall recovery and market resumption in the event of a similar recurrence, the number of business continuity planning scenarios will also be increased.
Detailing Thursday's disruption, SGX said at 9.38am it had detected input/output errors on a disk that runs the application to send out clearing confirmation messages to members.
"As the application did not detect the disk failure, which it should have, it did not automatically cutover to SGX's backup secondary system. SGX initiated a manual cutover from the primary to secondary systems at 1012 hours."
As a result of the disk failure, some clearing confirmation messages were not generated. Following the cutover, the application attempted to resend all missed messages, which resulted in some messages being duplicated, the exchange explained.
Trading ceased at 11.38am, and from 11.51am, members could reconcile and manage their orders including those which were traded. Trading was orderly throughout the time the market was open.
"While there was a problem with the clearing confirmation messaging application, there was no problem with the trade confirmation messaging application, which continued to work as normal," SGX said.
"To reconcile their orders and trades, members can rely either on trade confirmation messages or clearing confirmation messages. Those using the latter in this case experienced problems with reconciling orders and trades," it said.
SGX said order and trade reconciliation required the exchange and its members to match their trade or clearing confirmation data files. It added that the entire process took longer than expected as members use different systems, and either the trade confirmation or clearing confirmation messages, or both.
Mr Loh said the exchange has filed an initial report with the Monetary Authority of Singapore (MAS) and would submit a final report by the end of the month.
The extended shutdown last week was the fifth time in two years that SGX has suspended one of its platforms, frustrating traders and raising concerns.
In June 2015, MAS reprimanded SGX for outages in November and December 2014, highlighting that SGX had failed to recover critical systems within a four-hour target in the November disruption.
In addition to the two 2014 disruptions, SGX also experienced disruptions to its derivatives platform in August and October 2015.