Worldwide markets roiled by data-centre snafu in Chicago suburb
The 450,000-square-foot Aurora complex has served as the primary hub of digital operations for CME for nearly two decade
[NEW YORK] One of the first signs of trouble arrived at 9.41 pm Eastern time on Thursday (Nov 27), when most of Wall Street was shut and traders were still enjoying the Thanksgiving holiday in the US.
“Due to technical issues,” CME Group said in a one-line e-mail to clients, its futures and options “markets have been halted”.
The problem, it turned out, was the cooling system at a data-centre complex in the suburban town of Aurora, Illinois, some 50 miles (80 kilometres) from Chicago, which serves as the main hub for trillions of US dollars of derivatives traded each day. Inside, temperatures soared past 100F (38 deg C) despite the frigid weather, according to sources familiar with the matter.
At the time, the CME’s information indicated the outage at the facility – run by private-equity owned CyrusOne – would be brief, according to a source with direct knowledge of the situation. As a result, it opted against switching over to a backup facility near New York City.
But before long, the magnitude of that fateful decision would become painfully clear. Despite a flurry of e-mails from CME assuring clients the problem would be fixed in the “near term”, the malfunction persisted and brought down vast parts of the global financial system for hours. From Tokyo to London, and eventually New York, trading in everything from gold and oil to wagers on the direction of US interest rates came to an abrupt halt.
Even after trading was largely restored on Friday, disruptions extended well into the US session, with CME Direct, a trading platform provided by the exchange, offline for most of the day.
The outage underscored a vulnerability in increasingly integrated global markets that rely on a handful of dominant exchanges. It also raises questions about the contingency plans of CME, one of the world’s largest exchanges for derivatives contracts, and its heavy reliance on the data centre it sold in 2016 to CyrusOne, a company now owned by KKR and Global Infrastructure Partners.
The shutdown “shows how concentrated futures markets really are, there just are not many alternative venues for the main products”, said Thomas Texier, group head of clearing at Marex Group, a London brokerage.
The 10-hour outage surpassed the one that hit CME in 2019 and again underscored how much the exchange has become an integral part in the world’s markets. On average in October, derivatives trading volumes amounted to more than 26 million contracts every day, according to data from the group.
While it came during a day when US trading was expected to be subdued, due to the Thursday holiday, it still proved vexing to investors around the globe who needed to make month-end adjustments by rolling positions from one contract to another.
In Singapore, one oil trader said when the initial alert was issued, they thought it was a hoax because the trades and quotes were still streaming in. But a few minutes later, the screen suddenly froze and they were booted out of the Nymex platform. A trader in London thought he had Wi-Fi connection issues.
“We have had to trade some cash Treasuries today and it was noticeably thinner and wider,” said James Athey, a portfolio manager at Marlborough Investment Management. “Month-end, day after Thanksgiving, CME down. It’s not an ideal combo.”
By the end of the trading day on Friday, CME had restored all of its trading operations, including CME Direct. A representative for the exchange declined to comment beyond the updates it had provided to clients throughout the day.
CyrusOne said that the problem was caused by a machinery failure that affected the systems used to cool its computer systems and it was “working around the clock to restore normal operations as quickly and safely as possible”. It said that it had successfully restarted several chilling systems at limited capacity and deployed temporary cooling equipment to supplement its operations.
It’s unclear what exactly happened to CyrusOne’s cooling system. But the data centre does have a redundancy system and offers free cooling when temperatures fall below 30F, according to information on CyrusOne’s website.
The 450,000-square-foot Aurora complex has served as the primary hub of digital operations for CME for nearly two decades. It’s famous among high-frequency traders and Wall Street firms, who have long jostled for positions around the site to get an edge on competitors by shaving fractions of a second off the time it takes for trades.
In 2016, CME decided it wanted to shift away from owning infrastructure and sold the site to Dallas-based CyrusOne. As part of the deal, CME agreed to rent space from CyrusOne for 15 years so it could continue to house the computers at the site that keep its markets running, essentially outsourcing its day-to-day operations. KKR and Global Infrastructure Partners agreed to buy CyrusOne in 2021.
The centre has continued to serve as a key node for traders across the globe, and the impact was felt broadly. In London hours, for example, the trading of US Treasury futures was halted, gold saw erratic moves and US crude and palm oil on the Bursa Malaysia exchange were also affected.
Even as the trading system was restored, some market makers remained hesitant to engage in trades until they could be assured the problem was fixed, according to sources familiar with the matter.
Others saw the occurrence during what’s typically a slow day in the US as one saving grace.
“I woke up thinking my Wi-Fi was out,” said Ritik Katte, chief investment officer at MCD Capital, a London-based investment firm. “Liquidity is lower than usual, so it seems like the Thanksgiving holiday has been extended.” BLOOMBERG
Decoding Asia newsletter: your guide to navigating Asia in a new global order. Sign up here to get Decoding Asia newsletter. Delivered to your inbox. Free.
Share with us your feedback on BT's products and services
