- by Stocktry Expert
- 07th Oct 2020
Reasons Why Tokyo's $6 Trillion Stock Market Came to a Halt:
At 7:04 a.m. on an autumn Thursday in Tokyo, the stewards of the world’s third-largest equity market realized they had a problem.
Due to the malfunction of a data device critical to the Tokyo Stock Exchange’s trading system the automatic backup had failed to kick in. It was less than an hour before the system, called Arrowhead, was due to start processing orders in the $6 trillion equity market. Exchange officials could see no solution.
In 1999 since the exchange switched to a fully electronic trading system the full-day shutdown ensued. It attracted criticism from market participants and authorities and shone a spotlight on a lesser-discussed vulnerability in the world’s financial plumbing — not software or security risks but the danger when one of the hundreds of pieces of hardware that make up a trading system decides to give up the ghost.
Finance Minister Taro Aso told reporters in Tokyo that, "Exchanges are a crucial part of market infrastructure and it’s unacceptable that trading opportunities were denied." He further added, "You’re dealing with machines so it’s always possible they will break. They need to create the infrastructure with that possibility of a breakdown in mind.”
The TSE’s Arrowhead system launched to much fanfare in 2010, billed as a modern-day solution after a series of outages on an older system embarrassed the exchange in the 2000s. The “arrow” symbolizes speed of order processing, while the “head” suggests robustness and reliability, according to the exchange. The system of roughly 350 servers that process buy and sell orders had a few hiccups but no major outages in its first decade.
That all changed on Thursday, when a piece of hardware called the No. 1 shared disk device, one of two square-shaped data-storage boxes, detected a memory error. These devices store management data used across the servers and distribute information such as commands and ID and password combinations for terminals that monitor trades.
The system called failover arose when the error happened which is automatic switching to the No. 2 device. The exchange’s executives couldn’t explain behind the failure of the process which had a knock-on effect on servers called information distribution gateways that are meant to send market information to traders.
Disappearing data:
At 8 a.m., traders preparing at their desks for the market open an hour later should have been seeing indicative prices on their terminals as orders were processed. But many saw nothing, while others reported seeing data appearing and disappearing. They had no idea if the information was accurate.
A minute later, the bourse made its first communication, informing systems administrators at securities firms that there had been an issue. At some brokerages, that didn’t immediately filter down to befuddled trading desks.
At about 8:05 a.m., Twitter — often used by traders to communicate outside of more official communication channels monitored by compliance — began to buzz with rumours of an issue. Traders described a growing sense of confusion as few answers came from the bourse.
Masaya Akiba, a broker at Marusan Securities Co.’s stock-trading department said, "We didn’t know if it was our system or the exchange." She further added, "We only confirmed it when the exchange put out a release."
At 8:36 a.m., the bourse finally informed securities firms that trading would be halted. Three minutes later, it issued a press release on its public website — although only in Japanese. A confusingly translated English release wouldn’t follow for more than 90 minutes.
It was the first time in almost fifteen years that the exchange had suffered a complete trading outage. The Tokyo bourse has a policy of not shutting even during natural disasters, so for many on trading floors in the capital, this experience was a first.
Historic decision:
Some market participants fumed at the closure. Others, with nothing to do, occupied their time by reading research notes or trading commodities.
The chief fund manager at Mitsubishi UFJ Kokusai Asset Management Co. in Tokyo said, "I didn’t think much of it at first. Previous outages were quickly resolved so I assumed orders would just be delayed.”
In 2012, after the switchover to Arrowhead, the exchange had quickly resolved limited issues. Many expected the bourse to do the same this time, too.
But as the hours passed, Hajime Sakai, the chief fund manager at Mito Securities Co., grew increasingly uneasy.
“I really couldn’t pay attention to much else,” he said. “I wasn’t like, ‘Open the market!’ It was more like, ‘whichever it is, make your call on it, fast.’”
The call was a daunting one. After the failed switch to the backup, the exchange had manually forced switchover to the No. 2 shared disk device. At this point, the administrators had a choice: they could seek to restart trading, but this would have entailed a full reset of the system — shutting down the power and rebooting.
Data for orders already received from securities firms would have been lost, without having been cancelled. That would have led to anarchy, securities firms told the exchange. After speaking with market participants, the exchange made its decision: trading would be called off for the entire day.
Many in the market say they were relieved. A call to resume trading would have been chaotic, said one worker at a Tokyo-based brokerage, with no way to tell which existing client orders remained active, while also trying to process new asks and bids.
Technical discussion:
At 4:30 p.m. local time, four TSE executives, including Chief Executive Officer Koichiro Miyahara and Chief Information Officer Ryusuke Yokoyama, faced journalists at the exchange to explain the outage. In a briefing that lasted about 100 minutes, they bowed in apology in front of the crowded room before going into a detailed technical discussion of the breakdown.
If the bourse was criticized for its communications earlier in the day, it won praise for how it handled the press conference. The executives answered questions from the media with relative ease, discussing areas such as systems architecture in highly technical terms. They also squarely accepted responsibility for the incident, rather than trying to deflect blame onto the system vendor Fujitsu Ltd. It bore little resemblance to gaffe-filled briefings by other Japanese firms in the past. On Twitter, the Japanese public voiced its approval.
Megumi Takarada, a senior analyst at Toyo Securities Co. in Tokyo said, "Management explained very clearly during the briefing last night. The briefing provided some reassurance that management clearly understands the issue.”
Later in the evening, the announcement came that the bourse would restart trading Friday. While that passed without issue, many questions remain unanswered. One of the biggest is whether the same kind of hardware-driven failure could happen in other stock markets. For one strategist, it almost certainly could — but that’s not something to worry too much about.
“There’s nothing uniquely Japanese about this,” said Nicholas Smith of CLSA Ltd. in Tokyo. “I think we’ve just got to put that in the box of ‘stuff happens.’ These things happen. They shouldn’t, but they do.” – Bloomberg.
