This post is co-authored by Stephen Diamond and Jennifer Kuan. Jennifer is an economist based at the Stanford Institute for Economic Policy Research. The post is based in part on an event study we conducted on the impact of Reg. NMS. We will be presenting the paper at the meetings of both ISNIE (Duke) and SASE (Northwestern) this summer.

A firestorm erupted on Wall Street recently sparked by author Michael Lewis’ accusation that the stock markets are “rigged.” Mr. Lewis’ bases his claim on the allegedly manipulative behavior of so-called “high frequency traders,” or HFTs, in today’s financial markets.

Our own study of the changing structure of those markets over several years leads us to conclude Mr. Lewis is correct when he contends many investors trade at a disadvantage to HFTs. We found a significant widening of “spreads,” and therefore costs to investors, following rule changes by the SEC in 2007. Significant structural reform will be needed to restore transparency and fairness to our financial system.

While the issues at stake are complex, the heart of the matter is that HFTs have largely replaced stock exchange “specialists” as intermediaries between buyers and sellers of shares. HFTs trade large volumes of stock, so they claim to provide “liquidity” to the markets. This sounds reassuring to investors who think they can easily buy or sell at reliable and visible prices.

In fact, HFTs are largely free of the obligations and oversight once imposed on specialists by the New York Stock Exchange. HFTs are not mandated to maintain an orderly market like specialists and often disappear at the very moment they are so desperately needed. There is evidence this kind of behavior contributed to events like the “flash crash” of May 2010 as well as the failed IPO of Facebook in 2012.

Exchanges are now eager to profit from HFTs’ vast trading volumes so they help HFTs exploit advantages over other investors, allowing the use of complex and arguably manipulative order types as well as selling them access to data about other investors’ orders. Other enablers of HFTs include the telecommunications firms that allow the HFTs to engage in “fiber arbitrage” to gain privileged high-speed access to data and markets. HFTs use these advantages to move more quickly and flexibly than other investors and thus to trade ahead of ordinary investors at a profit.

The most important enabler, however, is the federal government itself. In 1975 Congress mandated the creation by the SEC of a “national market system.” Congress decided that if the SEC could create computer-based competition with the long dominant New York Stock Exchange’s manual trading floor then costs for the average investor would fall.

The SEC implemented a wave of new rules over the next thirty-five years that did, in fact, reduce trading costs. New electronic markets such as the Nasdaq now compete effectively with the NYSE. Smaller startup companies like Intel, Apple and Microsoft, which did not meet the stringent listing standards of the NYSE, were able to access investor capital on the Nasdaq.

But this was not enough for the SEC. Their goal was an end to the NYSE’s dominance of trading in blue chip firms listed on the NYSE. As the Charlie Sheen character Bud Fox would famously say in the film Wall Street, they were “going after the majors.” One backer of the new approach was Bernie Madoff, who led the automation of the Cincinnati Stock Exchange in the 1980s to draw trading volume away from the NYSE.

The NYSE and the large banks that dominated its board resisted these efforts for many years. But new demand for faster trades from institutional investors provided the political support the SEC needed to push through Regulation National Market System, or Reg. NMS, in 2007.

This was the straw that broke the camel’s back.

Until 2007, despite the earlier rule changes by the SEC, the NYSE still handled more than 80% of the trading volume of companies listed there. The NYSE was a monopoly but it stabilized price changes with narrow spreads using a self-regulatory framework crafted over its 200-year history.

Two features were key to that framework. First, because large underwriting firms wielded significant influence at the non-profit member-owned NYSE, they could and did impose stringent standards on firms that wanted to list their shares on the Exchange. Second, to attract investors to trade on the Exchange those same underwriters insured that floor brokers and specialists behaved fairly. The result was good quality information about listing firms as well as orderly pricing facilitated by specialists in both bull and bear markets.

But Reg. NMS uprooted that system. Brokers could now route their clients’ trades to any electronic venue even if it meant that the client did not get a better price available on the NYSE floor. As a result, the volume of NYSE shares traded off the NYSE exploded. The motivation to own the Exchange in order to attract investors with orderly prices was gone and the underwriters quickly sold the Exchange to public investors.

With stock prices no longer kept in check by the NYSE’s longstanding rules, our study found that spreads widened, volatility increased and the cost to the average investor went up. Congress had a useful idea in 1975 when it helped create a market for risky technology start-ups and other small firms. They need to step in again to deal with the unintended consequences of that important innovation.