How Software Stifles Competition and Innovation

Innovation is not what it used to be, and software is part of the reason. In many industries—industries well beyond Big Tech—dominant firms have built large software-based platforms delivering important consumer benefits, but these platforms also slow the rise of innovative rivals, including productive startups.⁵ Because access to these platforms is limited, competition has been constrained, creating a troubling market dynamic that slows economic growth.

To be sure, the advanced economies of the world continue to generate new ideas at an impressive pace. The numbers of scientific publications and the numbers of patents are growing robustly, although these indices are not necessarily good measures of the economic importance of new ideas. Perhaps more economically relevant, entrepreneurs are creating technology startups at a healthy rate, down a bit from the heady years of the dotcom boom, but robust, nevertheless. Tech startups have been a bellwether of industrial dynamism. In the classic Schumpeterian story, innovative startups enter an industry and grow rapidly until they “disrupt” incumbent industry leaders, thus bringing more productive technology to society. While tech startups are still being created at a healthy rate, it is the latter part of this story that shows troubling signs. Innovative startups are not growing as fast as they used to, they are not disrupting industries as they used to, and productivity growth is slower than in the past as a result.

Back to Top

Slower Innovation

Several measures show the slower growth of startups. The time required for startups to receive venture funding has lengthened substantially.¹⁶ The median time from the founding of a startup to the time it receives seed round funding grew from 0.9 years in 2006 to 2.5 years in 2020. The median time to late-stage venture funding rose from 6.8 years to 8.1 years. Moreover, this slowdown occurred despite an ample increase in the amount of venture funding available.¹⁶ For firms that were acquired, the average time from first financing to acquisition tripled from a little more than two years in 2000 to 6.3 years in 2018.¹⁴ For firms that went public, the comparable times rose similarly. A study of high-quality tech startups found that after 2000, they were less likely to grow sufficiently for a high-value acquisition or IPO.¹¹

But the clearest measure of the pace of innovation is how fast productive firms grow. Productivity is a measure of the goods and services produced relative to inputs such as labor and machinery. An innovative firm is more productive because it produces better-quality goods and/or produces them at lower cost. When the economy as a whole is more productive, there are more goods to be shared, raising average incomes. Thus, the rate of productivity growth measures the rate of the entire innovation process. And, of much concern to economists, that rate has substantially slowed in recent years.¹⁷ A key component of aggregate productivity growth is the growth of innovative firms. On this front the news is troubling. After 2000, firms with a given level of productivity grew only half as fast as firms with that same level of productivity grew in the 1980s and 1990s on average. Economists using U.S. Census microdata find this slower growth of productive firms, especially startups, is a major cause of a slowdown in aggregate productivity growth.^9,10

So, the early stages of the innovation process are alive and well—new ideas are developed, startups are created. For example, we see plenty of startups with potentially disruptive ideas in fin-tech, healthcare, and medical devices. However, it is the later stages—where new ideas are adopted and used to boost productivity—that have slowed substantially, and this is harder to see. In fact, the actual rate of disruption has been declining in most industries, measured as the probability that a firm in the top four of its industry drops out of the top four. During the 1990s, the four-year disruption rate for banks was 14.9%; that fell in the 2010s to 10.1%. Indeed, fintech firms have not become industry disruptors partly because the large banks have made it difficult for consumers to transfer their data. Similar declines in disruption rates are seen in healthcare, medical devices, insurance, and in many other industries. And there is a corresponding growth in the market shares of the top firms. In banking, the top four firms held 14% of the market in 1997; now they hold 20% and similar changes are also seen in most industries.

Several measures show the slower growth of startups.

Moreover, researchers have tied these changes to investments made by dominant firms in “proprietary software,” software they develop for their own use.^{2,3,4,6,7,8,13} These investments are correlated with slower growth of smaller productive firms in the same industry. The overall rate of disruption is half of what it was in 2000 and the decline is accounted for by the slower growth of innovators less able to leapfrog industry leaders.⁵ And own-account software has also been tied to growth in the market shares of dominant firms that has been seen across industries.^2,3 So, the slower growth of innovative firms, the faster growth of market-dominating firms, and the decline in disruption are closely related to each other and to the use of proprietary software. Furthermore, the link between proprietary software and the rising dominance of large firms has been shown to be plausibly causal using econometric method.⁷ And the slower growth of innovative firms has been shown to account for a substantial decline in innovation measured as productivity growth.

Back to Top

New Software-Enabled Paradigm

Why is software playing this role? After all, software has long been seen as a disrupting force. Prepackaged software and inexpensive computing as well as SaaS have put tremendous capabilities in the hands of many startups and productive small firms. The PC revolution and the Internet powered waves of disruption where industry leaders have been dethroned. But now something different is happening. Firms, mainly large firms, are making huge investments on internal systems, both developing their own software and contracting outside developers. They are spending approximately $240 billion per year on this software. And these systems typically also include hardware, extensive data acquisition, AI, and new forms of organizing work. Indeed, it is the combination of all of these elements that make these systems difficult to imitate even though some components might be available on the cloud.

Modular software allows firms to customize product features and selection to better meet consumer needs.

What do these systems do? The common thread running through them is that these systems manage complexity. Walmart systems manage more than 100,000 items per store—far more than traditional department stores, they adjust inventory to respond quickly to changing consumer demand, and they use advanced logistics to get these items to the stores quickly and efficiently. Amazon, of course, handles an order of magnitude greater selection. Big banks use extensive consumer transaction data to tailor offers of credit card and home equity lines to individual borrowers and to target the marketing to them. And Google and Facebook provide data-driven targeting of Internet advertising. Automobiles and airplanes are now built on large software systems that provide unprecedented features and functions, including many subtle changes in performance. And so on for insurance companies, pharmacy benefit managers, and even waste-disposal companies.

These systems deliver substantial competitive advantage because they allow firms to better meet the heterogeneous and changing needs of consumers. This development stands in sharp contrast to the mass production paradigm that emerged during the late 19^th and early 20^th centuries. Then, by standardizing products in industries such as steel, automobiles, and petroleum, new technology enabled manufacturers to produce at large scale, sharply cutting costs. But because the products had to be standardized to benefit from economies of scale, not all customer needs could be met.

Under the new paradigm, firms using advanced information technology can achieve a degree of “mass customization” that far better meets consumers’ idiosyncratic needs and wants. This advantage arises because modular software is extensible—that is, compared to hardware-based systems, a software-based system can more readily add a new product to store selection, add a new feature to a product, or rebundle features to create new products. While software engineers have long understood the benefits of modularity, its economic significance is only now becoming apparent. Walmart’s systems enabled it to better anticipate the needs of shoppers, disrupting Sears and Kmart in the 1990s, emerging as the dominant general merchandise retailer in the U.S. since then. Four big banks used their systems to expand credit lines to thousands of consumers, edging out hundreds of smaller banks to dominate the credit card industry. And so on.

Modular software allows firms to customize product features and selection to better meet consumer needs. That is good news and it deserves to be celebrated. But it also means the firms with these software-enabled platforms can dominate their industries in ways that slow the growth of innovative rivals, bringing down aggregate productivity growth. A startup might have a truly innovative technology, but it may nevertheless struggle to grow if it lacks the platform needed to market many products with many features to compete with dominant firms.

Back to Top

The Challenge

The fault here does not lie in the software itself, but rather in the way firms are using the software. Ideally, we want more firms, including startups, to be able to better meet the needs of their customers. The problem is limited access to the technology. Critically, it is difficult for startups to gain access to these platform technologies and associated data. In the past, when dominant firms have developed advanced technologies, they licensed them to other firms, or the innovations “diffused” as key personnel left to form spinouts or other researchers independently developed alternatives. But evidence shows this process is happening at a much slower pace than in the past, leaving a growing gap between the dominant firms and the rest.^1,5 The very complexity of these platforms makes them difficult to imitate or to create independently. The platforms companies are also adept at “locking in” customers, making it difficult for customers to switch to newcomer firms.

The fault here does not lie in the software itself, but rather in the way firms are using the software.

But we are also seeing evidence of an emerging solution: open platforms. Dominant firms may choose to open their internal platforms to the public, for a fee. For example, Amazon unbundled its internal IT platform to create AWS, the first cloud service. The cloud enables startup firms to access advanced IT infrastructure, removing an obstacle to their growth. While there may be downsides to the cloud—small firms may become “locked in” to a cloud provider—research shows that overall it promotes the growth of productive startups.¹⁵

Back to Top

Conclusion

A new generation of software platforms has allowed large firms to increase their domination over markets, impeding the growth of innovative startups and thereby reducing aggregate productivity growth. Of course, other factors affect startup growth, such as difficulty hiring talented workers, something made worse by the spread of non-compete agreements. But substantial evidence points to a major role for software in rising large firm dominance/slowing startup growth.

This critical problem will abate if large firms continue to open up their proprietary platforms as Amazon has done with AWS. The highly profitable AWS stands as a model encouraging other firms to follow suit.¹² Yet the decision to open up can be difficult, trading known competitive advantage for the uncertain rewards of a larger market. Perhaps government policy can encourage or compel companies to share data or software in some circumstances. Better policy will, however, take years to develop. Nevertheless, it is key to recognize software/big data/AI have created a new economic paradigm promising great benefit well beyond the Big Tech firms, but which also creates major social and economic challenges.