As of July 2014, Facebook, founded in 2004, is in the top 20 of the most valuable companies in the S&P 500, putting the 10-year-old software company in the same league as IBM, Oracle, and Coca-Cola.3 Of the top five fastest-growing companies with regard to market capitalization in 2014 (Table 1), three are software companies: Apple, Google, and Microsoft (in fact, one could argue that Intel is also driven by software, making it four out of five).
Hot and upcoming companies such as Uber, Tesla, and Airbnb, which are disrupting the traditional taxi, car, and hotel markets, respectively, are also all fundamentally software companies.
Conversely, the bottom five companies, which lost market capitalization in the first half of 2014 (Table 2), are mostly traditional enterprises in business for decades.
Given this information, the logical question to ask is why are software-based companies taking over the world?2 The answer is simply that powering companies by software allows them to be responsive and data driven and, hence, able to react to changes quickly. To explain this, let's take an informal look at control theory.
In control theory, an open-loop (no-feedback) control system computes the control input to the system using only the external input, but without taking into account the current output of the system (Figure 1). An open-loop control system works well with full knowledge of a static world, but it falls apart when the environment evolves, or when there is no perfect model of the system under control. An example of an open-loop system is the cab driver who after every trip returns to the same hotel to hang out with his fellow cabbies and smoke a few cigarettes, with a small chance of picking up new customers, and without taking into account that theater performances downtown have just finished and thus demand is elsewhere.
A closed-loop control system takes into account the current output of the system and feeds this information back to the controller, which combines these measurements with the current external input to continuously correct and optimize the system under control (Figure 2). Uber, for example, knows the current traffic conditions, the exact supply and demand for transportation at any instant, and the historical price elasticity of customers; hence, it can optimize pricing for taxi rides in real time and direct drivers to the locations with the highest demand for transportation.
In Microsoft Office, new features were always introduced and tested via extensive user studiesuntil Office 2007 when the existing user interface was replaced by a "ribbon," to the dismay of many users. When the start button disappeared in Windows 8, every Windows user in the world lost their accumulated muscle memory for commanding the Windows operating system. Such missteps occur when companies grow older and become deaf to feedback and external input. Companies that ignore feedback become no-control systems that produce output based only on their internal echo chambers (Figure 3).
Internal processes and the desire to control risk start to overrule real innovation. Investors' pressure to optimize for quarterly results and the bottom line further encourage deafness. Last but not least, the workforce ossifies because of a recursive feedback system that over time ejects employees that are open to external stimuli in favor of yes-men who worship legacy and process.
In the next decade every business will be digitized and effectively become a software company. Leveraging software, and, in general, computational thinking, to make a business responsive to change using a closed-loop feedback system will be crucial to surviving in this new world where business = data + algorithms. Some examples of responsive companies follow here.
Unlike traditional cab companies, Uber does not own its fleet. Its value is in the real-time data and algorithms Uber uses to transform this data into decisions. In a few years, Uber will likely automate away all humans in the equation by replacing drivers with self-driving cars, which themselves are possible only because of advances in software technology.
Netflix does not operate its own data centers but runs its whole operation on public cloud infrastructure. Simply put, cloud computing virtualizes hardware into software.5 Instead of dragging in a physical computer or storage device, cloud computing achieves the same effect with a few lines of software code that in the blink of an eye allows companies to add compute and storage capacity, or to release surplus.
Software promotes agility by dramatically speeding up the feedback loop between output and input. In the past, companies could measure their performance every quarter, making it difficult to adjust quickly to changes in the environment. In contrast, Facebook ships new versions of its product multiple times a day, with enhancements and fixes determined by real-time feedback from actual use of the website. Companies such as Amazon and Booking.com continuously perform A/B testing or multi-armed bandit experiments on users to optimize purchase rates on their sites.
Amazon even makes its A/B testing technology available to third-party developers for free (https://developer.amazon.com/public/apis/manage/abtesting). The same holds for other companies such as Facebook and Netflix, which make most of their internal software available as open source.
Continuous delivery and A/B or multi-armed bandit testing not only require new products to be rolled out quickly, but also mean that wrong decisions can be rolled back quickly. Accepting that decisions can be wrong, and acknowledging them, requires courage that traditional risk-averse enterprises tend to eliminate by introducing process.
Responsive enterprises accept that failures will always happen and guard themselves from cascading failures by purposefully causing failures. Just the thought of creating a deliberate failure to make a system more robust against failure immediately sends many traditional middle managers over the edge. At Netflix, however, it is common practice to "let an army of monkeys loose" in the data center to pull out virtual cables and wreak havoc. Many agile development methods propose writing tests before writing code. It is impossible, however, to faithfully model the complexities of the environment in which deployed code runs in the test environment. Instead, software application failures should be treated as any other failures, deploying code in production immediately and rolling it back when problems occur.
Elementary psychology teaches us that in order for humans to learn (that is, improve), feedback about their actions must be immediate. Smokers keep smoking because lighting up gives them immediate satisfaction, but the feedback effect of lung disease comes years later. Developers are humans, too, and, hence, can be stimulated to write better software by providing them immediate and physical feedback about the quality of their code. Companies can implement this feedback loop by making developers wear pagers that wake them up in the middle of the night in case problems arise and by making them responsible for the "whole stack" (also known as the DevOps model). Decoupling the development process from the operations side removes a valuable (recursive) feedback loop in the system and thus reduces the responsiveness of the system as a whole.
Another way of viewing a closed-loop system is as a (nondeterministic) Mealy machine.7 This is a finite-state machine where the next output value and the next state are determined by the current state and the current input (Figure 4). The Mealy machine view emphasizes an organization's "corporate memory" and posits that decisions on the current input may be influenced by a state that has been accumulated over many iterations using past inputs and outputs. At a meta level, however, the environment in which the enterprise operates will also change over time; hence, the corporate memory needs to be "reset" occasionally since it is not tracking relevant inputs anymore.
As a concrete example, take Microsoft, which since 1975 has accumulated an impressive corporate memory of shipping boxed products. While the world was craving boxed software, the Microsoft machine was able to optimize the output to generate spectacular profits. The environment changed, however, and began demanding cloud services, and suddenly the knowledge of optimizing for boxed software became an impediment for growth. Microsoft has had to lay off thousands of people recently to "forget" the past and change to a "cloud first" direction.
Every company should have a "10th man" ("When nine people agree on something, it's the tenth man's responsibility to disagree no matter how improbable the idea")6 or devil's advocate who injects a certain amount of randomness and chaos into the process. This prevents falling into the trap of getting stuck in a local optimum, or worse, making wrong decisions because changes in customer preferences were ignored and nobody dared to point out that the emperor had no clothes.
From a people's perspective, running a company powered by software is totally different from running a traditional enterprise. Once a company realizes the road to success is to accept that a responsive enterprise is powered by software, it must apply open-loop feedback control to make the engine run smooth, and, moreover, it must deeply embrace developers as the engine for growth.
Software developers are nothing like traditional suit-wearing corporate employees, and it is this aspect that is often the most difficult for pointy-haired bosses to understand. Perhaps the difference is most succinctly expressed here: "You cannot race sheep, and you cannot herd racehorses;"8 and "Domesticate developers like beekeepers domesticate bees."1
The "Seven Aspects of our Culture," as described in a Netflix slide presentation (Values are what we Value; High Performance; Freedom and Responsibility; Context, not Control; Highly Aligned, Loosely Coupled; Pay Top of Market; and Promotions and Development) provides a crisp formulation of people-management values for a software-centric enterprise.4 Here is our take on four of Netflix's seven core values:
Context, not control. Anyone who has performed improvisational jazz or theater knows from experience that creativity can only flourish under strong constraints. Letting developers wander around without clear goals in the vastness of the software universe of all computable functions is one of the major factors why projects fail, not because of lack of process or planning. If we give developers strict guidelines and orders about what to optimize for, and provide them immediate and real-time feedback about the consequences of their decisions on those goals, the competitive nature of geeks will take over and the system will quickly iterate toward an optimal solution. The only goal of middle management is to provide and enforce this context, and keep developers happy and productive within these well-defined bounds. We do not care about developers' opinions on the long-term strategy of the enterprise.
Pay top of market. In professional sports it is normal to spend many millions of dollars to attract and pay top players to assemble a team that can compete in the top of the division. To attract and retain the best developers (and to entice young people to pursue a career in high-tech instead of becoming lawyers or stockbrokers), they need to be compensated well and given a big piece of the pie, based on the value they deliver and their limited shelf life, not on the number of hours they clock.
Highly aligned, loosely coupled. Replacing the word war with software in the following excerpt from the Fleet Marine Force Manual 1, Warfighting, immediately brings to mind the day-today practice of software development:9
"Software is a complex endeavor. It is shaped by the human will. It is characterized by friction, uncertainty, fluidity, danger and disorder. While the nature of software is constant, it remains unpredictable, and is affected by a mix of physical, moral and mental factors. While software has the characteristics of both art and science, it is primarily shaped by human experience."
The divisional organizational structure and operation of armies has been honed over many centuries to deal with such situations, and, hence, there is much to learn from the military. In particular, software development should follow the Philosophy of Command: "In order to support the fluid and chaotic nature of the battlefield, command must be decentralized. Subordinate leaders must use their own initiative to accomplish tasks which support their senior's intent."
Freedom and responsibility. Tradition, as defined by Wikipedia, "is a belief or behavior passed down within a group or society with symbolic meaning or special significance with origins in the past." In a responsive enterprise, traditions have no role. In the traditional enterprise, traditions are encoded as process. Process is what causes organizational deafness, which leads to the downfall of a no-feedback system. Blind adherence to process also drives out creative people and rewards nonproductive bean counters. Process is necessary to produce adequate results with mediocre employees in the fast-food industry. The contrapositive is that in a high-tech company composed of first-rate hackers, there is no need for process.
No company embodies the hacker culture better than Facebook.
No company embodies the hacker culture better than Facebook, and CEO and founder Mark Zuckerberg explained the idea eloquently in his letter to investors when Facebook filed for its IPO: "Hacker culture is also extremely open and meritocratic. Hackers believe that the best idea and implementation should always winnot the person who is best at lobbying for an idea or the person who manages the most people."10
Software is an unstoppable force. To survive, enterprises must ride the wave instead of resisting change and ending up in the killing fields along with blacksmiths and agile coaches. Here are some guidelines to keep in mind.
Sooner than you may think, every company will be a software company. The obvious way to run a software company is as a meta software application, recursively structured as a layer of commuting closed-loop feedback systems, using a strictly layered architecture modeled after the time-proven hierarchical structure of armies and applying software-inspired profiling and debugging techniques to optimize the profitability of the enterprise. On the operational side, instead of talking about code, companies should follow the "hacker way" and focus on writing code using continuous improvement and iteration, leveraging the best developers they can afford.
All Your Database Are Belong to Us
The Theft of Business Innovation
The Software Industry IS the Problem
1. Card, O.S. How software companies die. Windows Sources: 2008; http://www.netjeff.com/humor/item.cgi?file=DeveloperBees.
2. Denning, S. Why software is eating the world. Forbes (April 11, 2014); http://www.forbes.com/sites/stevedenning/2014/04/11/why-software-is-eating-the-world/.
3. Dogs of the Dow. Largest companies by market cap, 2014; http://www.dogsofthedow.com/largest-companies-by-market-cap.htm.
4. Hastings, R. Netflix culture: Freedom and responsibility, 2008; http://www.slideshare.net/reed2001/culture-1798664.
5. Helland, P. Condos and clouds. ACM Queue 10, 11 (Nov. 2012); http://queue.acm.org/detail.cfm?id=2398392.
6. Legge, A. What World War Z can teach you about critical thinking. Evidence Mag, 2014; http://evidencemag.com/world-war-z/.
8. Thomas, D. Developing expertise: Herding racehorses, racing sheep. InfoQueue, 2008; http://www.infoq.com/presentations/Developing-Expertise-Dave-Thomas.
9. U.S. Marine Corps. Warfighting. U.S. Government Printing Office, 1997; http://www.marines.mil/Portals/59/Publications/MCDP%201%20Warfighting.pdf.
10. Zuckerberg, M. Mark Zuckerberg's letter to investors: The hacker way. Reprinted in Wired (2012); http://www.wired.com/2012/02/zuck-letter/.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2014 ACM, Inc.
Wow, interesting article.
While I agree with the majority of what is written, I feel the need to point out two issues.
First, although relatively minor, in your section about context, not control, you conclude with "We do not care about developers' opinions on the long-term strategy of the enterprise." This seems extreme, and to at least slightly be in conflict with the immediately prior "keep developers happy and productive within these well-defined bounds." To extend your sports metaphor, do you really think you will get the best performance out of a team that feels it is headed in the wrong direction, and is guaranteed to lose? While you do not want each developer going off in their own direction, your developers should either have some belief in upcoming success, or belief in the capabilities of management. Doing otherwise is treating every developer as a short-term contractor with no real stake in the company.
Second, and more significant is the extreme ageism shown "Just like competitive athletes, they simply burn out by the time they reach their mid-30s.", and "their limited shelf life". This, in the same ACM issue as articles about "Pathways to Computing Careers", "The Whole Professional", and especially "Innovation and Inclusion". Certainly, that applies if the definition of "developer" is (illegally) restricted to "brogrammers", single males willing to spend their time only at work, and in competition with others of the same like - but isn't that a little extreme to advocate for all enterprises to switch to? E.g. "In the next decade every business will be digitized and effectively become a software company."
While there are many people who do tend to get set in their ways, and not make good software developers later in life, there are also those that can steadily innovate, and learn from mistakes rather than repeat them. Do we really need to aim for a world like Logan's Run, or promulgate the false Free Speech Movement ideology "Dont trust anyone over 30"? The Brain does not wear out from use, and if maximizing hours spent working is not the goal, such extreme burnout does not have to happen (at least beyond what a 6 or 8 week sabbatical, or assignment change could resolve).
Perhaps some people will accept restricting their contributions to complaining about "those darn kids", and "get off my lawn", but I for one will not.
Displaying 1 comment