‘Everything Fails All the Time’

Amazon vice president and chief technology officer Werner Vogels has kept the company at the forefront of scalable systems since he joined in 2004. Vogels was one of the main driving forces behind the architecture of Amazon Web Services (AWS). His insistent reminders that "everything fails all the time" have been hugely influential to the developers of high-availability systems. Here, he talks about continuing to innovate in database technology—and keeping up with customer needs.

You have followed an unconventional professional path, working at the Netherlands Cancer Institute before returning to school for a Ph.D. in computer science. What drew you to scalable systems?

Originally, computer science just seemed like a good career. In that era—the mid-1980s—there were many areas that were completely undiscovered, and distributed systems was one of them. I liked research, and it turned out I had a gift for it.

Let’s talk about Amazon Aurora, which recently won the 2019 ACM SIGMOD Systems Award. When did you first realize the time had come to reconceptualize the stack?

Our biggest goal with Aurora was to make life easier for our customers, and to do that, the database required a lot of modern innovations. Most database architectures, which were developed in the 1990s, don’t offer a good base for that—nor, even, do open-source databases like PostgreSQL and MySQL. We realized that if we could just rip the whole architecture apart and redesign it in a reliable, high-scale, cloud-native manner, we would suddenly be able to build all the functionalities that we wanted our customers to have.

---

"If we could just rip the whole architecture apart and redesign it in a reliable, high-scale, cloud-native manner, we would suddenly be able to build all the functionalities that we wanted our customers to have."

---

Such as?

Modern database users expect everything to be serverless and get seamless replication over at least three datacenters. They expect to be able to have instant backups and to be able to do instant schema changes without having to do complete table copies. We could only meet those requirements if we revamped how the database was built.

It must have been exciting to have an opportunity to rethink what a modern database should look like.

Building upon a db-log-aware storage service was a shift in mind-set, but we knew we had the engineers and the knowledge. Old-style database transactions are quite expensive, easily triggering 10–15 storage operations, because they were developed with an attached disk in mind.

AWS reinvented how companies access data storage and infrastructure—supported by a lot of education on your part. Is there anything you would have done differently, either in terms of developing AWS services or promoting them to customers?

Well, that’s a long list. The good thing is that we have a culture at Amazon that allows you to change your mind. One of the principles we had in the earlier days was that this should all be self-service. Customers should not have to talk to a human; they should be able to do everything by themselves. On the one hand, that removed a number of obstacles for people to start using our technology, because all you needed was a credit card and an email address. But over time, we discovered that enterprises as well as startups really want to have human contact. They need account management, professional services, and architectural support to help them build their systems. It’s something we didn’t appreciate enough in the early days.

What about in terms of technology?

There’s lots of things I would have done differently. For example, in the beginning, we combined identity and account. Those are two different things. Identity is a security component, and account is what you bill things to. If we had been smarter, we would have separated them, as we eventually did.

You have talked about the importance of giving developers operational responsibilities, and the Amazon product definition process of "working backwards" is also designed to incentivize people to take the customer’s view. What else have you learned about building and maintaining a product-centered culture?

The product nature is crucial to how we operate at Amazon. Things that are now common terms—service-oriented architecture, micro-services, DevOps, DevSecOps—were pioneered at Amazon before we had words for them.

That certainly goes for the operational side of things. If you really want your engineers and your product people to be in touch with your customers, there cannot be a layer of operations people between them. Our thinking has always been to make sure our engineers are in contact with their customers so that they can make decisions on their behalf. Take, for example, the AWS database groups. We have so many unique, purpose-built databases, and each of those groups is in contact with their most important customers. That creates a feedback loop that I have not seen anywhere else. In fact, more than 90% of all new features and services that we deliver are driven by customer requests.

---

"Over time, we discovered that enterprises as well as startups really want to have human contact. … It’s something we didn’t appreciate enough in the early days."

---

At the moment, AWS is quite a bit bigger than your competitors. How will you continue to differentiate yourselves in the future?

First, let me say that this is not going to be a winner-take-all market. There’s going to be a handful of global companies that are going to be really successful in this space, and that’s good, because customers need choices. The major thing that separates AWS from most other places is that we don’t follow our competitors. We focus on the feedback that our customers are giving us, and we’ll continue to go down that path—and at the same time, be a pioneer.

I’m reminded of the famous and probably apocryphal Henry Ford quote about how, if he’d asked for their feedback, his customers would have simply requested faster horses.

We are innovators in this space, and we also have to be conscious that we use our inventive brain power not just to fix a particular customer’s problem but, whenever possible, solve for issues that apply to many more businesses. That’s one of my major tasks, to listen to different types of customers and try to find the bigger patterns.

I imagine that Now Go Build, a video series in which you visit global entrepreneurs who use Amazon’s cloud-based technologies to do things like support Indonesian farmers and integrate refugees into the German workforce, has given you a nice window onto that.

Now Go Build is basically a recording of something that I’ve been doing for the past four or five years—meeting companies all over the world that have tremendous impact that AWS has helped them achieve. As technologists, we sometimes get so wrapped up in technology that we forget about the end-user stories. Most of these young businesses are solving really hard problems. They’re not hard problems in the technology sense, but they’re hard problems in the human sense.

I’m still so proud of the first episode that we did in Jakarta, where I met the founders of a start-up called Hara Token. These guys provide identity verification for the poorest farmers in Indonesia so they can get loans at regular banks instead of using loan sharks, who charge interest rates of 60%.

It must be a very interesting exercise—and very humbling.

The current general view of technology companies is that they focus on uninhibited growth or on profit. But these are entrepreneurs trying to build a business while also solving very difficult human problems.

In 2006, Jim Gray interviewed you in ACM Queue and noted how much it pains you to hear people describe Amazon as an online retailer, rather than a technology company. Now, I don’t think anyone would question your dominance in the tech world.

Jim was a mentor to me for a long time, and there are so many things in that interview that are still true today. His thinking has had a significant impact on technology architectures around the world, including at Amazon and AWS. I think we should continue to honor his memory and be grateful for all of the things that he impacted.

Back to Top