The Elephant in the Room

the back end of an elephant trying to fit through a narrow opening

Dear KV,

While working with our data science team, I began to notice something peculiar about their code, which is a combination of Python and C/C++ libraries as you find in iPython or Jupyter notebooks. The amount of code in their programs that is responsible for getting data to where they can work on it almost always overwhelms the amount of code that operates on the data itself. And this intellectual overload, which they rightly think they should not worry about, is a drag on their overall productivity. If they had a way to just operate on the data rather than fooling around with finding, opening, reading, writing, and closing files, to take one example, let alone managing their program’s memory when they delve into C and C++ libraries, it seems it would suit them a lot better. When I talk to the team, they just say they accept this situation, because their two choices seem to be either using a framework that’s so high-level that the performance is poor, or using a system with all the lower-level knobs exposed that—while fast—is error-prone and exposes them to a lot of the system plumbing. Surely there is a better middle ground somewhere?

All Plumbed Out

Dear Plumbed,

You ask a deep question: “Why do our programs take the shape in which we see them?” KV often waxes historical and will, alas, do so again. Software and hardware have followed a random walk over the history of computing. Hardware often leads and software often follows, although from time to time, software has forced itself upon the hardware side to get new ideas to execute at the speed of the hardware. Consider the history of Virtual Memory and Memory Management Units, or RISC processors that are meant to run that ever-popular programming language, C, as fast as possible.

The plethora of plumbing you see in most programs is due, in part, to an elephant that has been sitting on our collective backs for nearly 40 years: Posix, which is just a nice way of saying Unix, and even though they disavow it, Linux. Only two models of programming are in common use by anyone not working inside another program (did you know people think programming in Excel is programming?) and these are Posix/Unix/Linux and Windows, at least for most programmers. All these systems grew up in an era of hardware that was very different from our own. This is evidenced in the way that programmers are forced to interact with them. Let’s consider two examples: getting at data and sharing data between programs or threads.

The point of a program is to transmute data between states, a fact often lost when a programmer comes to write the code, because first, as you point out, they must deal with the plumbing. Current systems were developed in an era where secondary storage, aka spinning rust, was often provided in disk drives the size of a washing machine and was larger by several orders of magnitude than the system’s main memory.

The secondary storage was also slow and unreliable. Unix, which eventually became codified as Posix, created a huge amount of plumbing around secondary storage to make it more reliable and aid the programmer in finding just what it was they had stored in the first place. Lest you think that the designers were trying to make things more complex, you are quite wrong. The systems for handling secondary storage that preceded the ^*NIX age were complex and not portable, so the changes were most welcome, but once Posix had established itself in the industry, it acted as a huge elephant that prevented innovation and progress.

The classic example of this is embedded systems software in which there are many and varied kernels, some of which provide truly innovative features not found in typical desktop or server systems. The death of any of these systems is when a user asks, “What about Posix? How can I port my XXX program to run on your system?” Providing Posix-like semantics in these systems is their death knell, because the Posix way of thinking is so narrow and providing its varied illusions requires so many hidden changes and adherences to age-old assumptions that there is no way to have a flexible innovative system and serve the Posix elephant. This means that no matter how innovative and clever a system is, once it is tainted by Posix support, it becomes just another Posix system. And then we have the same plumbing, provided in the same way as before, and the innovative bits of allowing the programmer new ways to transmute their data gets lost in the noise.

The second example of how software plumbing has not kept up with the times is the way programs work with shared memory. If you look at most interprocess communication mechanisms, you can see that they are most often used by only two programs—usually a client and server. The Socket API, local IPC, and shared memory pretty much assume two programs, and the addition of threads to these programs is fraught with peril and often poorly provided for by programming languages. A two- (or small number-) process model of working with shared memory makes perfect sense when there is only one CPU in each computer, but the days of single-core computers went out with cellphones the size of bricks.

Once Posix had established itself in the industry, it acted as a huge elephant that prevented innovation and progress.

You would be hard-pressed to find a computer (and definitely not a server) in which there were not tens of CPUs. And it is not as if we were not all warned about this explosion of cores. In fact, this very magazine published about the coming of multicore systems nearly 20 years ago, and yet our way of providing programming APIs for this new world is based on APIs developed when computers rarely had more than one CPU. The pthreads API (part of Posix) remains the most common way for programmers to write code that can take advantage of current multicore designs—an API so difficult to use papers have been written about how no programmer should ever try to use it.

Most attempts to handle the plumbing problem have followed the age-old software paradigm of adding yet another abstraction. Hiding the plumbing is good, but once your toilet overflows for the 10^th time, maybe it’s time to change the plumbing rather than buy a longer snake.

What is required to get away from this plumbing problem is to rethink how programs and data interact in modern systems; a topic that Poul-Henning Kamp, a frequent contributor to Communications, covered in a talk more than a decade ago about how we should stop programming computers as if they were PDP-11s. If we look at a modern computer, it has several features that are not covered in software design classes, most of which still assume the single-core, small-RAM, big-disk model. Now, all computers have many cores. Most computers have main memories that dwarf the secondary storage of the systems used when your OS textbooks were written. The secondary storage is largely reliable and does not depend on flying heads. In such a world, we should be able to think about programming for our data rather than programming for our plumbing.

We should be able to assume that the data we want exists in main memory without having to keep telling the system to load more of it. APIs for managing threads and access to shared memory should be rethought with defaults created for manycore systems, and new schedulers have to be built to handle the fact that memory is not all one thing. Modern systems have fast caches near CPU, main memory, and flash memory. Soon we will have even more memory, disaggregated, which is faster than disk but slower than main RAM.

If we are to write programs for such machines, it is imperative to get the Posix elephant off our necks and create systems that express in software the richness of modern hardware. Only then will programs be about data, which they should be, and not about the plumbing to get at the data.