Kode Vicious: System Changes and Side Effects

Dear KV,

I maintain a legacy system at work that uses Makefiles for building the code. Recently my company switched to a different system—SCons—for doing software builds and now I'm being asked to update our legacy code to build with this system. Personally, I don't see that we will get any benefit switching a legacy system's build strategy, but it's an order from management so I will likely have no choice. Why would anyone bother to switch a working build system to a newer one? Are build systems really that different? Is there really any innovation here that could matter?

All Built up

Dear Built up,

Innovation, as someone should have said, is in the eye, or perhaps hands, of the beholder. KV rarely condones change for its own sake: the change has to have some appreciable benefit to those who are working with the system. As this relates to build systems, there are few things more important to a programmer's work flow and therefore productivity than how their system is built. A poorly implemented build system will impede progress and lead to wasteful programmer downtime, mostly made up of programmers playing games in the hallway, buying questionable items on the Web, and learning yo-yo tricks. All fine pursuits of course—I learned at least five yo-yo tricks on one job when it took 20 minutes to build a piece of software, and the build locked up the machine we all used for our work. Of course such crippling times are in the past now, with the advent of speedy CPUs, memory, and disks there is no longer any wasted programmer time, and we can all depend on a quick compile/link phase. Uh, perhaps not.

There are several reasons why people switch build systems. I think the most common one is that people hate and fear Makefiles. Other than the misnamed autotools there is probably no more maligned component in a build system, and with good reason. While a simple Makefile is easy to read, a large software system requires a set of interlocking, and usually undocumented, Makefiles that can be as difficult to understand as modern architecture, and quite often is just as pretty to look at. Pull out just one piece and the whole edifice will come crashing down upon your head. The complexity of a system of Makefiles does not, normally, come from some perverse turn of mind of the authors, but is in fact due to three problems. The first is that dependency analysis, which is what the make program does, is a non-trivial exercise; the second is that Makefiles and build systems are generally not designed, they are accreted; and the third is that the build system is usually stretched to do all kinds of things that were not originally foreseen.

In the beginning was the Makefile, and it was good. It contained at most a few targets and a handful of source files. The rules to determine what should be built and when were straightforward and all was right with "make world." Over time all software systems grow and so the Makefile becomes the root of a tree of Makefiles that are spread across a set of directories. Next the include directive is used, and then chaos ensues. The tree that was rooted in the original Makefile becomes a set of creeping vines, with a strong resemblance to poison ivy because no one wants to touch it.

A typical build system not only takes source files and turns them into binary programs, but it may also be co-opted to produce documentation from the source code, perform pre-checkin testing, and so forth. Every time you add a new responsibility to the system you must tweak it in some way, which usually has unintended side effects. Eventually the Makefiles become so complex that they're impossible for new users to modify, and even experienced users make mistakes because modifications are likely to be infrequent. Just as with code, engineers rarely document their Makefiles, as a matter of fact they're likely never documented, even if the code is. The most common documentation for a build system consists of the rather unhelpful "type make<rtn>," which works in the typical case but gives no information as to how to debug a problem in the make system.

Every time you add a new responsibility to the system you must tweak it in some way, which usually has unintended side effects.

Speaking of debugging, this is one of the places where the make program fails miserably. While it does have a command line flag to say "don't do what I say, just show me what you'd do" as well as another command line flag to show debugging information, the output is usually so voluminous that it is unusable, and you wind up deleting and commenting lines in the Makefile until your bug is hidden, because removing lines doesn't actually make the bug go away. Debugging Makefiles can make you tear your hair out, and for what, you probably just wanted to add a build option or a new file to your system and now you've wasted two hours with your Makefiles. So, Makefiles are ugly and make people nervous and they tend to start scratching themselves absent-mindedly if you ask them to fix them.

If you think make is bad, well, SCons is certainly more interesting. It seems that what make lacked was a scripting language, since we all know that adding a language to a system will make it easier to use! SCons is written in Python and Python is the language in which its SConstruct files—the equivalent of Makefiles—are written. Much like make if you want to do something simple with SCons it's pretty easy to do so. If you thought that a complex set of Makefiles was difficult to understand, try reading a complex set of SConstruct files. You'll not only need to load up the mental context for your project, you'll also need to load your brain with Python. Now, don't get me wrong, Python is a fine thing, I code in it frequently, but I don't want to have to hack Python to get my software builds going. Having recently been exposed to SCons I can say that I don't prefer it over make. Although the system I was working with had supposedly been extended to build code with profiling turned on that extension was clearly not working. When I asked another engineer how he built the system with profiling his reply was, "I just hack it." "OK," I thought to myself, "let's see if I can just fix this." After all, I was going to need profiled code for quite a while and it seemed dumb of me to check out an extra copy of our tree just to have what should be possible with a compiler flag. Had I known what lay ahead, and were I a less stubborn engineer, I would have checked out another copy of the tree. I was determined to "do things right" and therein lies the road to madness. It took about four hours to load up the proper mental context, and to understand the interlocking SConstruct files. After many detours and chasing several wild geese, I was able to get my profile flags correctly into the build system. Strangely enough the number of lines I changed was less than 10, but finding which 10 lines was the interesting part. If I had not already been a Python programmer it would have probably taken days to get this to work.

Don't get me wrong, SCons certainly has some interesting features, including built-in support for object caching, but I would be hard-pressed to call it an improvement over make. My advice to you is to stick with what you have, that is if you can, in particular since you say the system is a legacy system. And my advice to those who wish to make my life easier by changing basic tools is to study the existing tools very carefully and to figure out which changes really help and which changes are just there for the sake of change.

KV