Monday, February 24, 2014

The Build System Shootout

Summary: I am collecting examples to show how build systems relate in terms of power. Contributions welcome.

A while back I tried to classify the relative power of various build systems. I did that by reasoned argument, which, unsurprisingly, meant I didn't get it right. In an attempt to properly clarify the relative power of various build systems I've started the Build System Shootout. Compared to the Computer Language Shootout, this Shootout attempts to answer whether a build system is capable of expressing a particular dependency structure, but does not measure performance. The following build systems have at least seven entries:

  • Make (GNU version), the de facto standard build system.
  • Ninja, as used by Chrome.
  • Shake, my Haskell build system.
  • tup, contributed by Andrew Wagner.
  • fabricate, with help from the fabricate mailing list.

The project currently consists of twelves examples, each of which has pseudo-code for the equivalent untracked straight-line shell script and a set of test cases that are used to decide if a fragment correctly expresses the build example.

I welcome contributions, including:

  • Examples in different build systems (e.g. Scons, Ant, Waf).
  • New implementations for existing build systems (e.g. parallelism for fabricate).
  • New test cases (provided they show something interesting) (e.g. dependencies by hash not modification time).
  • Corrections of my egregious errors.

4 comments:

Anonymous said...

Good luck. You've got a lot of work ahead of yourself.

spaces-make:

output\ file: input\ file
cp input\ file output\ file

Neil Mitchell said...

Anon: Thanks! I've updated the example.

Michal said...

I hoped for some performance tests.

Build systems require scalability, why not try on Linux kernel codebase?

Neil Mitchell said...

Michal: I agree, performance tests would be a great complement, and would definitely be useful. Unfortunately they are also much harder to write (harder to figure out the rules, and a lot of work) and harder to run (require lots of CPU). Since both are needed, I tried to tackle the easier one first.

As a start to performance tests, I have been benchmarking Ninja (which has a focus on speed, and intends to be the fastest built system) against Shake (which so far, seems to be mostly a bit faster than Ninja for most interesting workloads), but even that narrow comparison is fraught with disclaimers (I'll be blogging more about it in a week or so).