Building LLVM using Shake

Summary: You can now build LLVM using Shake, and a rebuild with nothing to do goes massively faster than make (0.8s vs 199s) and fractionally faster than Ninja (0.8s vs 0.9s).

As of Shake 0.10.4 the shake tool can execute Ninja build files. LLVM can be built with CMake, and CMake can generate a Ninja build file, so you can compile LLVM with Shake. I've included the full steps I followed at the end of this post.

The main thing I wanted to test was how fast a rebuild with nothing to do was using Shake vs Ninja, as Ninja prides itself on having "a focus on speed". When compiling LLVM on Windows with GCC, a nothing to do build using make takes 199s, Shake takes 0.8s and Ninja takes 0.9s. The CMake generator does not use one of the latest Ninja build features (the deps keyword), but if it did, Shake would be about 0.1s faster and Ninja would be at least 0.1s faster.

Full builds with Shake and Ninja both take about the same time, but with anything higher than 2 CPUs the linker phase ends up contending heavily and the machine thrashes the disk, making robust measurements impossible. The solution would be to use finite resources on the linkers, something that needs implementing in the CMake Ninja generator, and would then allow more CPUs to be used.

Other than speed, why would you use Shake to compile LLVM?

• If you build with --report the file report.html will be generated. Open that report file and you can see numerous details about the build - how good the parallel utilisation was, what changed to cause what to rebuild, summary statistics, a dependency graph and more. See the Help page in any generated report for more details.
• If you build with --progress the console titlebar will display a predicted completion time, how many seconds until your build completes. The predicted time will be fairly inaccurate the first time round, but future runs are influenced by recorded timings, and can produce useful guesses.
• If your CPU has a preference for functional languages it will make the registers happier.

Existing Ninja users may also be interested in a guide to running Ninja builds with Shake, which gives a few more details on using Shake like Ninja.

Compiling LLVM with Shake

These instructions are how I compiled LLVM with Shake, on Windows, with GCC. I didn't run into any significant problems, but there were two minor niggles I had to work though (both listed below). I compiled LLVM with make, then Ninja, then Shake, to check each phase as I went - but only the final Shake compile is actually necessary.

• Install Shake with cabal update && cabal install shake --global, if you are new to Haskell package installation, see here.
• Get LLVM and compile it with make, I followed the instructions at http://bencode.net/clangonwindows, which has disappeared in the last few days (I have emailed the web master to see where it went).
• Install Ninja.
• Run CMake over LLVM like this, configuring with -G Ninja.
• To build with Ninja I had to edit build.ninja line 17697 to delete lib/clang/3.4/lib/windows/libclang_rt.i386.a, which won't build on my system and isn't built at all by the make system - I suspect this is a tip/mingw issue. At this stage you can compile LLVM with Ninja.
• Type touch tools/clang/lib/Basic/CMakeFiles/clang_revision_tag to create a dummy file. There is a Ninja rule to create such a file, but the rule is wrong since it doesn't actually produce the file, and Shake's sanity checking spots that.
• Run shake -j2 in the build directory. Come back later and you will have a build.
• Run shake -j2 again to enjoy the fast nothing to do build.