Linking to the manual page which doesn’t actually require any Haskell knowledge.
There’s also a conference paper which does require Haskell knowledge.
Linking to the manual page which doesn’t actually require any Haskell knowledge.
There’s also a conference paper which does require Haskell knowledge.
I learned about Shake from the Build System a la carte paper. Most notably, dependencies in Shake are not declared statically but can be built fully dynamically as part of the build process, which is pretty rare and makes it really flexible.
Edit: after reading Shake’s paper I realized that Redo, which I wasn’t able to grok before, is based on the same idea, although the implementation is vastly different.
Build System a la carte is such a great read for anyone who has built anything remotely close to a build system.
IMO that paper seems to undersell Tup. They do mention a couple of its features, but to me, the most important feature is that it is the only general-purpose build system that I know of that is able to (and always does) detect undeclared dependencies with file-level granularity. That is, it will error out when it detects that some build step is reading or writing a file that it wasn’t supposed to, regardless of which compiler (or any other custom command) you are using as your build step. This makes builds race-free.
This feature works without needing language-specific support which, in other build tools, can be a pain if you have a project using multiple languages or using potentially-racy custom build commands. In my experience, this makes builds far more reliable than they tend to be compared to other general-purpose build systems, especially for massively parallel builds. Another scenario where this helps is when you rename or move a file and forget to update the build rule dependencies, which can also lead to silent race conditions with some build tools.
Tup is also quite fast and its ability to use Lua as the language for specifying the build steps also makes it quite flexible, although other modern build systems can be comparable in these regards.
Shake can integrate with my https://github.com/jacereda/fsatrace to automatically compute dependencies.
That’s good to know, I wasn’t aware of that!
On Linux, Tup transparently uses FUSE to intercept filesystem accesses, so it also works for statically linked binaries, such as Go programs.
What about Bazel? That is one of its main features – the BUILD files declare deps exactly and there are a bunch of sandboxes on each platform that enforce it, and it is highly parallel
This is one of the things I want to add to my Ninja-based system
But the reason it’s fiddly / not more common is there’s no way to implement this with portable file system APIs – e.g. you need a bunch of highly OS-specific sandboxes, or FUSE, as you mention
It seems to rely on language-specific integrations, and has specific knowledge and rules you must follow about building binaries vs libraries vs tests, etc. So it doesn’t seem as general-purpose of a build system to me.
Apart from that, I don’t like that it seems to require you to use their own custom language (if you want to extend it, at least). It also seems way too complex and tries to do way too many things for my taste; I’d rather have simpler tools that can be debugged more easily when things go wrong. But of course, to each his own :)
Bazel is definitely general purpose, and it has the feature you want – it finds undeclared dependencies reliably, and the resulting build is correct and parallel.
Personally I don’t expect to ever use it again, or really want to – it’s more geared toward big monorepo environments – but it does have a bunch of good features.
The extension language is basically Python – conceptually it’s like Scons or Meson or many other build systems.
The author if Shake is also the author of Buck2, which is way nicer to use than Bazel, and both share the same configuration language.
For reference:
Shake is great. I wrote a static site generator over it, and I wrote a post about it.