Compilation cache in opam

There is some work going on opam to add support for compilation cache. Compilation cache means instantaneous re-installation of packages that have been already installed previously, even if it was in another switch. It is a very convenient feature when working with local switches, git worktree and lock files.

If each git branch is in its own directory with its own opam switch, it might be very long to create a switch for each branch. It is possible to use opam switch link to reduce the number of actual switches. But it doesn't work when the branches are using different versions of the same packages.

Also there are some tools like utop, merlin, ocp-indent, ocp-index, that we want to install in basically every switch. And we don't want to spend a long time waiting for them to get compiled.

Those are situations where compilation cache shines. As long as a package has been installed once, installing it again is basically a no-op.

To enable the cache, we need to download this file: opam-bin-cache.sh. And to to make it executable.

The sandbox.sh hook is stored by opam in $HOME/.opam/opam-init/hooks. opam-bin-cache.sh can be moved there too.

curl "https://raw.githubusercontent.com/ocaml/opam/4756ca1870d65130d5b909309c2041ce05b06a4e/shell/opam-bin-cache.sh" -O
chmod +x opam-bin-cache.sh
mv opam-bin-cache.sh $HOME/.opam/opam-init/hooks/

Then the opam configuration file has to be modified. At the end of the configuration, there are 3 wrap commands using sandbox.sh. It looks like this:

wrap-build-commands:
 ["%{hooks}%/sandbox.sh" "build"] {os = "linux" | os = "macos"}
wrap-install-commands:
 ["%{hooks}%/sandbox.sh" "install"] {os = "linux" | os = "macos"}
wrap-remove-commands:
 ["%{hooks}%/sandbox.sh" "remove"] {os = "linux" | os = "macos"}

Those lines must be replaced by:

pre-install-commands:
  ["%{hooks}%/opam-bin-cache.sh" "restore" build-id name] {?build-id}
wrap-build-commands: [
  ["%{hooks}%/opam-bin-cache.sh" "wrap" build-id] {?build-id}
  ["%{hooks}%/sandbox.sh" "build"] {os = "linux"}
]
wrap-install-commands: [
  ["%{hooks}%/opam-bin-cache.sh" "wrap" build-id] {?build-id}
  ["%{hooks}%/sandbox.sh" "install"] {os = "linux"}
]
wrap-remove-commands: ["%{hooks}%/sandbox.sh" "remove"] {os = "linux"}
post-install-commands:
  ["%{hooks}%/opam-bin-cache.sh" "store" build-id installed-files]
    {?build-id & error-code = "0"}

For people using multiple ocaml root, another solution is to put opam-bin-cache.sh in the PATH, and to modify $HOME/.opam/config this way:

pre-install-commands:
  ["opam-bin-cache.sh" "restore" build-id name] {?build-id}
wrap-build-commands: [
  ["opam-bin-cache.sh" "wrap" build-id] {?build-id}
  ["%{hooks}%/sandbox.sh" "build"] {os = "linux"}
]
wrap-install-commands: [
  ["opam-bin-cache.sh" "wrap" build-id] {?build-id}
  ["%{hooks}%/sandbox.sh" "install"] {os = "linux"}
]
wrap-remove-commands: ["%{hooks}%/sandbox.sh" "remove"] {os = "linux"}
post-install-commands:
  ["opam-bin-cache.sh" "store" build-id installed-files]
    {?build-id & error-code = "0"}

Let's see how it works.

mkdir /tmp/demo{1,2}
cd /tmp/demo1
opam switch create . --empty
opam install ocaml-base-compiler.4.06.1 merlin utop core # takes 5 minutes
cd /tmp/demo2
opam switch create . --empty
opam install ocaml-base-compiler.4.06.1 merlin utop core # takes 10 seconds

I hope this setting can push people to use more local switches and avoid some of the confusion coming with global switches.

Please note that this is still experimental. There are some issues with ocamlfind that are documented here.

esy, another cli tool to get opam packages, also provides compilation cache. They claimed to have solve those problems.

andreypopp : @Khady in esy findlib.conf only contains a single entry which points to ocamlfind itself, then esy constructs $OCAMLPATH which includes all packages in a sandbox

andreypopp : we also have custom build steps for ocamlfind which re-create $OCAMLLIB inside ocamlfind so that ocamlfind can put topfind in there (this is needed b/c some pacages like to see topfind inside $OCAMLLIB).

andreypopp : details: https://github.com/esy-ocaml/esy-opam-override/blob/6/packages/ocamlfind.<1.7.0/files/_esy/build