# Module Topkg

    module Topkg: sig .. end

The transitory OCaml package builder.

See the basics and the menagerie of `pkg.ml` files.

v1.0.1 - homepage

## Preliminaries

In the most simple cases you won't need this, jump directly to the basics or package description API.

    val (>>=) : ('a, 'b) Stdlib.result ->
           ('a -> ('c, 'b) Stdlib.result) -> ('c, 'b) Stdlib.result

`r >>= f` is `f v` if `r = Ok v` and `r` if `r = Error _`.

    val (>>|) : ('a, 'b) Stdlib.result -> ('a -> 'c) -> ('c, 'b) Stdlib.result

`r >>| f` is `Ok (f v)` if `r = Ok v` and `r` if `r = Error _`.

    type ('a, 'b) r = ('a, 'b) Stdlib.result =

`|` | `Ok of `'a``
`|` | `Error of `'b``

This definition re-export `result`'s constructors so that an `open Topkg` gets them in scope.

    type 'a result = ('a, [ `Msg of string ]) r

The type for topkg results.

    module R: sig .. end

Result value combinators.

    val strf : ('a, Stdlib.Format.formatter, unit, string) Stdlib.format4 -> 'a

`strf` is `Printf.asprintf`.

    module String: sig .. end

Strings.

    type fpath = string

The type for file system paths.

    module Fpath: sig .. end

File system paths.

    module Cmd: sig .. end

Command lines.

    module Log: sig .. end

Topkg log.

    module OS: sig .. end

OS interaction.

    module Vcs: sig .. end

Version control system repositories.

## Package description

    module Conf: sig .. end

Build configuration.

    module Exts: sig .. end

Exts defines sets of file extensions.

    module Pkg: sig .. end

Package description.

## Private

    module Private: sig .. end

Private definitions.

## Basics

`Topkg` is a packager for distributing OCaml software. Fundamentally it only provides a simple and flexible mechanism to describe an opam `install` file according to the build configuration which is used to infer one corresponding invocation of your build system.

This simple idea brings the following advantages:

1. It frees you from implementing an install procedure in your build system: this task is delegated to opam, to the `opam-installer` tool or anything that understands an opam install file.
2. It doesn't reclaim control over your build system. It only invokes it once with a list of targets determined by the package install description.
3. It is very flexible, supports a wide range of installation scenarios and is expressed in the reasonable language.

Beyond this a `Topkg` package description provides information about the package's distribution creation and publication procedures. This enables swift and correct package distribution management via the `topkg` tool, see Package care.

- Source repository setup
- opam and package build instructions
- Package description
- Install description
- Package care
- Advanced topics
  - Storing build configuration information in software
  - Multiple opam packages for a single distribution
- Menagerie of pkg.ml files

## Source repository setup

The root of you source repository should have the following layout:

- `pkg/pkg.ml`, the package description written with `Topkg`'s API. See package description.
- `pkg/META`, an ocamlfind META file describing your package. See `Topkg.Pkg.describe` to configure this.
- `_tags`, ocamlbuild file with at least a `true : bin_annot` line. See handling cmt and cmti files for details.
- `opam`, the package's opam metadata. See opam and package build instructions.
- `README.md`, your readme file. See `Topkg.Pkg.describe` to configure this.
- `LICENSE.md`, your license file. See `Topkg.Pkg.describe` to configure this.
- `CHANGES.md`, your change log. See `Topkg.Pkg.describe` to configure this.

### Quick setup (advertisement)

If you start a new library `carcass` can generate the structural boilerplate with your personal information and preferences. Invoke:

    carcass body topkg/pkg mypkg
    (cd mypkg && git init && git add . && git commit -m "First commit.")
    opam pin add -kgit mypkg mypkg#master

and you have a library that is `{opam,ocamlfind}`-able with correct version watermarks on releases and opam pins. You are now only a few invocations away to release it in the OCaml opam repository, see `topkg help release` for doing so; but don't forget to document it and make it do something useful.

## opam and package build instructions

The package needs to build-depend on `topkg` as well as `ocamlfind` which is used by the package description file `pkg/pkg.ml` to find the `topkg` library; it is likely that you are using `ocamlbuild` too. So the depends field of your opam file should at least have:

    depends: [
      "ocamlfind" {build}
      "ocamlbuild" {build}
      "topkg" {build & >= 0.9.0} ]

The build instructions of the package are simply an invocation of `pkg/pkg.ml` with the `build` command and a specification of the build configuration on the command line. In the simplest case, if your package has no configuration options, this simply boils down to:

    # For opam >= 2.0
    build: [[ "ocaml" "pkg/pkg.ml" "build" "--dev-pkg" "%{dev}%" ]]

    # For opam < 2.0
    build: [[ "ocaml" "pkg/pkg.ml" "build" "--dev-pkg" "%{pinned}%" ]]

The `"--dev-pkg"` configuration key is used to inform the package description about the build context. This invocation of `pkg/pkg.ml` executes your build system with a set of targets determined from the build configuration and generates in the root directory of your distribution an opam `install` file that opam uses to install and uninstall your package.

This is all you need to specify. Do not put anything in the remove field of the opam file. Likewise there is no need to invoke `ocamlfind` with your `META` file. Your `META` file should simply be installed in the directory of the `lib` field which happens automatically by default.

If you described tests then you should specify the instructions as follows (unfortunately for opam < 2.0 this involves the repetition of the build line):

    # For opam >= 2.0

    build:
    [[ "ocaml" "pkg/pkg.ml" "build" "--dev-pkg" "%{dev}%"
                                    "--tests" "%{build-test}%" ]]
    run-test:
    [[ "ocaml" "pkg/pkg.ml" "test" ]]

    # For opam < 2.0

    build:
    [[ "ocaml" "pkg/pkg.ml" "build" "--dev-pkg" "%{pinned}%" ]]

    build-test:
    [[ "ocaml" "pkg/pkg.ml" "build" "--dev-pkg" "%{pinned}%" "--tests" "true" ]
     [ "ocaml" "pkg/pkg.ml" "test" ]]

Beyond opam. If you need to support another package system you can invoke `pkg/pkg.ml` as above and then manage the installation and uninstallation at a given `$DESTDIR` with the generated opam `install` file using `opam-installer` tool or any other program that understand these files.

## Package description

The `Topkg.Pkg.describe` function has a daunting number of arguments and configuration options. However if you keep things simple and stick to the defaults, much of this does not need to be specified.

For example, if we consider the basic setup mentioned above for a library described with an OCamlbuild `src/mylib.mllib` file, your package description file `pkg/pkg.ml` simply looks like this:

    #!/usr/bin/env ocaml
    #use "topfind"
    #require "topkg"
    open Topkg

    let () =
      Pkg.describe "mylib" @@ fun c ->
      Ok [ Pkg.mllib "src/mylib.mllib" ]

Tip. To allow merlin to function correctly in your package description, issue `M-x merlin-use topkg` in `emacs` or `:MerlinUse topkg` in `vim`.

This simple description builds and installs the library described by the `src/mylib.mllib` file. It works correctly if native code compilation or native dynamic linking is not available. It automatically installs the package's META file in the directory of the `lib` field and your readme, change log and license file in the directory of the `doc` field.

Try to test the package build description with `topkg build`, this builds the package according to the description and build configuration and writes a `mylib.install` at the root of the distribution that describes the package installation. If everything went well, you are now ready to release, see `topkg help release` for the procedure.

To debug package descriptions it useful to dry run the build. This prevents the package from building and only writes the `mylib.install` file determined according to the build configuration.

    topkg build -d    # Only write the opam install file
    topkg build -d -v # Also print the build configuration
    topkg help troubleshoot # More troubleshooting tips

Note that `topkg build` does nothing more than invoke `ocaml "pkg/pkg.ml" build`. If you would like to see the build configuration options of a package description you should do:

    ocaml pkg/pkg.ml help
    ./pkg/pkg.ml help     # If has exec right

## Install description

An opam `install` file is a description of a standard UNIX install. It has fields for each of the standard directories `lib`, `bin`, `man`, `etc`, etc. Each of these fields lists the files to install in the corresponding directory (or subdirectories). See the install file specification in the opam developer manual for more information.

A topkg install description is just a convenient and compact way to describe an opam install file according to the build configuration. In turn this also describes what needs to be built which allows topkg to call the build system appropriately.

For each opam install field there is a corresponding field function that you can use to generate install moves. The documentation of `Topkg.Pkg.field` and `Topkg.Pkg.exec_field` describes how you can use or omit their various arguments to simplify the description. Topkg also provides a few higher-level convenience functions like `Topkg.Pkg.mllib` and `Topkg.Pkg.clib` which allow to reuse the description work already done for OCamlbuild.

In the following we review a few basic install use cases. The menagerie provides links to full and more complex examples.

### Installing libraries and C stubs

It is possible to use the `Topkg.Pkg.lib` field function and appropriate file extensions to manually install a library, but this quickly becomes tedious. The higher-level `Topkg.Pkg.mllib` install function brings this to a single line by reading from a OCamlbuild `mllib` file. Given a library described in `src/mylib.mllib` file use:

    Pkg.mllib "src/mylib.mllib"

This will make all the modules mentioned in the `mllib` file part of the API (i.e. install its `cmi` files). You can restrict the API by using the `~api` optional argument. In the following, only `Mod1` and `Mod2` define the API, the other modules of `mllib` remain hidden to the end user (but unfortunately not to the linkers...):

    Pkg.mllib ~api:["Mod1"; "Mod2"] "src/myllib.mllib"

A shortcut also exists for installing C stubs: `Topkg.Pkg.clib`. Simply use it on an existing `src/libmystub.clib` file (N.B. OCamlbuild mandates that your `clib` file name starts with `lib`):

    Pkg.clib "src/libmystub.clib"

This will generate the appropriate install moves in the `lib` and `stublib` fields.

### Installing binaries

In opam, binaries can only be installed by using the `bin`, `sbin` and `libexec` fields. Trying to install a binary in other fields will not set its executable bit (discussion).

Most of the time one wants to install native binaries if native code compilation is available and bytecode ones if it is not. The `auto` argument of `Topkg.Pkg.exec_field` does exactly this if omited.

So if you have an executable to install in the `bin` field whose `main` is in `src/myexec.ml`, describe it with:

    Pkg.bin "src/myexec"

As with any other field it easy to rename the executable along the way with the `dst` argument:

    Pkg.bin "src/myexec" ~dst:"my-exec"

Note that using the default value of `auto` also automatically handles the extension business for Windows platforms.

### Conditional install

An easy and readable way to handle conditional installs is to use the `cond` argument of field functions. The following shows how to conditionaly build and install a binary depending on the opam `cmdliner` package being installed:

    let cmdliner = Conf.with_pkg "cmdliner"
    let () =
      Pkg.describe "mypkg" @@ fun c ->
      let cmdliner = Conf.value c cmdliner in
      Ok [ Pkg.bin ~cond:cmdliner "src/myexec" ]

Note that configuration keys must be created before the call to `Topkg.Pkg.describe`. Their value can then be accessed with the `Topkg.Conf.value` from the configuration given to the install move function you specify.

For this conditional install the opam build instructions look like this:

    depopts: [ "cmdliner" ]
    build: [[
      "ocaml" "pkg/pkg.ml" "build"
              "--dev-pkg" "%{dev}%" # use "%{pinned}%" for opam < 2.0
              "--with-cmdliner" cmdliner:installed ]]

### Automatic install conditions

Some conditions related to native code and native code dynamic linking happen automatically. For example moves with paths ending with `.cmxs` are automatically dropped if `Topkg.Conf.OCaml.native_dynlink` is `false` in the current build configuration. This behaviour can be disabled by using the `force` argument of field functions.

### Extension sets and platform dependent extensions

The field functions have an optional `exts` argument. If present these extensions are appended to the `src` path given to the function. The module `Topkg.Exts` defines a few predefined extension sets. For example a single module library archive implemented in `src/mylib.ml` can be declared by:

    Pkg.lib ~exts:Exts.module_library "src/mylib"

which is, effectively, a shortcut for:

    [ Pkg.lib "src/mylib.mli";
      Pkg.lib "src/mylib.cmti";
      Pkg.lib "src/mylib.cmi";
      Pkg.lib "src/mylib.cmx";
      Pkg.lib "src/mylib.cma";
      Pkg.lib "src/mylib.a";
      Pkg.lib "src/mylib.cmxa";
      Pkg.lib "src/mylib.cmxs"; ]

Extensions sets are also used to support platform independent installs. For example to install a static C library archive you should use the second invocation, not the first one:

    Pkg.lib "src/libmylib.a" (* DON'T do this *)
    Pkg.lib ~exts:Exts.c_library "src/libmylib" (* Do this *)

this ensures that the correct, platform dependent, suffix is used.

### Renaming and installing in subdirectories

By default install moves simply install the file at the root directory of the field. Using the `dst` optional argument you can rename the file and/or install it to subdirectories.

    Pkg.lib "src/b.cmo" ~dst:"hey"  (* Install as [hey] file. *)
    Pkg.lib "src/b.cmo" ~dst:"hey/" (* Install in [hey] subdirectory *)
    Pkg.lib "src/b.cmo" ~dst:"hey/ho.cmo"  (* Install as [ho.cmo] in [hey]. *)

### Handling cmt and cmti files

Since the OCaml tools generate `.cmt` and `.cmti` files only as a side effect they are treated specially: they are not built. For OCamlbuild you should add this line to your `_tags` file:

    true : bin_annot

this will build them as a side effect of other build invocations. In the generated opam install file the `cmt` and `cmti` files are prefixed by a ? so that if they are not built the install does not fail.

## Package care

Developing and distributing packages implies a lot of mundane but nevertheless important tasks. The `topkg` tool, guided by information provided by `Topkg.Pkg.describe` helps you with these tasks.

For example `topkg lint` makes sure that the package source repository or distribution follows a few established (or your) conventions and that the META and opam files of the package pass their respective linter. `topkg distrib` will create watermarked and reproducible distribution archives (see `Topkg.Pkg.distrib`) while `topkg publish` and `topkg opam` will help publishing them. All this and more is described with details in `topkg`'s help system, invoke:

    topkg help release
    topkg help

to get an extended introduction and pointers to these features.

### Documentation care

Topkg provides support to make it easier to write and publish your package documentation. The `topkg doc -r` command generates and refreshes renderings of the package documentation while you work on it.

Documentation publication is always derived from a generated distribution archive (the latter doesn't need to be published of course). So to push documentation fixes and clarifications simply invoke:

    topkg distrib
    topkg publish doc

Make sure you include %‌%VERSION%% watermarks in the documentation so that readers know exactly which version they are reading. By default this will be substituted by a VCS tag description so it will be precise even though you may not have properly tagged a new release for the package.

## Advanced topics

### Storing build configuration information in software

The following sample setup shows how to store build configuration information in build artefacts.

In this example we store the location of the install's `etc` directory in the build artefacts. The setup also works seamlessly during development if build artefacts are invoked from the root directory of the source repository.

We have the following file layout in our source repository:

    etc/mypkg.conf       # Configuration file
    src/mypkg_etc.ml     # Module with path to the etc dir

the contents of `src/mypkg_etc.ml` is simply:

    (* This file is overwritten by distribution builds. During development
       it refers to the [etc] directory at the root directory of the
       source repository. *)

    let dir = "etc"

the value `Mypkg_etc.dir` is used in the sources to refer to the `etc` directory of the install. In the package description file `pkg/pkg.ml` we have the following lines:

    let (* 1 *) etc_dir =
      let doc = "Use $(docv) as the etc install directory" in
      Conf.(key "etc-dir" fpath ~absent:"etc" ~doc)

    let (* 2 *) etc_config c = match Conf.build_context c with
    | `Dev -> Ok () (* Do nothing, the repo src/mypkg_etc.ml will do *)
    | `Pin | `Distrib ->
        let config = strf "let dir = %S" (Conf.value c etc_dir) in
        OS.File.write "src/mypkg_etc.ml" config

    let () =
      let build = Pkg.build ~pre:etc_config () in
      Pkg.describe "mypkg" ~build @@ fun c ->
      Ok [ (* 3 *) Pkg.etc "etc/mpypkg.conf"; ... ]

In words:

1. We declare a configuration key `"etc-dir"` that holds the location of the install `etc` directory.
2. We have a pre-build hook that writes the file `src/mypkg_etc.ml` with its actual value on ``Pin` and ``Distrib` builds.
3. We install the `etc/mypkg.conf` configuration in the install `etc` directory.

The opam build instructions for the package are:

    build: [[
      "ocaml" "pkg/pkg.ml" "build"
              "--dev-pkg" "%{dev}%" # use "%{pinned}%" for opam < 2.0
              "--etc-dir" mypkg:etc ]]

### Multiple opam packages for a single distribution

It is not too hard to define multiple opam packages for the same distribution. Topkg itself uses this trick to manage its dependencies between `topkg` and `topkg-care`.

To achieve this your package description file can simply condition the package install description on the package name communicated by the configuration. In this setup you'll likely have one `$PKG.opam` per `$PKG` at the root of your source repository, you should declare them in the description too, so that they get properly linted and used by the `topkg` tool when appropriate (see how the opam file is looked up according to the package name in `Topkg.Pkg.describe`). Here is a blueprint:

    let () =
      let opams =
        let install = false in
        [ Pkg.opam_file ~install "mypkg-main.opam";
          Pkg.opam_file ~install "mypkg-snd.opam"; ]
      in
      Pkg.describe ~opams "mypkg-main" @@ fun c ->
      match Conf.pkg_name c with
      | "mypkg-main" ->
          Ok [ Pkg.lib "mypkg-main.opam" ~dst:"opam";
               (* mypkg-main install *) ]
      | "mypkg-snd" ->
          Ok [ Pkg.lib "mypkg-snd.opam" ~dst:"opam";
               (* mypkg-snd install *) ]
      | other ->
          R.error_msgf "unknown package name: %s" other

The build instructions of these opam files need to give the name of the package to the build invocation so that the right install description can be selected:

    build: [[
      "ocaml" "pkg/pkg.ml" "build"
              "--pkg-name" name
              "--dev-pkg" "%{dev}%" # use "%{pinned}%" for opam < 2.0
    ]]

In general you will use the default, main, package name and its opam file to create and publish the distribution archive file and all packages will use the same distribution; the opam packages will only differ in their opam file. Releasing the set of packages then becomes:

    # Release the distribution and base package (use topkg bistro
    # for doing this via a single invocation)
    topkg distrib
    topkg publish
    topkg opam pkg
    topkg opam submit

    # Create and release the other opam package based on the ditrib
    topkg opam pkg --pkg-name mypkg-snd
    topkg opam submit --pkg-name mypkg-snd

See `topkg help release` for more information about releasing packages with `topkg`.

## Menagerie of `pkg.ml` files

This is a menagerie of `pkg.ml` with a description of what they showcase. The examples are approximatively sorted by increasing complexity.

In all these packages the readme, change log and license file are automatically installed in the directory of the `doc` field and the ocamlfind META file and opam file of the package are automatically installed in the directory of the `lib` field.

Hmap (META, `opam`)

- Single module library archive `hmap`. The simplest you can get.

Fpath (META, `opam`)

- Single module library archive `fpath`.
- Private library archive `fpath_top` for toplevel support.

Astring (META, `opam`)

- Library archive `astring` namespaced by one module.
- Private library archive for toplevel support (`astring_top`).
- Installation of sample code in the `doc/` directory.

Fmt (META, `opam`)

- Single module library archive (`fmt`).
- Private library archive `fmt_top` for toplevel support.
- Single module library archive `fmt_tty` conditional on the presence of the opam `base-unix` package.
- Single module library archive `fmt_cli` conditional on the presence of the opam `cmdliner` package.

Ptime (META, opam)

- Single module library archive `ptime`.
- Private library archive `ptime_top` for toplevel support.
- Library archive `ptime_clock` targeting regular OSes using C stubs and installed in the `os/` subdirectory of `lib` field along with a private library archive `ptime_clock_top` for toplevel support.
- Library archive `ptime_clock` targeting JavaScript installed in the `jsoo/` subdirectory of `lib` field conditional on the presence of the opam `js_of_ocaml` package.
- Installation of sample code in the `doc/` directory.

Uucp (META, opam)

- Library archive `uucp` namespaced by a single module.
- Custom watermark for substituting the supported Unicode version.
- Generation of distribution time build artefacts via a massage hook which invokes an OCaml script that downloads the UCD XML file and extracts compact and efficient representation of it as OCaml source data structures it writes in the `src/` directory.
- Adds the `support/` path to the paths to exclude from the distribution.
- Installation of development information and sample code in the `doc/` directory.

Carcass (META, opam)

- Library archive `carcass` namespaced by a single module.
- Single module library archive `carcass_cli`.
- Executable `carcass`.
- Stores install `etc` location in the software artefacts with a pre-build hook.
- Adjusts the files to watermark to ignore the files in the `etc` file hierarchy of the distribution.
- Installs the `etc` hierarchy of the distribution in the `etc` field directly from the source tree (i.e. the files are not built).

Topkg (META, topkg.opam, topkg-care.opam)

- Ignore the funky source bootstraping (`#mod_use` directives), that's only for using `topkg` on itself.
- Makes multiple opam packages for the same distribution.
- Multiple opam file declaration and dependency linting exclusions: the build system mentions packages that are not relevant to all opam files. Manual, per package, opam file install in the `lib` field.
- Manual `META` install, a single one is installed for all opam packages by the base package `topkg`. This leverages the `if_exists` ocamlfind mecanism.
- The `topkg` package installs the library archive `topkg` namespaced by a single module.
- The `topkg-care` package installs the library archive `topkg-care` namespaced by a single module and the binaries `topkg` and `toy-github-topkg-delegate`