James Routley

To make the application code and data easy to bundle together, I created fnlpack. It takes directories containing Lua and Fennel source code and turns them into a single Lua file that embeds a virtual filesystem with all the Lua code and data necessary to run the application, so long as the native libraries are in place.

When we combine these two components, the bundle created by fnlpack and the native binary created by Zig, we produce the desired outcome: a single standalone binary that we can copy to any Linux system and run.

Lua hacker Sophie "sOwOphie" wrote a primer on bundling Lua:

[A bundler concatenates] a number of Lua modules into a single script, which can be deployed easily without having to worry about LUA_PATH. […]

[This] is achieved through the use of the package.preload table. It can be filled with functions that return the corresponding module when called. The built-in load function is the perfect fit for this. It takes Lua code in string form, plus a source name, and returns a function that executes said code. […]

[The bundler can] get around string escaping issues with Lua "literal string" literals. A literal string begins with [=[, with any number of = characters. It continues until a corresponding ]=] with the same number of = characters. Inside the literal string, escapes are not processed, everything is taken verbatim.

I didn't read her primer before beginning my implementation, but the technique I ended up using matches her description pretty much exactly, so I recommend reading her full primer to gain understanding of the technique.

The main thing that I added to fnlpack which is not described there, beyond adding support for Fennel source code in addition to Lua, is my virtual filesystem implementation. I describe it like this in the README for fnlpack (under "Expand to view VFS detail"):

You can use the virtual file system (VFS) to locate packed files at runtime.

Modules packed with fnlpack will already be preloaded in your module cache, so you do not need to use the VFS at all unless your application needs to read those files again during operation.

All files in your pack are available as strings at runtime. You can find them in the table __fnlpack.files which maps file paths → file contents. To find which path corresponds to a given Lua module, inspect the table __fnlpack.modules which maps Lua module names → file paths.

The flags --patch-dofile and --patch-io trigger inclusion into your pack of patches for dofile + fennel.dofile, or io, respectively, to make them find files in the VFS when they would otherwise fail because they cannot find a file. In cases where a file can be found at runtime, the default behavior applies. You do not need to use either flag to enable the VFS, they are only for convenience in cases where you want to make unmodified code use the VFS implicitly.

This allows fnlpack users the flexibility to build bundles that include not just source code but all the data necessary to run an application, and a convenient adapter to make Lua's native io APIs work with that data. technomancy/search was a useful practical demonstration of how this tool can simplify deployment of existing Fennel/Lua applications.

To build the bundle for search, we invoke:

  fnlpack --add-requires \
    --patch-dofile \
    --patch-io \
    --no-fennel-sandbox \
    --pack sql \
    . >/opt/build/dist/technomancy-search.lua

Before we do that, though, we run a zig build which installs our source files into the build target directory:

  const lua_http_lib = lua_http_dep.artifact("lua-http");
  const install_lua_http_scripts = b.addInstallDirectory(.{
      .source_dir = lua_http_dep.namedWriteFiles("lua-http")
          .getDirectory(),
      .install_dir = .prefix,
      .install_subdir = "share/lua/5.4",
  });

  const search_scripts =
      b.addNamedWriteFiles("technomancy-search");
  _ = search_scripts.addCopyDirectory(
      b.path("src"),
      ".",
      .{ .include_extensions = &.{ ".fnl", ".lua" } },
  );
  _ = search_scripts.addCopyDirectory(
      b.path("sql"),
      "sql",
      .{ .include_extensions = &.{".sql"} },
  );
  _ = search_scripts.addCopyDirectory(
      b.path("static"),
      "static",
      .{},
  );
  const install_search_scripts = b.addInstallDirectory(.{
      .source_dir = search_scripts.getDirectory(),
      .install_dir = .prefix,
      .install_subdir = "share/lua/5.4",
  });

The namedWriteFiles("lua-http") are files that we put into a "named files" box during the lua-http build, anticipating wanting to refer to them in downstream builds:

  const lua_scripts = b.addNamedWriteFiles("lua-http");
  _ = lua_scripts.addCopyFile(
      fifo_dep.path("fifo.lua"),
      "fifo.lua",
  );
  _ = lua_scripts.addCopyFile(
      basexx_dep.path("lib/basexx.lua"),
      "basexx.lua",
  );
  _ = lua_scripts.addCopyDirectory(
      lpeg_patterns_dep.path("lpeg_patterns"),
      "lpeg_patterns",
      .{},
  );
  // and so on...

So this sets up the final assembly:

Zig builds each dependency, including lua-http, which puts its various Lua modules it depends upon into its "lua-http" named files box
We use fnlpack to pack all those scripts, together with our own Fennel sources and SQL data, into a module preload pack
Zig inserts that preload pack into the binary, along with Zig code that loads it at runtime

Build Standalone Lua Applications with Zig and fnlpack