Moat Operator Guide

How to install, configure, pick default VM templates, harden, and operate moatd on your own Linux host.

Linux x86_64 + KVM Config: /etc/moat/moatd.json Data: /var/lib/moat MIT License

01How to run Moat

Moat ships four binaries that work together or independently:

  • moatd — the long-running HTTP/JSON (and Unix socket) daemon that manages sandboxes, warm pools, egress proxying, secret scanning, and the audit log.
  • moat — the operator and developer CLI for one-shot runs (moat run), host checks (moat doctor), templates (moat template), snapshots, audit queries (moat audit), orphan reaping (moat reap), and removal (moat uninstall).
  • moat-mcp — stdio Model Context Protocol bridge that lets Claude Code, Cursor, or any MCP client drive moatd.
  • moat-uffd — optional userfaultfd demand-paging helper for snapshot restores with use_uffd: true.

Start the daemon under systemd

# 1. Verify KVM, Firecracker, cgroup v2 delegation, and images.lock
moat doctor

# 2. Enable and start moatd (system install)
sudo systemctl enable --now moatd
sudo journalctl -u moatd -f

# Or user-mode install (--user)
systemctl --user enable --now moatd

One-shot CLI usage (no daemon required)

# Run an untrusted script in a fresh microVM with file I/O and a timeout
moat run -put ./script.py:/work/script.py -get /work/out.json:./out.json \
  -timeout 30s -- python3 /work/script.py

# Use a specific kernel or rootfs image
moat run -rootfs /var/lib/moat/python-rootfs.ext4 -- python3 -c "print(2+2)"

Every CLI run copies the rootfs into a private per-VM run directory and deletes it on exit unless you pass -keep, so one-shot runs never mutate shared template images.

02Choosing & setting the default VM

A template is a named kernel + ext4 rootfs pair with resource caps and optional exec defaults (env, workdir, default_argv). Every sandbox starts from one template.

Rule for the default VM In moatd.json, the first entry in the templates array is the default template. Whenever an API caller or MCP agent omits "template" on POST /v1/sandboxes (or moat_create), moatd starts a sandbox from templates[0].

Which template should be your default?

TemplateBase userlandDisk sizeBest default for...
default (built-in) Alpine Linux (musl) + guest-agent 256 MiB (verified by images.lock) Smallest attack surface, fastest starts, shell scripts, and verified supply-chain environments.
python (imported) python:3.12-slim (Debian glibc) ~400 MiB AI coding agents doing data analysis, Python scripting, and test execution.
dev-shell Python, Node, Go, C toolchain, git 1 GiB General-purpose coding assistants that switch languages across tasks.
rhel9 Red Hat UBI 9 (glibc + dnf) 1 GiB Enterprise Linux compatibility and glibc-compiled binaries.

How to change the default VM in moatd.json

Put your preferred template first in templates and restart moatd. You can also keep default as a second entry or restrict specific tokens to specific templates.

{
  "image_verify": "enforce",
  "templates": [
    {
      "name": "python",
      "kernel": "/var/lib/moat/vmlinux",
      "rootfs": "/var/lib/moat/python-rootfs.ext4",
      "vcpus": 2,
      "mem_mib": 1024,
      "cpu_max": 2,
      "mem_max_mib": 1024,
      "pids_max": 256,
      "pool": 2,
      "read_only_root": true,
      "scratch_mib": 512,
      "workdir": "/work",
      "env": { "PYTHONUNBUFFERED": "1" }
    },
    {
      "name": "minimal",
      "kernel": "/var/lib/moat/vmlinux",
      "rootfs": "/var/lib/moat/rootfs.ext4",
      "vcpus": 1,
      "mem_mib": 256,
      "read_only_root": true,
      "scratch_mib": 256
    }
  ],
  "tokens": [
    { "id": "agent", "secret": "replace-me", "max_concurrent": 4, "max_ttl": "20m" }
  ]
}

How to override the default VM for the CLI

The moat CLI resolves its default kernel and rootfs in this order:

  1. Explicit flags: moat run -kernel /path/vmlinux -rootfs /path/custom.ext4 -- ...
  2. $MOAT_DATA directory (reads $MOAT_DATA/vmlinux and $MOAT_DATA/rootfs.ext4)
  3. $XDG_DATA_HOME/moat, then ~/.local/share/moat, then /var/lib/moat

To make a custom template the CLI default for your shell, point MOAT_DATA at a directory containing your vmlinux and rootfs.ext4 (or symlink rootfs.ext4 to your chosen image when not using image_verify=enforce on that directory).

Inspecting installed templates

moat template list                  # list all *-rootfs.ext4 images in MOAT_DATA
moat template show python           # print metadata, source digest, ENV, WORKDIR, ENTRYPOINT

03Importing container images as VM templates

You do not need to write a Dockerfile to create a custom VM. moat template import converts any linux/amd64 OCI container image or archive into an ext4 rootfs with /sbin/guest-agent injected, preserving the image's ENV, WORKDIR, and ENTRYPOINT:

# Pull from a registry and register it directly into moatd.json
moat template import python:3.12-slim --name python --config /etc/moat/moatd.json

# Import an air-gapped docker-save or OCI tarball
moat template import --from-archive ./internal-ci.tar --name ci --config /etc/moat/moatd.json

# Reload moatd after updating moatd.json
sudo systemctl restart moatd

If you want the newly imported template to be the default VM, move its object to index 0 of "templates" in moatd.json.

04Read-only rootfs vs. sparse copy

Moat never shares writable disk state between two sandboxes. Each template supports two storage modes:

ModeConfigHow it worksWhen to use
Read-only rootfs + overlayfs (recommended) "read_only_root": true,
"scratch_mib": 512
Attaches the template ext4 read-only (/dev/vda) and pairs it with a tiny sparse per-VM ext4 scratch disk (/dev/vdb) mounted as an overlayfs upper layer at boot. Best for large templates (python, dev-shell, rhel9) and high sandbox churn. Zero base image copying; fast startup; low disk I/O.
Private sparse/reflink copy (default) "read_only_root": false Copies the template rootfs into the sandbox run directory using cp --reflink=auto --sparse=always and deletes it on destroy. Simple default on XFS/Btrfs/overlay-free workflows or when tools inside the guest expect a single mutable ext4 root.

05Warm pools & snapshot pools

Warm pools keep pre-started, non-networked microVMs ready so POST /v1/sandboxes hands out a clean VM in ~1 ms instead of waiting for a cold start. Each VM is still strictly single-use and destroyed on release.

  • Per-template pools: set "pool": 4 directly on any template in moatd.json. Each template gets its own isolated pool — requests for python never borrow a VM from default.
  • Global fallback: "defaults": { "pool": N } (or -pool N) sets the pool size for the default template when that template leaves "pool": 0.
  • Snapshot-templated warm pools: pre-warm an environment once with moat snapshot, then point the template at it with "snapshot_dir": "/var/lib/moat/snaps/python" (and optionally "use_uffd": true for demand-paged memory). Moat automatically resyncs the guest wall clock after every restore so TLS and timestamps stay accurate.
# Build a primed snapshot once
moat snapshot -id python-snap -rootfs /var/lib/moat/python-rootfs.ext4 \
  -out /var/lib/moat/snaps/python -exec 'python3 -c "import ssl, json"'

# Reference it in moatd.json
{
  "name": "python",
  "rootfs": "/var/lib/moat/python-rootfs.ext4",
  "pool": 4,
  "snapshot_dir": "/var/lib/moat/snaps/python",
  "use_uffd": false
}

06Sizing CPU, RAM, disk & quotas

Every sandbox enforces both guest-level and host-level cgroup v2 caps:

FieldScopeRecommendation
vcpus, mem_mibGuest VM sizing1 vCPU / 256 MiB for shell/minimal; 2 vCPU / 1024 MiB for Python/Node; 4 vCPU / 4096 MiB for Go/Rust builds.
cpu_max, mem_max_mib, pids_maxHost cgroup v2 on VMMSet mem_max_mib slightly above mem_mib (e.g., +64–128 MiB for VMM overhead) and pids_max: 256 to neutralize fork-bombs.
disk_bw, net_bwFirecracker token buckets (bytes/sec)Cap noisy build sandboxes (e.g., 104857600 = 100 MiB/s) so one sandbox cannot starve host I/O.
defaults.global_max, token_maxConcurrency limitsKeep global_max * avg_mem_mib below 75% of host RAM. Excess requests get HTTP 429 and an audit event.
defaults.default_ttl, max_ttlSandbox lifetime15m default / 1h max prevents forgotten agent sandboxes from lingering.

07Tokens & per-tenant policy

Every /v1/... API route requires Authorization: Bearer <secret>. Callers can only list, exec in, or destroy sandboxes created by their own token. Give each agent, CI pipeline, or team its own token with least-privilege rules:

{
  "id": "ci-python",
  "secret": "use-a-long-random-secret",
  "templates": ["python"],
  "max_concurrent": 4,
  "max_ttl": "20m",
  "deny_network": false,
  "egress_allow": ["pypi.org", "files.pythonhosted.org"],
  "egress_deny": ["evil.pypi.org"],
  "allow_secret_override": false,
  "audit_read": ""
}
  • egress_allow / egress_deny scope outbound hosts per token. Global deny rules always win over per-token allow rules.
  • allow_secret_override defaults to false. Keep it off for AI agents so they cannot bypass the ingress secret firewall.
  • audit_read: "all" grants read-only visibility across all tokens on GET /v1/audit and GET /v1/sandboxes/{id}/report (intended for a dedicated SecOps token).

08Networking & strict egress

Sandboxes have no network interface by default ("networked": false). When a caller requests "networked": true (and the token permits it):

  • Moat allocates a dedicated /30 subnet from net_pool (default 10.62.0.0/16) inside a per-VM network namespace, and releases it on sandbox destroy.
  • nftables unconditionally drops routes to the host, loopback, 169.254.0.0/16 (cloud metadata), and RFC-1918 private ranges.
  • Run moatd with -egress-transparent 0.0.0.0:3128 and "defaults": { "strict_egress": true } so guest TCP 80/443 is DNATed transparently to the SNI/Host-filtering proxy and the forward chain defaults to policy drop (allowing only proxy traffic and configured DNS).
Privileges required for networking Per-VM network namespaces and nftables require root or systemd capabilities CAP_NET_ADMIN and CAP_SYS_ADMIN. Uncomment the capability lines in /etc/systemd/system/moatd.service when enabling networked sandboxes.

09Unix sockets, TLS/mTLS & jailer

Local-only host (recommended for solo devs and local MCP)

Bind moatd exclusively to a 0660 Unix domain socket with no TCP listener:

# In /etc/moat/moatd.json
"socket": "/run/moat/moatd.sock"

# In systemd ExecStart
ExecStart=/usr/local/bin/moatd -addr "" -config /etc/moat/moatd.json

Remote API with native TLS 1.2+ or mTLS

# In /etc/moat/moatd.json (or via -tls-cert / -tls-key / -tls-client-ca)
"tls_cert": "/etc/moat/server.pem",
"tls_key": "/etc/moat/server.key",
"tls_client_ca": "/etc/moat/clients-ca.pem"

Setting tls_client_ca enforces mutual TLS (RequireAndVerifyClientCert) on every TCP connection in addition to bearer-token auth.

Firecracker jailer confinement

Enable "defaults": { "jail": true, "jail_uid": 1001, "jail_gid": 1001 } (or -jail) so each Firecracker VMM runs chrooted, under a seccomp filter, and with dropped privileges. Requires root or CAP_SYS_ADMIN.

10Image verification & Cosign

At startup, moatd reads images.lock beside each template's images. In the default "image_verify": "enforce" mode:

  • Every template covered by images.lock must match its kernel and rootfs SHA-256 digests. Failed templates are dropped before serving, and moatd exits non-zero if zero usable templates remain.
  • Secondary imported templates that share the verified kernel still have their kernel digest checked.
  • Set "cosign_identity" and "cosign_issuer" (or MOAT_COSIGN_IDENTITY / MOAT_COSIGN_ISSUER) to also verify images.lock.sig with Cosign before startup.

11Audit logs, reports & SIEM forwarding

The audit log is enabled by default (written to /var/lib/moat/audit or audit.dir). Every event is metadata-only (no secrets, file bodies, or stdout) and chained by SHA-256.

# Follow live events
moat audit tail

# Query a specific token or event type
moat audit query -token ci -type secret.block -since -24h

# Rollup everything one sandbox did (works after destruction)
moat audit report sb-7f3a19c4

# Verify the entire hash chain against tampering
moat audit verify

To stream events to Splunk, Datadog, Vector, or a custom webhook collector without slowing API calls, add "webhook_url": "https://siem.internal/moat" under "audit" (or pass -audit-webhook).

12Monitoring, recovery & uninstall

Prometheus metrics

Scrape GET /metrics and alert on:

  • rate(moat_secret_findings_total{action="block"}[5m]) > 0 — credentials hitting the ingress firewall.
  • rate(moat_quota_rejects_total[5m]) > 0 — callers hitting concurrency caps.
  • moat_pool_ready == 0 — warm pool exhausted (falling back to cold starts).
  • increase(moat_audit_dropped_total[5m]) > 0 — audit writer falling behind or disk pressure.

Crash recovery & orphan reaping

moatd automatically reaps orphaned Firecracker processes left by a host crash on startup and logs orphan.reap audit events. To sweep orphans manually:

moat reap

Clean uninstall

moat uninstall --dry-run      # preview what will be removed
moat uninstall --keep-data    # remove binaries + unit, keep images & audit logs
moat uninstall                # full removal (add --user for $HOME installs)