Moat Operator Guide
How to install, configure, pick default VM templates, harden, and operate moatd on your own Linux host.
01How to run Moat
Moat ships four binaries that work together or independently:
moatd— the long-running HTTP/JSON (and Unix socket) daemon that manages sandboxes, warm pools, egress proxying, secret scanning, and the audit log.moat— the operator and developer CLI for one-shot runs (moat run), host checks (moat doctor), templates (moat template), snapshots, audit queries (moat audit), orphan reaping (moat reap), and removal (moat uninstall).moat-mcp— stdio Model Context Protocol bridge that lets Claude Code, Cursor, or any MCP client drivemoatd.moat-uffd— optional userfaultfd demand-paging helper for snapshot restores withuse_uffd: true.
Start the daemon under systemd
# 1. Verify KVM, Firecracker, cgroup v2 delegation, and images.lock
moat doctor
# 2. Enable and start moatd (system install)
sudo systemctl enable --now moatd
sudo journalctl -u moatd -f
# Or user-mode install (--user)
systemctl --user enable --now moatd
One-shot CLI usage (no daemon required)
# Run an untrusted script in a fresh microVM with file I/O and a timeout
moat run -put ./script.py:/work/script.py -get /work/out.json:./out.json \
-timeout 30s -- python3 /work/script.py
# Use a specific kernel or rootfs image
moat run -rootfs /var/lib/moat/python-rootfs.ext4 -- python3 -c "print(2+2)"
Every CLI run copies the rootfs into a private per-VM run directory and deletes it on exit unless you pass -keep, so one-shot runs never mutate shared template images.
02Choosing & setting the default VM
A template is a named kernel + ext4 rootfs pair with resource caps and optional exec defaults (env, workdir, default_argv). Every sandbox starts from one template.
moatd.json, the first entry in the templates array is the default template. Whenever an API caller or MCP agent omits "template" on POST /v1/sandboxes (or moat_create), moatd starts a sandbox from templates[0].
Which template should be your default?
| Template | Base userland | Disk size | Best default for... |
|---|---|---|---|
default (built-in) |
Alpine Linux (musl) + guest-agent |
256 MiB (verified by images.lock) |
Smallest attack surface, fastest starts, shell scripts, and verified supply-chain environments. |
python (imported) |
python:3.12-slim (Debian glibc) |
~400 MiB | AI coding agents doing data analysis, Python scripting, and test execution. |
dev-shell |
Python, Node, Go, C toolchain, git | 1 GiB | General-purpose coding assistants that switch languages across tasks. |
rhel9 |
Red Hat UBI 9 (glibc + dnf) |
1 GiB | Enterprise Linux compatibility and glibc-compiled binaries. |
How to change the default VM in moatd.json
Put your preferred template first in templates and restart moatd. You can also keep default as a second entry or restrict specific tokens to specific templates.
{
"image_verify": "enforce",
"templates": [
{
"name": "python",
"kernel": "/var/lib/moat/vmlinux",
"rootfs": "/var/lib/moat/python-rootfs.ext4",
"vcpus": 2,
"mem_mib": 1024,
"cpu_max": 2,
"mem_max_mib": 1024,
"pids_max": 256,
"pool": 2,
"read_only_root": true,
"scratch_mib": 512,
"workdir": "/work",
"env": { "PYTHONUNBUFFERED": "1" }
},
{
"name": "minimal",
"kernel": "/var/lib/moat/vmlinux",
"rootfs": "/var/lib/moat/rootfs.ext4",
"vcpus": 1,
"mem_mib": 256,
"read_only_root": true,
"scratch_mib": 256
}
],
"tokens": [
{ "id": "agent", "secret": "replace-me", "max_concurrent": 4, "max_ttl": "20m" }
]
}
How to override the default VM for the CLI
The moat CLI resolves its default kernel and rootfs in this order:
- Explicit flags:
moat run -kernel /path/vmlinux -rootfs /path/custom.ext4 -- ... $MOAT_DATAdirectory (reads$MOAT_DATA/vmlinuxand$MOAT_DATA/rootfs.ext4)$XDG_DATA_HOME/moat, then~/.local/share/moat, then/var/lib/moat
To make a custom template the CLI default for your shell, point MOAT_DATA at a directory containing your vmlinux and rootfs.ext4 (or symlink rootfs.ext4 to your chosen image when not using image_verify=enforce on that directory).
Inspecting installed templates
moat template list # list all *-rootfs.ext4 images in MOAT_DATA
moat template show python # print metadata, source digest, ENV, WORKDIR, ENTRYPOINT
03Importing container images as VM templates
You do not need to write a Dockerfile to create a custom VM. moat template import converts any linux/amd64 OCI container image or archive into an ext4 rootfs with /sbin/guest-agent injected, preserving the image's ENV, WORKDIR, and ENTRYPOINT:
# Pull from a registry and register it directly into moatd.json
moat template import python:3.12-slim --name python --config /etc/moat/moatd.json
# Import an air-gapped docker-save or OCI tarball
moat template import --from-archive ./internal-ci.tar --name ci --config /etc/moat/moatd.json
# Reload moatd after updating moatd.json
sudo systemctl restart moatd
If you want the newly imported template to be the default VM, move its object to index 0 of "templates" in moatd.json.
04Read-only rootfs vs. sparse copy
Moat never shares writable disk state between two sandboxes. Each template supports two storage modes:
| Mode | Config | How it works | When to use |
|---|---|---|---|
| Read-only rootfs + overlayfs (recommended) | "read_only_root": true,"scratch_mib": 512 |
Attaches the template ext4 read-only (/dev/vda) and pairs it with a tiny sparse per-VM ext4 scratch disk (/dev/vdb) mounted as an overlayfs upper layer at boot. |
Best for large templates (python, dev-shell, rhel9) and high sandbox churn. Zero base image copying; fast startup; low disk I/O. |
| Private sparse/reflink copy (default) | "read_only_root": false |
Copies the template rootfs into the sandbox run directory using cp --reflink=auto --sparse=always and deletes it on destroy. |
Simple default on XFS/Btrfs/overlay-free workflows or when tools inside the guest expect a single mutable ext4 root. |
05Warm pools & snapshot pools
Warm pools keep pre-started, non-networked microVMs ready so POST /v1/sandboxes hands out a clean VM in ~1 ms instead of waiting for a cold start. Each VM is still strictly single-use and destroyed on release.
- Per-template pools: set
"pool": 4directly on any template inmoatd.json. Each template gets its own isolated pool — requests forpythonnever borrow a VM fromdefault. - Global fallback:
"defaults": { "pool": N }(or-pool N) sets the pool size for the default template when that template leaves"pool": 0. - Snapshot-templated warm pools: pre-warm an environment once with
moat snapshot, then point the template at it with"snapshot_dir": "/var/lib/moat/snaps/python"(and optionally"use_uffd": truefor demand-paged memory). Moat automatically resyncs the guest wall clock after every restore so TLS and timestamps stay accurate.
# Build a primed snapshot once
moat snapshot -id python-snap -rootfs /var/lib/moat/python-rootfs.ext4 \
-out /var/lib/moat/snaps/python -exec 'python3 -c "import ssl, json"'
# Reference it in moatd.json
{
"name": "python",
"rootfs": "/var/lib/moat/python-rootfs.ext4",
"pool": 4,
"snapshot_dir": "/var/lib/moat/snaps/python",
"use_uffd": false
}
06Sizing CPU, RAM, disk & quotas
Every sandbox enforces both guest-level and host-level cgroup v2 caps:
| Field | Scope | Recommendation |
|---|---|---|
vcpus, mem_mib | Guest VM sizing | 1 vCPU / 256 MiB for shell/minimal; 2 vCPU / 1024 MiB for Python/Node; 4 vCPU / 4096 MiB for Go/Rust builds. |
cpu_max, mem_max_mib, pids_max | Host cgroup v2 on VMM | Set mem_max_mib slightly above mem_mib (e.g., +64–128 MiB for VMM overhead) and pids_max: 256 to neutralize fork-bombs. |
disk_bw, net_bw | Firecracker token buckets (bytes/sec) | Cap noisy build sandboxes (e.g., 104857600 = 100 MiB/s) so one sandbox cannot starve host I/O. |
defaults.global_max, token_max | Concurrency limits | Keep global_max * avg_mem_mib below 75% of host RAM. Excess requests get HTTP 429 and an audit event. |
defaults.default_ttl, max_ttl | Sandbox lifetime | 15m default / 1h max prevents forgotten agent sandboxes from lingering. |
07Tokens & per-tenant policy
Every /v1/... API route requires Authorization: Bearer <secret>. Callers can only list, exec in, or destroy sandboxes created by their own token. Give each agent, CI pipeline, or team its own token with least-privilege rules:
{
"id": "ci-python",
"secret": "use-a-long-random-secret",
"templates": ["python"],
"max_concurrent": 4,
"max_ttl": "20m",
"deny_network": false,
"egress_allow": ["pypi.org", "files.pythonhosted.org"],
"egress_deny": ["evil.pypi.org"],
"allow_secret_override": false,
"audit_read": ""
}
egress_allow/egress_denyscope outbound hosts per token. Global deny rules always win over per-token allow rules.allow_secret_overridedefaults tofalse. Keep it off for AI agents so they cannot bypass the ingress secret firewall.audit_read: "all"grants read-only visibility across all tokens onGET /v1/auditandGET /v1/sandboxes/{id}/report(intended for a dedicated SecOps token).
08Networking & strict egress
Sandboxes have no network interface by default ("networked": false). When a caller requests "networked": true (and the token permits it):
- Moat allocates a dedicated
/30subnet fromnet_pool(default10.62.0.0/16) inside a per-VM network namespace, and releases it on sandbox destroy. nftablesunconditionally drops routes to the host, loopback,169.254.0.0/16(cloud metadata), and RFC-1918 private ranges.- Run
moatdwith-egress-transparent 0.0.0.0:3128and"defaults": { "strict_egress": true }so guest TCP 80/443 is DNATed transparently to the SNI/Host-filtering proxy and the forward chain defaults topolicy drop(allowing only proxy traffic and configured DNS).
nftables require root or systemd capabilities CAP_NET_ADMIN and CAP_SYS_ADMIN. Uncomment the capability lines in /etc/systemd/system/moatd.service when enabling networked sandboxes.
09Unix sockets, TLS/mTLS & jailer
Local-only host (recommended for solo devs and local MCP)
Bind moatd exclusively to a 0660 Unix domain socket with no TCP listener:
# In /etc/moat/moatd.json
"socket": "/run/moat/moatd.sock"
# In systemd ExecStart
ExecStart=/usr/local/bin/moatd -addr "" -config /etc/moat/moatd.json
Remote API with native TLS 1.2+ or mTLS
# In /etc/moat/moatd.json (or via -tls-cert / -tls-key / -tls-client-ca)
"tls_cert": "/etc/moat/server.pem",
"tls_key": "/etc/moat/server.key",
"tls_client_ca": "/etc/moat/clients-ca.pem"
Setting tls_client_ca enforces mutual TLS (RequireAndVerifyClientCert) on every TCP connection in addition to bearer-token auth.
Firecracker jailer confinement
Enable "defaults": { "jail": true, "jail_uid": 1001, "jail_gid": 1001 } (or -jail) so each Firecracker VMM runs chrooted, under a seccomp filter, and with dropped privileges. Requires root or CAP_SYS_ADMIN.
10Image verification & Cosign
At startup, moatd reads images.lock beside each template's images. In the default "image_verify": "enforce" mode:
- Every template covered by
images.lockmust match its kernel and rootfs SHA-256 digests. Failed templates are dropped before serving, andmoatdexits non-zero if zero usable templates remain. - Secondary imported templates that share the verified kernel still have their kernel digest checked.
- Set
"cosign_identity"and"cosign_issuer"(orMOAT_COSIGN_IDENTITY/MOAT_COSIGN_ISSUER) to also verifyimages.lock.sigwith Cosign before startup.
11Audit logs, reports & SIEM forwarding
The audit log is enabled by default (written to /var/lib/moat/audit or audit.dir). Every event is metadata-only (no secrets, file bodies, or stdout) and chained by SHA-256.
# Follow live events
moat audit tail
# Query a specific token or event type
moat audit query -token ci -type secret.block -since -24h
# Rollup everything one sandbox did (works after destruction)
moat audit report sb-7f3a19c4
# Verify the entire hash chain against tampering
moat audit verify
To stream events to Splunk, Datadog, Vector, or a custom webhook collector without slowing API calls, add "webhook_url": "https://siem.internal/moat" under "audit" (or pass -audit-webhook).
12Monitoring, recovery & uninstall
Prometheus metrics
Scrape GET /metrics and alert on:
rate(moat_secret_findings_total{action="block"}[5m]) > 0— credentials hitting the ingress firewall.rate(moat_quota_rejects_total[5m]) > 0— callers hitting concurrency caps.moat_pool_ready == 0— warm pool exhausted (falling back to cold starts).increase(moat_audit_dropped_total[5m]) > 0— audit writer falling behind or disk pressure.
Crash recovery & orphan reaping
moatd automatically reaps orphaned Firecracker processes left by a host crash on startup and logs orphan.reap audit events. To sweep orphans manually:
moat reap
Clean uninstall
moat uninstall --dry-run # preview what will be removed
moat uninstall --keep-data # remove binaries + unit, keep images & audit logs
moat uninstall # full removal (add --user for $HOME installs)