State: volumes, secrets & the backup pipeline

Volumes, config & secrets

The named volumes hold everything you can't afford to lose; the config files and the Postgres password are supplied from the host at read-only. Note that the same container often mounts a volume at a different access level than its neighbours — pub-files is rw for app but ro for web.

flowchart TB
  subgraph hostfs["Host filesystem — your deployment dir"]
    playconf@{ shape: doc, label: "play-framework.conf" }
    sitesconf@{ shape: doc, label: "sites-enabled/" }
    pgsecret@{ shape: doc, label: "postgres_password.txt" }
  end

  web["`**web**
_reverse proxy_`"]
  app["`**app**
_forum logic_`"]
  cache["`**cache**
_redis_`"]
  rdb["`**rdb**
_postgres_`"]
  search["`**search**
_elasticsearch_`"]

  subgraph vols["Named volumes (<project>_*)"]
    pgdata[("`**pg-data**
_irreplaceable_`")]
    esdata[("`**es-data + es-logs**
_rebuildable index_`")]
    redisdata[("`**redis-data**
_ephemeral cache_`")]
    pubfiles[("`**pub-files**
_public uploads_`")]
    privfiles[("`**priv-files**
_private uploads_`")]
    webgen[("`**web-generated**
_auto-ssl ACME acct key_`")]
  end

  sitesconf -->|"ro"| web
  playconf -->|"ro"| app
  pgsecret -->|"secret"| app
  pgsecret -->|"secret"| rdb

  web ---|"ro"| pubfiles
  web --- webgen
  app ---|"rw"| pubfiles
  app ---|"rw"| privfiles
  cache --- redisdata
  rdb --- pgdata
  search --- esdata

  classDef front fill:#dcfce7,stroke:#16a34a,color:#14532d
  classDef appt fill:#e0e7ff,stroke:#6366f1,color:#312e81
  classDef data fill:#fef3c7,stroke:#d97706,color:#78350f
  classDef conf fill:#dbeafe,stroke:#2563eb,color:#1e3a5f
  class pgdata,esdata,redisdata,pubfiles,privfiles,webgen data
  class web front
  class app,cache,rdb,search appt
  class playconf,sitesconf,pgsecret conf

• pg-data (rdb) — the PostgreSQL cluster; the talkyard database is your entire forum: users, pages, posts, votes, settings. This is the single most important volume.
• es-data + es-logs (search) — the Elasticsearch full-text index. Derived data: it can be rebuilt by re-indexing from Postgres, so it isn't part of the backup set.
• redis-data (cache) — Redis persistence. Holds ephemeral-ish state (presence, watchbar, one-time login secrets, link-preview cache), not authoritative forum content.
• pub-files — public uploads (avatars, attachments). Mounted rw by app (which writes them) and ro by web (which serves the bytes directly).
• priv-files — private uploads. Mounted rw by app only. Note the backup caveat below.
• web-generated (web) — storage for the built-in lua-resty-auto-ssl ACME account key. Dormant if you terminate TLS in a reverse proxy in front of web; still present.
• Bind mounts (ro): conf/web/sites-enabled/ into web, and conf/app/play-framework.conf into app. These are your config, read once at container start.
• Docker secret: postgres_password.txt is delivered as a file-based secret to app and rdb (and to the backup job) rather than an environment variable.

The backup pipeline

Talkyard ships a stock backup service defined under profiles: [backup], so it never auto-starts — it's a one-shot container you invoke on demand (docker compose run --rm). Point your own scheduler at it: a cron entry or a systemd timer, on whatever schedule you like. Each run produces a self-contained set of archives in a bind-mounted output directory.

flowchart LR
  sched["`**Your scheduler**
_cron / systemd timer_
(any schedule you like)`"]
  backup["`**backup container**
_docker compose run --rm_
profiles: [backup] one-shot`"]
  rdb[("`**rdb**
_postgres cluster_`")]
  pubfiles[("`**pub-files**
_public uploads_`")]
  privfiles[("`**priv-files**
_private uploads_`")]
  redisdata[("`**redis-data**
_dump.rdb_`")]
  projdir@{ shape: doc, label: "project dir (conf + .env + compose + secrets/)" }
  bksecret@{ shape: doc, label: "backup_password.txt" }
  bkdir[("`**backup archives dir**
_rw bind mount_`")]

  sched ==>|"/ty/backup.sh"| backup
  rdb ==>|"pg_dumpall + gzip<br/>(all dbs + roles)"| backup
  pubfiles ==>|"ro — only uploads/<br/>subdir archived"| backup
  redisdata ==>|"ro — dump.rdb"| backup
  projdir ==>|"ro — whole dir tarred into config<br/>archive incl. secrets: why GPG matters"| backup
  bksecret -->|"secret — GPG passphrase"| backup
  privfiles -.->|"mounted ro but NEVER archived<br/>(known gap)"| backup
  backup ==>|"GPG AES256 only while backup_password<br/>non-empty, else SILENT plaintext"| bkdir

  classDef data fill:#fef3c7,stroke:#d97706,color:#78350f
  classDef conf fill:#dbeafe,stroke:#2563eb,color:#1e3a5f
  classDef idle fill:#f3f4f6,stroke:#9ca3af,color:#4b5563,stroke-dasharray:5 5
  classDef world fill:#f1f5f9,stroke:#64748b,color:#334155
  class rdb,pubfiles,redisdata,privfiles,bkdir data
  class projdir,bksecret conf
  class backup idle
  class sched world

• What the script dumps. /ty/backup.sh runs pg_dumpall against rdb (piped through gzip) — that captures all databases plus roles, not a single-database pg_dump. Redis is captured from its dump.rdb.
• Uploads. From pub-files, only the uploads/ subdirectory is archived (read-only mount).
• The config archive contains your secrets. The whole project directory — conf/, .env, the compose files, and the secrets/ directory — is tarred (read-only) into a config archive. That means the archive includes your Postgres password and any other secrets on disk. This is the reason encryption matters.
• Optional GPG-at-rest. If the backup_password secret (secrets/backup_password.txt) is non-empty, the output is encrypted with GPG AES256. Caveat: if that secret is empty, the script does not fail — it silently falls back to writing the archives (including the secrets-bearing config archive) in plaintext. Set a passphrase, or your at-rest backups expose everything.
• Output. Archives land in a read-write bind-mounted directory on the host. Retention/pruning of old archives, and copying archives off-box, are things you arrange yourself around this one-shot job.

Known caveat: priv-files is mounted but not archived

priv-files (private uploads) is mounted into the backup container read-only, but the stock backup.sh never reads it — nothing from priv-files ends up in any archive. If your site stores private uploads, the default backup does not protect them; you'll want to archive that volume separately until this gap is closed upstream.

Summary