Add stamp-monitor; move Bee to 2.8.1

This is the config that was already running on the host, brought back
into the repo where it belongs, plus the Bee bump it implies.

stamp-monitor watches prod_bee's postage batches and renews them before
they lapse. It exists because nothing did: the gateway's batch fell under
minTtlDays, the gateway tried to renew it itself, the top-up (64.64 xBZZ
on a depth-24 batch) exceeded the wallet, and the throw took down every
route that boots the gateway context. A batch running out was never
supposed to be able to return 500 on /v1/models.

It runs next to Bee rather than reading the chain from elsewhere because
bucket fullness is not on-chain -- stamps are off-chain signatures, so
only a node that saw them knows how full a batch is. A batch can be fully
funded and still refuse writes, and that is the failure the chain cannot
show you.

Bee 2.8.0 -> 2.8.1 for a related reason: 2.8.0 omits utilizationRatio
from /stamps, so a batch at 215 of 256 reported as empty. An absent
number read as zero, and zero means "plenty of room" -- fullness alerts
would have stayed quiet on exactly the batch that needed them. The
monitor now derives the figure either way, but the node should not be the
thing lying. Digest-pinned: this node holds a chequebook and a stake.

BEE_STAMP_MANAGE defaults to false in the example now, and the batch ID
is pinned. Leaving the managed path reachable is what caused the outage,
and an empty BEE_POSTAGE_BATCH_ID is what re-enables it.

The dashboard is deliberately not routed through Traefik -- it can spend
xBZZ. Readme documents the tunnel instead.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YLYiaPYror1vLKApYKrxTu
This commit is contained in:
Felix FaerberandClaude Opus 5 committed 2026-08-19 01:57:18 +03:00
1 parent 7fcd9df9d3
commit b4c62905e6
3 files changed
+113 -5

No files matched your search

+33 -1
View File
@@ -11,7 +11,7 @@ Layout mirrors [cipherdolls/devops/hostinger/](https://github.com/cipherdolls/de
| Stack | File | What it runs |
| --- | --- | --- |
| `system` | `system-stack.yaml` | Traefik (TLS + ingress), Grafana, Loki, Prometheus, node-exporter, cadvisor. Owns the `public` + `internal` overlay networks. |
| `prod` | `prod-stack.yaml` | hosted-gateway api + worker + one-shot prisma migrate, plus postgres / redis / bee / postgres-exporter. |
| `prod` | `prod-stack.yaml` | hosted-gateway api + worker + one-shot prisma migrate, plus postgres / redis / bee / postgres-exporter / stamp-monitor. |
| `dev` | `dev-stack.yaml` | Staging on `dev.t4t-gateway.com` — the `:develop` image tag against its own postgres / redis / bee volumes. |
Supporting configs:
@@ -90,6 +90,38 @@ lives in `/opt/t4t-gateway/.env` (chmod 600). Future work: migrate to a
proper Docker secret backed by the host's secret manager — never an env var
in the stack file.
## Postage batches (stamp-monitor)
The gateway's uploads are paid for by a postage batch that expires. When it
lapsed, the gateway returned 500 on every route — see the comment above the
`stamp-monitor` service for the full chain. `stamp-monitor` now owns renewal:
it polls `prod_bee` every 5 min, tops a batch up when it drops under 14 days,
and refuses any single action over 8 xBZZ.
It runs beside Bee rather than reading the chain because **bucket fullness is
not on-chain** — postage stamps are off-chain signatures, so only a node that
saw them knows how full a batch is. A batch can be fully funded and still
refuse writes.
No Traefik route: it is on `internal` only, and its UI can spend xBZZ. Reach it
over an SSH tunnel instead of publishing it:
```bash
ssh -L 8088:127.0.0.1:8088 root@<host>
docker run --rm --network prod_internal -p 127.0.0.1:8088:8088 \
alpine/socat tcp-listen:8088,fork,reuseaddr tcp-connect:stamp-monitor:3000
# then http://127.0.0.1:8088 with STAMP_MONITOR_ADMIN_TOKEN
```
Quick check without the UI:
```bash
docker service logs --tail 50 prod_stamp-monitor
```
`WEBHOOK_URL` is unset, so alerts (`wallet_low`, `batch_full`, `topup_blocked`)
only reach the logs. Set it to make them reach a person.
## Backups
Postgres volume: