Add stamp-monitor; move Bee to 2.8.1
This is the config that was already running on the host, brought back into the repo where it belongs, plus the Bee bump it implies. stamp-monitor watches prod_bee's postage batches and renews them before they lapse. It exists because nothing did: the gateway's batch fell under minTtlDays, the gateway tried to renew it itself, the top-up (64.64 xBZZ on a depth-24 batch) exceeded the wallet, and the throw took down every route that boots the gateway context. A batch running out was never supposed to be able to return 500 on /v1/models. It runs next to Bee rather than reading the chain from elsewhere because bucket fullness is not on-chain -- stamps are off-chain signatures, so only a node that saw them knows how full a batch is. A batch can be fully funded and still refuse writes, and that is the failure the chain cannot show you. Bee 2.8.0 -> 2.8.1 for a related reason: 2.8.0 omits utilizationRatio from /stamps, so a batch at 215 of 256 reported as empty. An absent number read as zero, and zero means "plenty of room" -- fullness alerts would have stayed quiet on exactly the batch that needed them. The monitor now derives the figure either way, but the node should not be the thing lying. Digest-pinned: this node holds a chequebook and a stake. BEE_STAMP_MANAGE defaults to false in the example now, and the batch ID is pinned. Leaving the managed path reachable is what caused the outage, and an empty BEE_POSTAGE_BATCH_ID is what re-enables it. The dashboard is deliberately not routed through Traefik -- it can spend xBZZ. Readme documents the tunnel instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YLYiaPYror1vLKApYKrxTu
This commit is contained in:
1 parent
7fcd9df9d3
commit
b4c62905e6
3 files changed
+113
-5
No files matched your search
+33
-1
@@ -11,7 +11,7 @@ Layout mirrors [cipherdolls/devops/hostinger/](https://github.com/cipherdolls/de
|
||||
| Stack | File | What it runs |
|
||||
| --- | --- | --- |
|
||||
| `system` | `system-stack.yaml` | Traefik (TLS + ingress), Grafana, Loki, Prometheus, node-exporter, cadvisor. Owns the `public` + `internal` overlay networks. |
|
||||
| `prod` | `prod-stack.yaml` | hosted-gateway api + worker + one-shot prisma migrate, plus postgres / redis / bee / postgres-exporter. |
|
||||
| `prod` | `prod-stack.yaml` | hosted-gateway api + worker + one-shot prisma migrate, plus postgres / redis / bee / postgres-exporter / stamp-monitor. |
|
||||
| `dev` | `dev-stack.yaml` | Staging on `dev.t4t-gateway.com` — the `:develop` image tag against its own postgres / redis / bee volumes. |
|
||||
|
||||
Supporting configs:
|
||||
@@ -90,6 +90,38 @@ lives in `/opt/t4t-gateway/.env` (chmod 600). Future work: migrate to a
|
||||
proper Docker secret backed by the host's secret manager — never an env var
|
||||
in the stack file.
|
||||
|
||||
## Postage batches (stamp-monitor)
|
||||
|
||||
The gateway's uploads are paid for by a postage batch that expires. When it
|
||||
lapsed, the gateway returned 500 on every route — see the comment above the
|
||||
`stamp-monitor` service for the full chain. `stamp-monitor` now owns renewal:
|
||||
it polls `prod_bee` every 5 min, tops a batch up when it drops under 14 days,
|
||||
and refuses any single action over 8 xBZZ.
|
||||
|
||||
It runs beside Bee rather than reading the chain because **bucket fullness is
|
||||
not on-chain** — postage stamps are off-chain signatures, so only a node that
|
||||
saw them knows how full a batch is. A batch can be fully funded and still
|
||||
refuse writes.
|
||||
|
||||
No Traefik route: it is on `internal` only, and its UI can spend xBZZ. Reach it
|
||||
over an SSH tunnel instead of publishing it:
|
||||
|
||||
```bash
|
||||
ssh -L 8088:127.0.0.1:8088 root@<host>
|
||||
docker run --rm --network prod_internal -p 127.0.0.1:8088:8088 \
|
||||
alpine/socat tcp-listen:8088,fork,reuseaddr tcp-connect:stamp-monitor:3000
|
||||
# then http://127.0.0.1:8088 with STAMP_MONITOR_ADMIN_TOKEN
|
||||
```
|
||||
|
||||
Quick check without the UI:
|
||||
|
||||
```bash
|
||||
docker service logs --tail 50 prod_stamp-monitor
|
||||
```
|
||||
|
||||
`WEBHOOK_URL` is unset, so alerts (`wallet_low`, `batch_full`, `topup_blocked`)
|
||||
only reach the logs. Set it to make them reach a person.
|
||||
|
||||
## Backups
|
||||
|
||||
Postgres volume:
|
||||
|
||||
Reference in new issue
Block a user