Felix FaerberandClaude Opus 5
b4c62905e6
This is the config that was already running on the host, brought back into the repo where it belongs, plus the Bee bump it implies. stamp-monitor watches prod_bee's postage batches and renews them before they lapse. It exists because nothing did: the gateway's batch fell under minTtlDays, the gateway tried to renew it itself, the top-up (64.64 xBZZ on a depth-24 batch) exceeded the wallet, and the throw took down every route that boots the gateway context. A batch running out was never supposed to be able to return 500 on /v1/models. It runs next to Bee rather than reading the chain from elsewhere because bucket fullness is not on-chain -- stamps are off-chain signatures, so only a node that saw them knows how full a batch is. A batch can be fully funded and still refuse writes, and that is the failure the chain cannot show you. Bee 2.8.0 -> 2.8.1 for a related reason: 2.8.0 omits utilizationRatio from /stamps, so a batch at 215 of 256 reported as empty. An absent number read as zero, and zero means "plenty of room" -- fullness alerts would have stayed quiet on exactly the batch that needed them. The monitor now derives the figure either way, but the node should not be the thing lying. Digest-pinned: this node holds a chequebook and a stake. BEE_STAMP_MANAGE defaults to false in the example now, and the batch ID is pinned. Leaving the managed path reachable is what caused the outage, and an empty BEE_POSTAGE_BATCH_ID is what re-enables it. The dashboard is deliberately not routed through Traefik -- it can spend xBZZ. Readme documents the tunnel instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YLYiaPYror1vLKApYKrxTu
Hostinger (production) stacks
Docker Swarm stack files for the t4t hosted-gateway, deployed to a single
Hostinger VPS at ssh root@195.35.25.26 and reachable at
t4t-gateway.com.
Layout mirrors cipherdolls/devops/hostinger/.
Stacks
| Stack | File | What it runs |
|---|---|---|
system |
system-stack.yaml |
Traefik (TLS + ingress), Grafana, Loki, Prometheus, node-exporter, cadvisor. Owns the public + internal overlay networks. |
prod |
prod-stack.yaml |
hosted-gateway api + worker + one-shot prisma migrate, plus postgres / redis / bee / postgres-exporter / stamp-monitor. |
dev |
dev-stack.yaml |
Staging on dev.t4t-gateway.com — the :develop image tag against its own postgres / redis / bee volumes. |
Supporting configs:
loki.yaml— Loki single-binary filesystem configprometheus.yml— scrape targets (traefik, grafana, loki, node-exporter, postgres-exporter, cadvisor, hosted-gateway api).env.example— copy to.envon the host, fill inOPERATOR_PRIVATE_KEY/JWT_SECRET_KEY/ etc.
DNS records (Hostinger DNS panel)
Point all of these at 195.35.25.26:
A t4t-gateway.com 195.35.25.26
A www.t4t-gateway.com 195.35.25.26
A api.t4t-gateway.com 195.35.25.26
A dev.t4t-gateway.com 195.35.25.26
A traefik.t4t-gateway.com 195.35.25.26
A grafana.t4t-gateway.com 195.35.25.26
A loki.t4t-gateway.com 195.35.25.26
A prometheus.t4t-gateway.com 195.35.25.26
One-time host setup
ssh root@195.35.25.26
# Docker + swarm
curl -fsSL https://get.docker.com | sh
docker swarm init --advertise-addr 195.35.25.26
# Loki driver (so container logs ship to Loki via the `loki` logging driver)
docker plugin install grafana/loki-docker-driver:2.9.7 \
--alias loki --grant-all-permissions
# Drop the stack files + configs on the host
mkdir -p /opt/t4t-gateway && cd /opt/t4t-gateway
# rsync this directory up here (or `git clone` the devops repo)
cp .env.example .env && chmod 600 .env
$EDITOR .env # fill in OPERATOR_PRIVATE_KEY, JWT_SECRET_KEY, ...
Deploy
The system stack creates the shared overlay networks, so deploy it first.
cd /opt/t4t-gateway
# 1. Networks + ingress + monitoring
docker stack deploy -c system-stack.yaml system
# 2. App + datastores
docker stack deploy -c prod-stack.yaml prod
# (optional) staging on dev.t4t-gateway.com
docker stack deploy -c dev-stack.yaml dev
Re-deploying after a new image tag:
docker service update --image ghcr.io/token-for-token/hosted-gateway:main \
--force prod_api
docker service update --image ghcr.io/token-for-token/hosted-gateway:main \
--force prod_worker
Secrets
OPERATOR_PRIVATE_KEY is the on-chain identity for every tenant. For now it
lives in /opt/t4t-gateway/.env (chmod 600). Future work: migrate to a
proper Docker secret backed by the host's secret manager — never an env var
in the stack file.
Postage batches (stamp-monitor)
The gateway's uploads are paid for by a postage batch that expires. When it
lapsed, the gateway returned 500 on every route — see the comment above the
stamp-monitor service for the full chain. stamp-monitor now owns renewal:
it polls prod_bee every 5 min, tops a batch up when it drops under 14 days,
and refuses any single action over 8 xBZZ.
It runs beside Bee rather than reading the chain because bucket fullness is not on-chain — postage stamps are off-chain signatures, so only a node that saw them knows how full a batch is. A batch can be fully funded and still refuse writes.
No Traefik route: it is on internal only, and its UI can spend xBZZ. Reach it
over an SSH tunnel instead of publishing it:
ssh -L 8088:127.0.0.1:8088 root@<host>
docker run --rm --network prod_internal -p 127.0.0.1:8088:8088 \
alpine/socat tcp-listen:8088,fork,reuseaddr tcp-connect:stamp-monitor:3000
# then http://127.0.0.1:8088 with STAMP_MONITOR_ADMIN_TOKEN
Quick check without the UI:
docker service logs --tail 50 prod_stamp-monitor
WEBHOOK_URL is unset, so alerts (wallet_low, batch_full, topup_blocked)
only reach the logs. Set it to make them reach a person.
Backups
Postgres volume:
docker run --rm \
-v prod_postgres-data:/from_volume \
-v $(pwd):/workdir \
alpine \
sh -c "cd /from_volume && tar cvf /workdir/prod_postgres-data.tar ."
scp root@195.35.25.26:/opt/t4t-gateway/prod_postgres-data.tar ~/Downloads
Bee data volume (postage batch / swarm keys — DO NOT lose this):
docker run --rm \
-v prod_bee-data:/from_volume \
-v $(pwd):/workdir \
alpine \
sh -c "cd /from_volume && tar cvf /workdir/prod_bee-data.tar ."
Troubleshooting
docker stack services system # health of ingress + monitoring
docker stack services prod # health of gateway + data
docker service logs -f prod_api
docker service logs -f prod_worker
# Replay a failed Prisma migration:
docker exec -it $(docker ps -q -f name=prod_prisma) sh
bunx prisma migrate resolve --rolled-back <migration_name>