Felix FaerberandClaude Opus 5
b4c62905e6
This is the config that was already running on the host, brought back into the repo where it belongs, plus the Bee bump it implies. stamp-monitor watches prod_bee's postage batches and renews them before they lapse. It exists because nothing did: the gateway's batch fell under minTtlDays, the gateway tried to renew it itself, the top-up (64.64 xBZZ on a depth-24 batch) exceeded the wallet, and the throw took down every route that boots the gateway context. A batch running out was never supposed to be able to return 500 on /v1/models. It runs next to Bee rather than reading the chain from elsewhere because bucket fullness is not on-chain -- stamps are off-chain signatures, so only a node that saw them knows how full a batch is. A batch can be fully funded and still refuse writes, and that is the failure the chain cannot show you. Bee 2.8.0 -> 2.8.1 for a related reason: 2.8.0 omits utilizationRatio from /stamps, so a batch at 215 of 256 reported as empty. An absent number read as zero, and zero means "plenty of room" -- fullness alerts would have stayed quiet on exactly the batch that needed them. The monitor now derives the figure either way, but the node should not be the thing lying. Digest-pinned: this node holds a chequebook and a stake. BEE_STAMP_MANAGE defaults to false in the example now, and the batch ID is pinned. Leaving the managed path reachable is what caused the outage, and an empty BEE_POSTAGE_BATCH_ID is what re-enables it. The dashboard is deliberately not routed through Traefik -- it can spend xBZZ. Readme documents the tunnel instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YLYiaPYror1vLKApYKrxTu
161 lines
5.1 KiB
Markdown
161 lines
5.1 KiB
Markdown
# Hostinger (production) stacks
|
|
|
|
Docker Swarm stack files for the t4t hosted-gateway, deployed to a single
|
|
Hostinger VPS at `ssh root@195.35.25.26` and reachable at
|
|
[t4t-gateway.com](https://t4t-gateway.com).
|
|
|
|
Layout mirrors [cipherdolls/devops/hostinger/](https://github.com/cipherdolls/devops).
|
|
|
|
## Stacks
|
|
|
|
| Stack | File | What it runs |
|
|
| --- | --- | --- |
|
|
| `system` | `system-stack.yaml` | Traefik (TLS + ingress), Grafana, Loki, Prometheus, node-exporter, cadvisor. Owns the `public` + `internal` overlay networks. |
|
|
| `prod` | `prod-stack.yaml` | hosted-gateway api + worker + one-shot prisma migrate, plus postgres / redis / bee / postgres-exporter / stamp-monitor. |
|
|
| `dev` | `dev-stack.yaml` | Staging on `dev.t4t-gateway.com` — the `:develop` image tag against its own postgres / redis / bee volumes. |
|
|
|
|
Supporting configs:
|
|
|
|
- `loki.yaml` — Loki single-binary filesystem config
|
|
- `prometheus.yml` — scrape targets (traefik, grafana, loki, node-exporter,
|
|
postgres-exporter, cadvisor, hosted-gateway api)
|
|
- `.env.example` — copy to `.env` on the host, fill in
|
|
`OPERATOR_PRIVATE_KEY` / `JWT_SECRET_KEY` / etc.
|
|
|
|
## DNS records (Hostinger DNS panel)
|
|
|
|
Point all of these at `195.35.25.26`:
|
|
|
|
```
|
|
A t4t-gateway.com 195.35.25.26
|
|
A www.t4t-gateway.com 195.35.25.26
|
|
A api.t4t-gateway.com 195.35.25.26
|
|
A dev.t4t-gateway.com 195.35.25.26
|
|
A traefik.t4t-gateway.com 195.35.25.26
|
|
A grafana.t4t-gateway.com 195.35.25.26
|
|
A loki.t4t-gateway.com 195.35.25.26
|
|
A prometheus.t4t-gateway.com 195.35.25.26
|
|
```
|
|
|
|
## One-time host setup
|
|
|
|
```bash
|
|
ssh root@195.35.25.26
|
|
|
|
# Docker + swarm
|
|
curl -fsSL https://get.docker.com | sh
|
|
docker swarm init --advertise-addr 195.35.25.26
|
|
|
|
# Loki driver (so container logs ship to Loki via the `loki` logging driver)
|
|
docker plugin install grafana/loki-docker-driver:2.9.7 \
|
|
--alias loki --grant-all-permissions
|
|
|
|
# Drop the stack files + configs on the host
|
|
mkdir -p /opt/t4t-gateway && cd /opt/t4t-gateway
|
|
# rsync this directory up here (or `git clone` the devops repo)
|
|
cp .env.example .env && chmod 600 .env
|
|
$EDITOR .env # fill in OPERATOR_PRIVATE_KEY, JWT_SECRET_KEY, ...
|
|
```
|
|
|
|
## Deploy
|
|
|
|
The system stack creates the shared overlay networks, so deploy it first.
|
|
|
|
```bash
|
|
cd /opt/t4t-gateway
|
|
|
|
# 1. Networks + ingress + monitoring
|
|
docker stack deploy -c system-stack.yaml system
|
|
|
|
# 2. App + datastores
|
|
docker stack deploy -c prod-stack.yaml prod
|
|
|
|
# (optional) staging on dev.t4t-gateway.com
|
|
docker stack deploy -c dev-stack.yaml dev
|
|
```
|
|
|
|
Re-deploying after a new image tag:
|
|
|
|
```bash
|
|
docker service update --image ghcr.io/token-for-token/hosted-gateway:main \
|
|
--force prod_api
|
|
docker service update --image ghcr.io/token-for-token/hosted-gateway:main \
|
|
--force prod_worker
|
|
```
|
|
|
|
## Secrets
|
|
|
|
`OPERATOR_PRIVATE_KEY` is the on-chain identity for every tenant. For now it
|
|
lives in `/opt/t4t-gateway/.env` (chmod 600). Future work: migrate to a
|
|
proper Docker secret backed by the host's secret manager — never an env var
|
|
in the stack file.
|
|
|
|
## Postage batches (stamp-monitor)
|
|
|
|
The gateway's uploads are paid for by a postage batch that expires. When it
|
|
lapsed, the gateway returned 500 on every route — see the comment above the
|
|
`stamp-monitor` service for the full chain. `stamp-monitor` now owns renewal:
|
|
it polls `prod_bee` every 5 min, tops a batch up when it drops under 14 days,
|
|
and refuses any single action over 8 xBZZ.
|
|
|
|
It runs beside Bee rather than reading the chain because **bucket fullness is
|
|
not on-chain** — postage stamps are off-chain signatures, so only a node that
|
|
saw them knows how full a batch is. A batch can be fully funded and still
|
|
refuse writes.
|
|
|
|
No Traefik route: it is on `internal` only, and its UI can spend xBZZ. Reach it
|
|
over an SSH tunnel instead of publishing it:
|
|
|
|
```bash
|
|
ssh -L 8088:127.0.0.1:8088 root@<host>
|
|
docker run --rm --network prod_internal -p 127.0.0.1:8088:8088 \
|
|
alpine/socat tcp-listen:8088,fork,reuseaddr tcp-connect:stamp-monitor:3000
|
|
# then http://127.0.0.1:8088 with STAMP_MONITOR_ADMIN_TOKEN
|
|
```
|
|
|
|
Quick check without the UI:
|
|
|
|
```bash
|
|
docker service logs --tail 50 prod_stamp-monitor
|
|
```
|
|
|
|
`WEBHOOK_URL` is unset, so alerts (`wallet_low`, `batch_full`, `topup_blocked`)
|
|
only reach the logs. Set it to make them reach a person.
|
|
|
|
## Backups
|
|
|
|
Postgres volume:
|
|
|
|
```bash
|
|
docker run --rm \
|
|
-v prod_postgres-data:/from_volume \
|
|
-v $(pwd):/workdir \
|
|
alpine \
|
|
sh -c "cd /from_volume && tar cvf /workdir/prod_postgres-data.tar ."
|
|
|
|
scp root@195.35.25.26:/opt/t4t-gateway/prod_postgres-data.tar ~/Downloads
|
|
```
|
|
|
|
Bee data volume (postage batch / swarm keys — DO NOT lose this):
|
|
|
|
```bash
|
|
docker run --rm \
|
|
-v prod_bee-data:/from_volume \
|
|
-v $(pwd):/workdir \
|
|
alpine \
|
|
sh -c "cd /from_volume && tar cvf /workdir/prod_bee-data.tar ."
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
```bash
|
|
docker stack services system # health of ingress + monitoring
|
|
docker stack services prod # health of gateway + data
|
|
docker service logs -f prod_api
|
|
docker service logs -f prod_worker
|
|
|
|
# Replay a failed Prisma migration:
|
|
docker exec -it $(docker ps -q -f name=prod_prisma) sh
|
|
bunx prisma migrate resolve --rolled-back <migration_name>
|
|
```
|