The ${STAMP_MONITOR_ADMIN_TOKEN} form rendered correctly under `docker
stack config` -- ADMIN_TOKEN: len=64 -- and still deployed with an EMPTY
value, twice, on a shell that had the variable exported. The result is a
service that starts, polls, alerts and serves its dashboard while every
admin route returns 503: it looks healthy and can show nothing.
A secret is mounted as a file, so there is no substitution left to
silently resolve to "". Same shape the homelab monitor already uses.
Create it once on the host:
printf %s "$TOKEN" | docker secret create stamp_monitor_admin_token -
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YLYiaPYror1vLKApYKrxTu
The gateway's /bee page shows a summary of the node. This exposes the
monitor behind it, which is where batch fullness, the bucket map, wallet
and chequebook runway, and the ledger of every top-up made or refused
actually live.
Left unrouted originally on the reasoning that its UI can spend xBZZ.
That was the wrong test. The protection is that the app fails CLOSED --
every /api/admin route demands an x-admin-token and returns 503 when
ADMIN_TOKEN is unset -- not that the port was hard to reach. Relying on
the latter is how the batch went unwatched long enough to take the
gateway down; a dashboard nobody can open is a dashboard nobody reads.
Bee itself stays unpublished. Reaching it through the monitor's
passthrough still requires the admin token, which is the whole reason the
monitor sits in front of it.
Also bumps the image to pick up the fix that makes Settings caps bind
manual spending, not just the poller.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YLYiaPYror1vLKApYKrxTu
Both found by the dashboard not working after the deploy.
`docker stack deploy` expands ${STAMP_MONITOR_ADMIN_TOKEN} from the shell
and does not read .env for substitution, so a deploy that forgets to
source it ships an empty ADMIN_TOKEN. The service still starts, polls and
alerts normally -- only the dashboard is dead, with a 503 that reads like
the app is broken rather than like the deploy was. That is exactly the
kind of failure worth writing down instead of rediscovering.
The API also takes x-admin-token, not Authorization: Bearer.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YLYiaPYror1vLKApYKrxTu
Found by following my own instructions and having them fail.
- Deploys run from /root/devops/hostinger, a clone of this repo, not from
/opt/t4t-gateway, which does not exist on the host. `git pull` is the
deploy step; the readme was telling people to rsync into a path that
isn't used, which is how the stamp-monitor service ended up edited on
the host and missing from this repo for a day.
- `internal` is not an attachable overlay, so `docker run --network
internal ...` is refused outright. Query Bee through a container that
is already on the network instead.
- The monitor listens on 3000, not 8088.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YLYiaPYror1vLKApYKrxTu
This is the config that was already running on the host, brought back
into the repo where it belongs, plus the Bee bump it implies.
stamp-monitor watches prod_bee's postage batches and renews them before
they lapse. It exists because nothing did: the gateway's batch fell under
minTtlDays, the gateway tried to renew it itself, the top-up (64.64 xBZZ
on a depth-24 batch) exceeded the wallet, and the throw took down every
route that boots the gateway context. A batch running out was never
supposed to be able to return 500 on /v1/models.
It runs next to Bee rather than reading the chain from elsewhere because
bucket fullness is not on-chain -- stamps are off-chain signatures, so
only a node that saw them knows how full a batch is. A batch can be fully
funded and still refuse writes, and that is the failure the chain cannot
show you.
Bee 2.8.0 -> 2.8.1 for a related reason: 2.8.0 omits utilizationRatio
from /stamps, so a batch at 215 of 256 reported as empty. An absent
number read as zero, and zero means "plenty of room" -- fullness alerts
would have stayed quiet on exactly the batch that needed them. The
monitor now derives the figure either way, but the node should not be the
thing lying. Digest-pinned: this node holds a chequebook and a stake.
BEE_STAMP_MANAGE defaults to false in the example now, and the batch ID
is pinned. Leaving the managed path reachable is what caused the outage,
and an empty BEE_POSTAGE_BATCH_ID is what re-enables it.
The dashboard is deliberately not routed through Traefik -- it can spend
xBZZ. Readme documents the tunnel instead.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YLYiaPYror1vLKApYKrxTu
Bind two ready-to-use dashboards at grafana.t4t-gateway.com:
- Logs — All services: per-service log volume timeseries + live tail,
with a service selector and a free-text search box.
- Hosted-gateway API — errors & PSS inbox: error rate, ack/deliver
counters, and a live tail filtered to PSS inbox traffic so the
webhook fallback is observable.
Datasources + dashboard providers are mounted via Docker configs from
hostinger/grafana/ so they survive grafana container restarts and live
in git rather than the Grafana UI.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
bootnode-mode is required by Bee 2.8 to honor --static-nodes, but
bootnode-mode collapses normal peer discovery (homelab dropped from
100+ peers to 2, depth jumped to 31). Use --bootnode to seed an
initial dial to the other gateway instead — that's just a hint, doesn't
change inbound behavior, and leaves hive discovery intact.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Bee 2.8 rejects static-nodes config with 'can only be configured on
bootnodes' unless --bootnode-mode is also set. The side-effect of
bootnode-mode (always accepting inbound connections) is fine here —
the gateway pair only matters to each other.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Bee 2.8's --static-nodes flag takes the 32-byte Swarm overlay address
(kademlia identifier) and looks the node up via DHT — not a libp2p
multiaddr. The multiaddr form crashes boot with
"invalid swarm address ... configured for static node".
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
PSS chunks targeting prod's ff89* neighborhood are being routed by
homelab pushsync to random peers in the ff prefix (e.g. ff7adaed) rather
than prod_bee at ff896a93, because Bee 2.8's routing table evicts
prod_bee from homelab's bin-1 when other ff peers appear. Same problem
in reverse — prod has no protection on its homelab connection.
--static-nodes keeps the libp2p connection alive across kademlia churn.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Pairs with hosted-gateway commit fcf184b. The api/worker containers now
read /data/pss.key on boot — without a shared volume they'd each get
their own PSS identity per container, which is what was hanging PSS
replies from the provider.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Homelab migrated to a fresh data dir with --target-neighborhood matching
its original swarmOverlay, so its on-chain provider registration stays
valid. Both ends on 2.8 now speak the same v15 handshake protocol; the
prior --bootnode workaround for v14 isolation is no longer needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The default mainnet.ethswarm.org bootnodes all reject Bee 2.7.x's
v14 handshake — prod Bee was stuck at 0 peers. Point --bootnode at
the homelab node, which is the same version and reachable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bee 2.8 changed the swarm handshake protocol; homelab Bee can't move
to 2.8 (silent exit 2 on the existing data dir) so the two nodes
couldn't directly /connect — PSS routing kept failing. Match versions
to restore peering.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bee dashboard reports reachability=Private and underlay only lists
172.x/127.0.0.1 — peers see our node as unreachable, PSS messages
won't route. Setting BEE_NAT_ADDR to the host's public IP+port lets
libp2p autonat confirm reachability.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Light nodes can dial the network but PSS delivery requires nodes to
participate as forwarders. Job 0x7355…99fe stayed Pending on-chain
because the provider never received the PSS notify.
Trade-off: full nodes consume more disk + bandwidth and need a funded
chequebook to swap with peers (currently 0 — may need a top-up).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Try the newest version again now that port 1634 is published and the
Hostinger firewall lets inbound through.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2.7.1 and 2.8.0 both get rejected by the mainnet bootnodes with
"protocols not supported: [/swarm/handshake/14.0.0/handshake]".
2.7.2-rc1 is the version bee.ffaerber.duckdns.org runs with 25 live
peers, so try that.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per request — don't fall back to the remote Bee. Local prod_bee will
need its peering fixed before /v1/chat/completions can dispatch.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Local prod_bee can't peer with the Swarm mainnet (bootnode handshake
mismatch — see prior commits pinning 2.7.1). Remote node at
bee.ffaerber.duckdns.org is 2.7.2-rc1 with 25 live peers + usable
stamps, so PSS dispatch works there. Keeping prod_bee in the stack
for now (idle but harmless) so we can swap back once mainnet bootnodes
are usable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The latest tag (currently 2.8.0) speaks /swarm/handshake/14.0.0 which
the live mainnet bootnodes reject:
protocols not supported: [/swarm/handshake/14.0.0/handshake]
2.7.1 was the last release before the bump. Node stays isolated on 2.8
no matter how many peers it dials, so PSS routing never works.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bee was listening on :1634 inside the container but the swarm service
had no ports: block, so external peers couldn't connect. Outbound-only
peering isn't enough for PSS forwarding.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add `frontend` service running ghcr.io/token-for-token/hosted-gateway-frontend
behind Traefik on t4t-gateway.com + www.t4t-gateway.com. API router drops
the apex hostnames; api.t4t-gateway.com remains the gateway.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Prisma 7 dropped --browser none and hardcodes spawn('xdg-open'). Alpine
doesn't ship xdg-utils, so the spawn ENOENT bubbles to process exit and
the studio task fails after the server has already bound to :5555.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Run `prisma studio` alongside the migrate step so the prod_prisma service
serves the data browser. Behind Traefik basicauth (admin-auth middleware
defined on the traefik service).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Was added to debug the v3 multi-host parse error. Dashboard remains
reachable behind basicauth at traefik.t4t-gateway.com.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bee 2.8's BEE_BLOCKCHAIN_RPC_ENDPOINT env var doesn't bind (chain stays
disabled, node exits with "factory fail: chain disabled"). The CLI flag
works. BEE_CHAIN_ENABLE is also redundant — providing the endpoint
implicitly enables chain.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bee 2.x renamed BEE_SWAP_ENDPOINT to BEE_BLOCKCHAIN_RPC_ENDPOINT. With
the old name, Bee silently disables the chain backend and shuts down
with `factory fail: chain disabled`. Wire the endpoint inline so it
doesn't depend on the host .env.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
v3 limits Host() to one parameter; chain with || for multiple. Error
seen in the api@swarm router: "unexpected number of parameters; got 3".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Why: prod_api router triggers ACME (cert issued for api.t4t-gateway.com)
but proxied requests return 404. Need the api endpoint to inspect why.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
v3 split the swarm provider out of `--providers.docker.swarmmode`. New
flags use `--providers.swarm.*`. v2.11.22 is well past its EoL.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors cipherdolls/devops layout. Hostinger Swarm stacks for the t4t
hosted-gateway and local docker-compose for development + e2e tests.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
