Seven bugs found by auditing the billing and PSS paths.
1. Reservations were never released on ack/delivery timeout. Both failure
paths in gateway/context.ts marked the GatewayJob terminal *before* calling
releaseReservation, and releaseReservation bailed out on exactly those
statuses — so every failed job permanently inflated the tenant's
reservedXbzzWei and shrank their available balance. Replace the status check
with GatewayJob.reservationReleasedAt as an explicit compare-and-set marker,
which makes the release exactly-once regardless of ordering, and give
settleXbzz the same guard so a job that was released locally and claimed
on-chain anyway can't decrement the reservation twice. The migration returns
the xBZZ already stuck on existing accounts.
2. job_ack / job_deliver were only checked for a valid signature, never for
*whose* signature. Anyone who learned a routing id could sign their own
job_deliver, pointing at a chunk encrypted to the gateway's (published) PSS
key, and hand the tenant an arbitrary completion. Drop any envelope whose
`from` isn't the provider the job was posted to.
3. BigInt(decimal.toString()) throws on any wei value >= 1e21 (1000 xBZZ),
because decimal.js switches to exponential notation there. That took out
GET /accounts/me and, worse, wedged the claim watcher's settle loop. Route
every Decimal→string conversion through toFixed().
4. A cancelJob failure at the ACK deadline left the job with no timer at all,
so the chat request hung forever holding a semaphore slot and a reservation.
Fall through to the delivery deadline instead.
5. `?limit=abc` reached Prisma as `take: NaN` (500). Clamp and fall back.
6. A fractional max_tokens made BigInt() throw; a negative one budgeted a
negative reservation. Validate at the schema and defend in the estimator.
7. model.<entity>.delete() announced the deletion on the queue before
performing it, so a failed delete still told consumers the row was gone.
Verified against a real Postgres: the reservation lifecycle (release-after-
terminal, settle-after-release, concurrent release/settle, idempotent settle)
and the migration's repair of leaked reservations on pre-fix data.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mfkh741kNCZeBdk4xyM9qR
uploadChunk only wrapped withBatchRecovery (402), so a Bee 429 —
returned when concurrent uploads hit the same postage batch (multiple
api replicas + worker) — failed the whole completion instead of backing
off. Wrap the upload in withRateLimitRetry so a transient throttle
becomes a slightly slower success.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TYhvtQe5q6SLAmcCaUk9HD
Bee 2.8's pushsync from homelab to prod consistently routes our trojan
chunks to neighbour ff7adaed instead of prod_bee at ff896a93, even with
both nodes directly peered. The chunk lands at a third-party node that
can't decrypt it, so the gateway never receives provider ack/deliver
envelopes and the chat times out.
Workaround: gateway advertises a clientReplyUrl in job_notify. Provider
POSTs the same signed envelopes there; gateway verifies the signature
(same trust check as the PSS subscription) and dispatches via the same
onEnvelope handler. PSS remains the fallback when no replyUrl is set
(non-hosted clients).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Bee closes the pssSubscribe WebSocket after ~8s of idleness (or on any
transient hiccup). The previous handler only logged the close — so the
gateway silently went deaf to every ack/deliver envelope that arrived
after the first close, breaking every chat whose provider takes more
than a few seconds to respond. Re-open the subscription on close with a
short delay; cancel() now blocks further reopens.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The 1M-token headroom (ported from t4t homelab gateway) makes maxPayment
so large that JobEscrow's concurrency cap fires on a fresh provider
with ~70 xBZZ stake — postJob reverts with InsufficientStakeForJob.
50k tokens still covers chat-template overhead with room to spare, and
brings per-job exposure down ~20x so ~20 concurrent jobs fit comfortably.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two bugs that together blackholed every reverse-PSS message from the
provider when the api ran more than one replica:
1) `pssKeys = generatePssKeypair()` ran per-container, so each replica
had a different ECIES identity. Provider encrypts its reply to ONE
of those keys, only that replica can decrypt — so two replicas =
~50% of chats hang forever.
2) `resolveSelfOverlay` swallowed errors and returned 0x00…00 when
Bee wasn't ready yet at boot. That zeroed overlay landed in the
gateway context AND in the job_notify envelope sent to the provider
— provider then sent PSS replies into the void.
Fix: load PSS key from PSS_KEY_PATH (shared volume across replicas);
retry getNodeAddresses up to 5 minutes and throw if still zero.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous formula computed maxPayment from the estimated input length
plus offering.maxContextTokens — but the on-chain offering for some
provider models has maxContextTokens=0, so output cost was 0 and the
posted maxPayment only covered the (under-estimated) input side. The
provider's claimJob then reverted with PaymentTooHigh, leaving the
gateway hung waiting for a PSS job_deliver that never came.
Match the t4t/container homelab gateway: assume both prompt and
completion can be (max_tokens + 1M-token headroom), budget at the full
split rate, let the JobEscrow contract refund the unused portion. The
provider's actual claim is always smaller, so PaymentTooHigh can't fire.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
swagger({ path: '/' }) renders the UI at root but doesn't register the
/json spec route, so the embedded UI loads then 404s on its own spec.
Keep the plugin defaults and bounce / → /swagger instead.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Was only building + pushing; swarm kept running the previous image
until someone ran service update --force. Mirror the frontend workflow:
ssh in and `docker stack deploy --resolve-image always` so every main
push rolls out.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Was at /swagger by default. Apex 404 was confusing — there's no other
content on the api host, so make the API docs the default landing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Claude