gateway: persistent PSS key + retry resolveSelfOverlay

Two bugs that together blackholed every reverse-PSS message from the
provider when the api ran more than one replica:

1) `pssKeys = generatePssKeypair()` ran per-container, so each replica
   had a different ECIES identity. Provider encrypts its reply to ONE
   of those keys, only that replica can decrypt — so two replicas =
   ~50% of chats hang forever.

2) `resolveSelfOverlay` swallowed errors and returned 0x00…00 when
   Bee wasn't ready yet at boot. That zeroed overlay landed in the
   gateway context AND in the job_notify envelope sent to the provider
   — provider then sent PSS replies into the void.

Fix: load PSS key from PSS_KEY_PATH (shared volume across replicas);
retry getNodeAddresses up to 5 minutes and throw if still zero.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Felix FaerberandClaude Opus 4.7 committed 2026-05-28 11:49:54 +03:00
1 parent 9503b19378
commit fcf184b016
2 files changed
+24 -6

No files matched your search

+6
View File
@@ -58,6 +58,12 @@ const EnvSchema = z.object({
HOSTED_GATEWAY_PORT: z.coerce.number().int().positive().default(8080), HOSTED_GATEWAY_PORT: z.coerce.number().int().positive().default(8080),
LOG_LEVEL: z.enum(['fatal', 'error', 'warn', 'info', 'debug', 'trace']).default('info'), LOG_LEVEL: z.enum(['fatal', 'error', 'warn', 'info', 'debug', 'trace']).default('info'),
// Operator PSS keypair file. All `prod_api` and `prod_worker` replicas
// share this key so providers can encrypt their PSS replies to a single
// identity. The compose mounts a named volume at /data so the key persists
// across deploys and is visible to every replica.
PSS_KEY_PATH: z.string().default('/data/pss.key'),
// Phase 2 — Stripe (optional) // Phase 2 — Stripe (optional)
STRIPE_SECRET_KEY: z.string().optional(), STRIPE_SECRET_KEY: z.string().optional(),
STRIPE_WEBHOOK_SECRET: z.string().optional(), STRIPE_WEBHOOK_SECRET: z.string().optional(),
+18 -6
View File
@@ -14,7 +14,7 @@ import { discoverUsableBatchId, downloadChunk, PssTransport } from '../t4t/lib/s
import { ensureManagedStamp, topUpIfBelow, type ManagedStamp } from '../t4t/lib/stamps'; import { ensureManagedStamp, topUpIfBelow, type ManagedStamp } from '../t4t/lib/stamps';
import { clientTopic } from '../t4t/lib/envelope'; import { clientTopic } from '../t4t/lib/envelope';
import { EciesCipher, jsonDecrypt } from '../t4t/lib/crypto'; import { EciesCipher, jsonDecrypt } from '../t4t/lib/crypto';
import { generatePssKeypair } from '../t4t/lib/keys'; import { loadOrCreatePssKey } from '../t4t/lib/keys';
import { ModelDiscovery } from '../t4t/modes/gateway/models'; import { ModelDiscovery } from '../t4t/modes/gateway/models';
import type { import type {
JobAckBody, JobAckBody,
@@ -89,12 +89,16 @@ async function bootGatewayContext(): Promise<GatewayContext> {
xbzz: env.XBZZ_ADDRESS as Hex, xbzz: env.XBZZ_ADDRESS as Hex,
}); });
// Operator PSS keypair — non-persistent for now; tenants reuse this single // Operator PSS keypair — persisted to disk so every api replica shares the
// operator key (providers see all jobs as coming from the same client PSS // same identity. Without this, each replica generates a fresh keypair at
// identity, which matches the on-chain wallet model). // boot and providers encrypt their PSS replies to a key only one replica
const pssKeys = generatePssKeypair(); // can decrypt, so most chat completions hang.
const pssKeys = loadOrCreatePssKey(env.PSS_KEY_PATH);
const cipher = new EciesCipher(pssKeys.privateKey); const cipher = new EciesCipher(pssKeys.privateKey);
// Block until Bee returns a non-zero overlay; without this, a replica that
// races Bee's boot persists `selfOverlay = 0x00…00` in its gateway context
// and providers send PSS replies into the void.
const selfOverlay = await resolveSelfOverlay(bee); const selfOverlay = await resolveSelfOverlay(bee);
const pss = new PssTransport({ bee, postageBatchId, logger, selfAddress: chain.address }); const pss = new PssTransport({ bee, postageBatchId, logger, selfAddress: chain.address });
@@ -297,10 +301,18 @@ async function resolvePostageBatch(bee: Bee): Promise<string> {
} }
async function resolveSelfOverlay(bee: Bee): Promise<Hex> { async function resolveSelfOverlay(bee: Bee): Promise<Hex> {
const zero = '0x' + '00'.repeat(32);
for (let attempt = 0; attempt < 60; attempt++) {
const info = await bee.getNodeAddresses().catch(() => null); const info = await bee.getNodeAddresses().catch(() => null);
const overlay = info?.overlay?.toString() ?? '00'.repeat(32); const overlay = info?.overlay?.toString();
if (overlay && overlay.replace(/^0x/, '') !== '00'.repeat(32)) {
return ('0x' + overlay.replace(/^0x/, '')) as Hex; return ('0x' + overlay.replace(/^0x/, '')) as Hex;
} }
logger.warn({ attempt }, 'bee.getNodeAddresses returned no overlay yet — retrying in 5s');
await new Promise((r) => setTimeout(r, 5000));
}
throw new Error('bee did not return a non-zero overlay after 60 attempts — refusing to boot with a zeroed selfOverlay');
}
/** Update a GatewayJob row. Direct prisma write (not model.gatewayJob.update) /** Update a GatewayJob row. Direct prisma write (not model.gatewayJob.update)
* because we don't need fan-out for status transitions in v1. */ * because we don't need fan-out for status transitions in v1. */