a month ago
Our Node worker on Railway receives webhook callbacks from Inngest at /api/inngest. Approximately 3-5% of these callbacks fail with HTTP 401 "Invalid signature" from the Inngest SDK. We've correlated 100% of failures with fresh TCP connections (non-zero DNSLookup in Inngest's outbound request trace) and 100% of successes with reused connections (DNSLookup=0). Successful and failed requests both show Server: railway-hikari and X-Railway-Edge: railway/us-east4-eqdc4a. Can you investigate whether Railway's edge handles request bodies differently on connection establishment vs. on reused connections? This breaks HMAC verification on the worker side.
1 Replies
a month ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • about 1 month ago
a month ago
Fixes
Fix A — Use a raw body parser that ignores encoding headers (recommended)
The Inngest SDK for Node uses the body it receives from Express/Node's request stream. Make sure you're not running express.json() before the Inngest route — if you are, the body has already been parsed and re-serialized, which can silently change byte content (e.g., key ordering, whitespace).
// ❌ This corrupts HMAC verification if mounted globally before /api/inngest
app.use(express.json());
// ✅ Mount body parser only on routes that need it, AFTER /api/inngest
app.use('/api/inngest', serve({ client: inngest, functions: [...] }));
app.use(express.json()); // all other routes
Fix B — Force connection reuse from Inngest's side
Not directly in your control, but you can reduce the fresh-connection rate by ensuring your worker responds with Connection: keep-alive and sets a reasonable Keep-Alive: timeout=60 header. Railway may already do this, but verify it's not stripping keep-alive headers on first-connection responses.
Fix C — Implement a signature verification wrapper that normalizes the body
import { createHmac } from 'crypto';
function verifyInngestSignature(rawBody: Buffer, sig: string, signingKey: string): boolean {
const ts = sig.match(/t=(\d+)/)?.[1];
const mac = sig.match(/s=([a-f0-9]+)/)?.[1];
if (!ts || !mac) return false;
const body = rawBody.toString('utf8'); // explicit UTF-8, no BOM handling
const expected = createHmac('sha256', signingKey)
.update(`${ts}.${body}`)
.digest('hex');return expected === mac;
}
And capture the raw body before any middleware touches it:
app.use('/api/inngest', express.raw({ type: '/' }), (req, res, next) => {
// req.body is now a Buffer, untouched
next();
});
app.use('/api/inngest', express.raw({ type: '/' }), (req, res, next) => {
// req.body is now a Buffer, untouched
next();
});
Fix D — Railway-specific: disable response buffering
In your railway.toml or Railway service settings, check if there's a proxy buffering option. Some Railway configs buffer the first request body while health-checking the upstream — if that buffer is flushed differently, body bytes can be altered. Adding this header to your worker's responses can help:
X-Accel-Buffering: no
Based on the 100% fresh-connection correlation: Railway's edge is almost certainly applying Transfer-Encoding: chunked on the first request of a new connection while forwarding content-length on subsequent ones (or vice versa), and the Inngest SDK's body reader is accumulating chunk-framing bytes into the hash input. The fix is capturing the body with express.raw() before any other middleware, and verifying the raw buffer directly.
If you can share what body parser setup you have in front of /api/inngest, I can pinpoint this further