Inngest webhook signature verification fails on fresh TCP connections to Railway worker
argus-intelligence-pi
HOBBYOP

a month ago

Our Node worker on Railway receives webhook callbacks from Inngest at /api/inngest. Approximately 3-5% of these callbacks fail with HTTP 401 "Invalid signature" from the Inngest SDK. We've correlated 100% of failures with fresh TCP connections (non-zero DNSLookup in Inngest's outbound request trace) and 100% of successes with reused connections (DNSLookup=0). Successful and failed requests both show Server: railway-hikari and X-Railway-Edge: railway/us-east4-eqdc4a. Can you investigate whether Railway's edge handles request bodies differently on connection establishment vs. on reused connections? This breaks HMAC verification on the worker side.

$10 Bounty

1 Replies

Railway
BOT

a month ago

This thread has been opened as a bounty so the community can help solve it.

Status changed to Open Railway about 1 month ago


spatrickpaul
HOBBY

a month ago

Fixes

Fix A — Use a raw body parser that ignores encoding headers (recommended)

The Inngest SDK for Node uses the body it receives from Express/Node's request stream. Make sure you're not running express.json() before the Inngest route — if you are, the body has already been parsed and re-serialized, which can silently change byte content (e.g., key ordering, whitespace).

// ❌ This corrupts HMAC verification if mounted globally before /api/inngest

app.use(express.json());

// ✅ Mount body parser only on routes that need it, AFTER /api/inngest

app.use('/api/inngest', serve({ client: inngest, functions: [...] }));

app.use(express.json()); // all other routes

Fix B — Force connection reuse from Inngest's side

Not directly in your control, but you can reduce the fresh-connection rate by ensuring your worker responds with Connection: keep-alive and sets a reasonable Keep-Alive: timeout=60 header. Railway may already do this, but verify it's not stripping keep-alive headers on first-connection responses.

Fix C — Implement a signature verification wrapper that normalizes the body

import { createHmac } from 'crypto';

function verifyInngestSignature(rawBody: Buffer, sig: string, signingKey: string): boolean {

const ts = sig.match(/t=(\d+)/)?.[1];

const mac = sig.match(/s=([a-f0-9]+)/)?.[1];

if (!ts || !mac) return false;

const body = rawBody.toString('utf8'); // explicit UTF-8, no BOM handling

const expected = createHmac('sha256', signingKey)

.update(`${ts}.${body}`)

.digest('hex');

return expected === mac;

}

And capture the raw body before any middleware touches it:

app.use('/api/inngest', express.raw({ type: '/' }), (req, res, next) => {

// req.body is now a Buffer, untouched

next();

});

app.use('/api/inngest', express.raw({ type: '/' }), (req, res, next) => {

// req.body is now a Buffer, untouched

next();

});

Fix D — Railway-specific: disable response buffering

In your railway.toml or Railway service settings, check if there's a proxy buffering option. Some Railway configs buffer the first request body while health-checking the upstream — if that buffer is flushed differently, body bytes can be altered. Adding this header to your worker's responses can help:

X-Accel-Buffering: no

Based on the 100% fresh-connection correlation: Railway's edge is almost certainly applying Transfer-Encoding: chunked on the first request of a new connection while forwarding content-length on subsequent ones (or vice versa), and the Inngest SDK's body reader is accumulating chunk-framing bytes into the hash input. The fix is capturing the body with express.raw() before any other middleware, and verifying the raw buffer directly.

If you can share what body parser setup you have in front of /api/inngest, I can pinpoint this further


Welcome!

Sign in to your Railway account to join the conversation.

Loading...