Failures Connecting to Railway Workers from Inngest Runs
rylax
HOBBYOP

2 months ago

Hi there!

We're experiencing persistent DNS resolution failures when Inngest tries to reach our worker functions hosted on Railway. This is severely impacting our production batch jobs with a 3-5% daily failure rate. The failures happen before our worker code even executes, pointing to infrastructure-level connectivity issues.

I have contacted Inngest, and they told me it is a Railway issue where Railway is throttling the Inngest DNS requests. They said it's RW's job basically to remove the rate limit and urged me to reach out to you instead.

We are actively loosing customers over this, so I'd really appreciate your help before we need to migrate away.

Error Pattern:

Error performing request to SDK URL: error performing request: 
lookup <worker-hostname> on [2607:e3c0:a040:9002::64:ff9b]:53: **server misbehaving**
dial tcp [64:ff9b::4221:16ef]:443: i/o timeout
Error performing request to SDK URL: error performing request: lookup <worker-hostname> on [2607:e3c0:a040:9002::64:ff9b]:53: server misbehaving dial tcp [64:ff9b::4221:16ef]:443: i/o timeout

Key Technical Details:

The 64:ff9b:: prefix indicates NAT64 (IPv6-to-IPv4 translation) but I'm honestly not sure. Railway logs show zero incoming requests at failure timestamps - DNS fails before HTTP connection Our Railway service is healthy (low CPU/memory, responds fine to direct requests) Failures are sporadic - successful runs happen immediately before/after failures Affects all our scheduled and event-triggered functions

Impact: This is a blocker for scaling our usage. We need reliable execution for production workloads, and the current 3-5% failure rate makes it difficult to commit to a paid tier. The intermittent nature suggests DNS resolver or NAT64 gateway instability on Inngest's side (at least after hours of AI debugging with Opus 4.5 we came to this conclusion) when connecting to Railway.

Solved

3 Replies

Anonymous
HOBBY

2 months ago

i spent a couple hours looking into this as its always DNS!

0 Days Since DNS sits on my desk as a reminder. Based off your great info, inngest’s runtime is using ipv6, translating to ipv4, then DNS is intermittently failing before any HTTP requests reache railway, this matches the log entries showing zero incoming requests

the trick will be to force inngest to hit an IPv4-only hostname

the two solutions i located are a new hostname that publishes only an A record meaning ipv4 address or use Cloudflare or any edge serivce in front and make that hostname for inngest


rylax
HOBBYOP

a month ago

is there any way actual Railway developers / support could help us out here? every day we got more and more angry customers and are hands seem to be tied. extremely frustrating experience.


a month ago

This thread has been escalated to the Railway team.

Status changed to Awaiting Railway Response itsrems about 1 month ago


a month ago

Howdy! Can you try again and let me know what you're seeing? Inngest reported that some of what you are running into is being resolved.


Status changed to Awaiting User Response Railway about 1 month ago


Railway
BOT

a month ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway 30 days ago


Loading...