4 months ago
Heya! we've had some issues over the past day or so where one or two of our replicas on this drop outbound connections. We can fire inbound connections to the deployments just fine but the return trip seems to get dropped. dashboard also shows 4/4 online too.
a redeploy does fix the problem but only for a few hours at most. From what we can tell its our EU west and US east deployments being affected.
Would appreciate any support or pointers you can offer!
13 Replies
4 months ago
Hey, we are going to need more information here.
4 months ago
Specific errors, what you've done to debug this, etc.
Sure, so as an example of one of the outbound calls our services makes that failed is this API key validation to our DB. our DB has no record of ever receiving the request.
{
code: '',
message: 'TypeError: fetch failed',
details: 'TypeError: fetch failed\n\n' +
'Caused by: ConnectTimeoutError: Connect Timeout Error ' +
'(attempted addresses: 104.18.38.10:443, 172.64.149.246:443, timeout: 10000ms) (UND_ERR_CONNECT_TIMEOUT)\n' +
' at onConnectTimeout (node:internal/deps/undici/undici:2746:28)\n' +
' at Immediate._onImmediate (node:internal/deps/undici/undici:2727:11)\n' +
' at process.processImmediate (node:internal/timers:483:21)'
}```
We also saw the same with our log drain, again never reaching the provider:
```[2026-06-03T09:03:41.722Z] ERROR [AxiomGateway] Failed to send logs to Axiom: TypeError: fetch failed
[2026-06-03T09:03:41.722Z] ERROR [AxiomGateway] Log buffer overflow, dropped oldest logs```We've already ruled out crashes, OOM and deployment health (according to the dashboard and our /health endpoint).
So far we've tried service restarts to no effect and redeploys that do seem to resolve the issue temporarily.
We haven't seen a reoccurrence since my initial message though but we'd like to know if there's anything we can do to prevent this in future
4 months ago
Are you doing any allowlisting?
Just bumping this up again, appreciate its a tricky one but we'd love to get this traced
4 months ago
Is this a guaranteed thing that outbound networking stops working a few hours after a deployment?
No it seemed specific to those couple of days, we haven't had reports of dropped connected since as far as I'm aware
4 months ago
We're aware of intermittent outbound connectivity drops affecting some services, where outbound connections fail while internal networking and DNS keep working, and we're actively investigating. If you hit it, the quickest workaround is to redeploy, which moves your service onto a fresh host and restores connectivity. If it happens again, reply here with the rough time it started and we'll jump on it while it's live. Those real-time reports are exactly what help us track down the cause.
Status changed to Awaiting User Response angelo-railway • 4 months ago
4 months ago
This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!
Status changed to Solved Railway • 4 months ago