a month ago
Hey team, since sometime today we've been observing some odd networking behavior, and I'm pretty sure it's not on our end.
Specifically, we occasionally wait seconds for the req body on POSTs to trickle in, and sometimes we get a Http: error reading a body from connection.
According to our telemetry, in all these slow spans, our server itself responds fast as usual (~5-10ms), but the overall request takes 5-10 seconds, spending almost the entirety on receiving the body. The request bodies are very small (mostly github webhook payloads).
We saw bursts of these slow requests at (UTC): 15:04:51–15:06:02 (15 slow requests in ~70 s), then sporadic recurrences at 16:00, 16:02, 17:11, 17:17 and 19:52. In between those moments, performance is fast as usual.
There were two other weird things today that may or may not be related:
- A Redis instance suddenly became unavailable via private networking while it was apparently up & running just fine around 15:12 UTC today. Restarting fixed it.
- Between ~14:44–15:13 UTC we saw some extremely slow GET requests (46 s, 221 s and 720 s+). The spans we recorded show no unusual increase in actual processing duration for these requests either, so again the cause seems to be a networking degradation.
I already tried redeploying the affected service, thinking it might move off a degraded host or something, but the issues keep reappearing. We haven't deployed any changes today, and we haven't ever observed this type of behavior previously in ~3 months of sustained load
IDs for the affected service: project/56cafcfa-394c-46c9-a811-dc3207bad3dc/service/3322253c-d09f-4975-9383-4c99294ca2a1?environmentId=ec032274-ccb3-4372-907d-acc4f1ca967f
20 Replies
Got a few more slow requests after the post. If it'd help I could provide a CSV with details on the exact offending traces
a month ago
I had the same problem that started at the same time as you and is now fixed. I think it was on the Cloudflare side for me, because I tried without proxied and it worked fine, but with proxied I had huge latency. I left proxied enabled on Cloudflare, and this morning there's no more latency without doing anything.
Attachments
Same issue here. Networking is slow with api requests. Our clients are complaining. We need a fix fast.
it's now 3 days since this weird behaviour and we tried to see if it's in our end. It's driving us crazy <:blob_help:1149662715409874944>
a month ago
This is a Railway infrastructure issue — not your code. Your server responds in 5-10ms but the network layer is adding seconds, which points to a degraded node or region-level problem.
What to do right now:
Go to railway.com/help and report it immediately with this info:
Project ID: 56cafcfa-394c-46c9-a811-dc3207bad3dc
Service ID: 3322253c-d09f-4975-9383-4c99294ca2a1
Time windows: 15:04–15:06, 16:00, 16:02, 17:11, 17:17, 19:52 UTC
Symptoms: slow POST body ingestion, Redis private network drop, extreme GET delays
Check status.railway.com for any ongoing incidents in your region
In Railway dashboard → your service → Settings → Region → redeploy to a different region temporarily to see if issues follow you or stay behind
Why redeploying didn't help:
Railway may be scheduling your container back onto the same degraded host. Only Railway staff can move you off a specific node.
Bottom line: Tag a Railway employee in your Station thread — this needs backend investigation. The Redis private network drop at the same time confirms this is infrastructure-level, not your app.
Fwiw, our problems seem to have disappeared, no more alerts today. It'd still be nice to know what happened!
25 days ago
@TFx @David Are you using the Cloudflare proxy in front of your Railway applications?
We use in our primary domain, but our subdomain isnt proxied, which we are currently accessing
@Phineas I don't think we use a Cloudflare proxy, our frontend is directly linked with a domain name but backend is used as is
last Friday everything seemed fine but today it's back at slow requests again
Our case is not yet resolved. We have checked everything in our side: microservices, code, database, app integrations. Everything is in order ! We're lost, any insight would be appreciated... The weird thing is: it only happen in PROD, staging and dev are working fine.
We're not considering scaling issue since we have about 1k requests per week.
19 days ago
Quick update on the latency you reported. We've rolled out a series of edge changes over the past week, tied to our new CDN, that should address the intermittent TTFB spikes.
Are you still seeing slow requests? If so, send a few X-Railway-Request-Id values from recent slow ones with rough timestamps, and we'll trace them through the edge to confirm whether anything's still off on our side.
Railway Team
Hey I'm experiencing same issue, I didn't change any code, everything is working as expected, but some PATCH/POST requests take up to 35s, all of those 35seconds is spent on sql query which is simply update on primary key. It looks like DB is waiting for something that is blocking for so long. Any updates?
Please, take a look at this, load average and CPU metrics differ
Attachments
Any update so far ? We're still experiencing the same issue, it's been about 2 weeks now
18 days ago
Railway containers run as cgroups so you are seeing the full system cpu
18 days ago
Either way, this thread has been completely derailed and you are all reporting separate, unrelated issues, so please create a new thread explaining your problems individually so we can properly triage these
18 days ago
@phin can you please check here https://station.railway.com/questions/postgres-not-connecting-2a2fe156
I don't agree @Phineas , we are all experiencing the same networking latencies. Why would it differ now? If load average is the full system cpu then there is a problem here. Everytime the load average hit 40 to 50 our system break
18 days ago
Please open a new thread.