12 days ago
I have a cron job that's been successfully executing with the same code for many months. Logs show that it has been having networking-related issues for the last few hours.
The cron job makes network requests to various services hosted in other places. Those network requests are timing out sporadically now, even though all the upstream services are accessible with low latency from elsewhere. It's unlikely to be a problem with the specific upstream services as they include PikaPods, Frantech BuyVM, and Hetzner, and I don't expect all three of those to have an outage at exactly the same time. Additionally all those upstream services are accessible from sources other than Railway, as I mentioned.
There are no resource issues with the cron job. It is configured to allow the maximum CPU and memory usage, but only actually utilizes 0.1 vCPU and 160 MB memory.
The timed-out requests don't even show up in network flow logs on Railway.
Based on the information available to me, this does not appear to be an application-related problem. It appears rather to be an issue with the Railway platform.
Can someone from the Railway team please check the networking configuration for the infrastructure servicing this deployment? I can provide specific IP addresses and timestamps privately.
2 Replies
Status changed to Awaiting Railway Response Railway • 12 days ago
12 days ago
One thing I just noticed looking at the timestamps in the logs: my requests aren't timing out after the configured application-level timeout. They're getting ETIMEDOUT immediately from https.request in Node.js.
Status changed to Open Railway • 11 days ago
Status changed to Awaiting Railway Response Railway • 10 days ago
Status changed to Awaiting User Response Railway • 10 days ago
10 days ago
I got emails about a couple of comments on this thread, but now that I'm looking at the thread, they don't seem to exist. Not sure what's up with that then. The thread says "Awaiting User Response" but there's nothing for me to respond to, so I guess the support request disappeared into the void.
Anyway... this has been far from the first issue I've had with Railway, and they have been piling up. So I figured rather than laboriously amassing enough evidence of a platform issue to have any chance of it being fixed, it would probably be easier to move my entire infrastructure off Railway and onto another provider. Turns out it was straightforward. Took a few hours to get all 11 services migrated to two open-source Coolify instances running on Hetzner VMs, and everything works better than it did before.
(In case anybody at Railway cares - is Angelo Saraceno still there, after our conversations back in July 2023? - then please feel free to reach out privately and I would be happy to direct your attention to the specific incidents over the years that have soured me on your platform in a way that has finally become permanent. They are unfortunately numerous but I don't think a public discussion board is the right forum for sharing that information.)
I guess since I no longer use Railway, the thread is "resolved" now. In a certain sense of the word.
Status changed to Awaiting Railway Response Railway • 10 days ago
Status changed to Solved brody • 10 days ago