3 months ago
Starting around 8:38am ET this morning, my team and I started seeing an enormous surge of ReadError, ConnectTimeout, and RemoteProtocolError when attempting to fetch from external APIs.
We have checked our logs from these other services (Supabase, for example) and everything on their end looks good - no limits being reached, no errors or warnings in the logs. But we saw huge spikes in our Railway metrics (ex. the CPU).
The last deployment before today was four days ago, a completely unrelated and tiny bugfix. We tried restarting and redeploying the service a few times, but the issue persists.
There have been no significant changes to our app. No sudden increase in traffic or users, no code changes, no dependency changes.
Can anyone help us resolve this, or point us in the right direction?
Thanks!
8 Replies
3 months ago
Hey, what region are your services deployed on?
3 months ago
Unfortunately, I do not see anything that points to a Railway issue here. I would recommend looking into your code or using observability tools like OpenTelemetry or Sentry to improve your understanding of the problem.
I'm also getting the same issue but in US West. All outgoing network requests from my app are receiving timeout errors
issue did not start until railway's outages today and continues to persist
I also suspect this is a Railway issue for the same reason - our network issues started when Railway's outages did. But I don't have enough information to know for sure. We're still experiencing these issues as well.
We've checked our observability tools, including Sentry, and all our monitoring dashboards in our external services. Supabase is showing no issues. Sentry is showing increased request times, which makes sense because we're consistently getting timeout errors for outgoing requests.
Beyond that, there's not much we can see because our vision into Railway's network is limited.
3 months ago
experiencing the same. everything seems healthy otherwise
3 months ago
Hey, sorry for the delay on this, but can y'all create your own help thread? (@zillarelli)
That will help us organize the issues and get a faster response from the team