Network outage, builds hanging
drmarshall
PROOP

3 months ago

My builds hanging/failing across multiple environments, and public network requests are 408/499ing in production. It seems like there is an outage that's impacting us-west-2 at least that's not reported on the status page.

34 Replies

Railway
BOT

3 months ago

Hey there! We've found the following might help you get unblocked faster:

If you find the answer from one of these, please let us know by solving the thread!


drmarshall
PROOP

3 months ago

=========================
Container failed to start
=========================
 
Failed to start deployment.
If this error persists, please reach out to the Railway team at https://station.railway.com.

Seeing this trying to redeploy a previously successful deploy.


guppykang
HOBBY

3 months ago

same


lightshire
PRO

3 months ago

same


aysan0
PRO

3 months ago

++ but I don't see an outage marked on the status page?


azuremoonchild
PRO

3 months ago

Bump. Same issue here


lukasfoisy
PRO

3 months ago

Same in US East


ctrlshifti
PRO

3 months ago

same for europe


polopopeye
HOBBY

3 months ago

same


biinkable
PRO

3 months ago

same


guppykang
HOBBY

3 months ago

cry


guppykang
HOBBY

3 months ago

works for me now on us east


jack
PRO

3 months ago

Looks like it may be an issue related to a recent incident mentioned in another post (https://station.railway.com/questions/failed-to-start-deployment-a4269225), supposedly the team is investigating it and some users are reporting it working in some regions.


drmarshall
PROOP

3 months ago

My builds are still hanging and production (us-west-2) is still 408/499ing all public traffic from western US, but not central/eastern US.


lowshinji
HOBBY

3 months ago

Same here, 408s and 499 for me on production builds


atkinsonben
HOBBY

3 months ago

+1 on the 408/499ing all public traffic (asia-SE-1)


kzgrey
HOBBY

3 months ago

Railway is broken but their status page shows its all green: https://status.railway.com/


jake
EMPLOYEE

3 months ago

Could you please give some examples of these issues? We had an incident prior https://status.railway.com/cmflvsdnb01081nuxvbbzkoyd, but it should be resolved


Status changed to Awaiting User Response Railway 3 months ago


sagefarrenholz
PRO

3 months ago

https://usevault.app


Status changed to Awaiting Railway Response Railway 3 months ago


jake
EMPLOYEE

3 months ago

I'm able to load that just fine? What is the steps to reproduce?

https://usevault.app


Status changed to Awaiting User Response Railway 3 months ago


atkinsonben
HOBBY

3 months ago

All traffic to my frontend service is returning 408. http://new-frontier-demo.up.railway.app/


Status changed to Awaiting Railway Response Railway 3 months ago


jake

Could you please give some examples of these issues? We had an incident prior https://status.railway.com/cmflvsdnb01081nuxvbbzkoyd, but it should be resolved

drmarshall
PROOP

3 months ago

https://railway.com/project/a41a4ccd-9849-4c35-b6a1-efbed80bea67/service/3dd4887f-5c0f-4943-9698-ff4eeab0e278?environmentId=ebd2d436-eaa8-4ea2-a536-d754646923bd&id=0e8c2671-ce81-4cb5-bd9c-e8d074a2d1a4#http

filter: -@httpStatus:200

– afaict, all public network requests from western US are 408/499ing at the metal edge (via a cloudflare proxied domain). If I VPN into central/eastern US, they succeed as 200s. If I generate a random public domain, all 200s from western US.


retr0rafay
PRO

3 months ago

We are experiencing this issue as well. It showed up earlier where we were getting 502s, then it was fixed for a brief moment, and now after another deployment it's not working. Check here: https://rounds.so/


jake

I'm able to load that just fine? What is the steps to reproduce?https://usevault.app

sagefarrenholz
PRO

3 months ago

hmm I'm seeing 408


lowshinji
HOBBY

3 months ago

Same for me as well. Calls to my fastify backend are all returning as 408s, 499s and we also had 502 earlier as well.


sagefarrenholz
PRO

3 months ago

Some requests getting through but intermittent

Attachments


jake
EMPLOYEE

3 months ago

Okay, I THINK we found an issue. It's for a pretty small portion of applications. We rolled out a fix; could you redeploy and check again?


Status changed to Awaiting User Response Railway 3 months ago


atkinsonben
HOBBY

3 months ago

Thats working now thankyou


Status changed to Awaiting Railway Response Railway 3 months ago


jake

I'm able to load that just fine? What is the steps to reproduce?https://usevault.app

kzgrey
HOBBY

3 months ago

Working now, thank you


jake
EMPLOYEE

3 months ago

You're VERY welcome. And here's an incident called for it.

https://status.railway.com/cmfotwolf00gl35cmg21dl2sr

Please note, it was a VERY small subset of traffic, and even occured before this thread was posted. Appreciate the patience.


Status changed to Awaiting User Response Railway 3 months ago


drmarshall
PROOP

3 months ago

Builds are still failing. cannot redeploy: ERROR: failed to build: failed to solve: Canceled: context canceled


Status changed to Awaiting Railway Response Railway 3 months ago


lowshinji
HOBBY

3 months ago

Managed to redeploy and it is working now for me. Thank you!


drmarshall
PROOP

3 months ago

Network issues fixed after a redeploy. Deploys are still super flaky.


Status changed to Solved drmarshall 3 months ago


Status changed to Open jake 3 months ago


willscottrod
HOBBY

3 months ago

This issue I believe is back


Loading...