Deploys hang at `creating containers` and our staging and prod have gone down

Deploys hang at

creating containers

and our staging and prod have gone down

thelucre

PROOP

23 days ago

https://status.railway.com/ shows all incidents resolved but our PRO plan environments are dead on the line. Builds are handing for over 10 mins or more at generating containers and no logs coming through. Serious issue. Third event this week and we're about to open public beta. Guys what can we do here?

I tried redploying all app services and restarting DBs, nothing… Deploys hang and restarts don't fix any problems.

Solved

18 Replies

thelucre

PROOP

23 days ago

Deploys hang at creating containers and our staging and prod have gone down

brody

EMPLOYEE

23 days ago

Please allow some time for new deployments to come online as our deployment pipeline works through the backlog.

thelucre

PROOP

23 days ago

but why are our previous containers, the live ones, failing?

thelucre

PROOP

23 days ago

502 from both apps regardless of deploy status, jsut downtime

brody

EMPLOYEE

23 days ago

We are looking into that.

thelucre

PROOP

23 days ago

please kkeep that status page active if there are known issues! 🙏

brody

EMPLOYEE

23 days ago

Build issues have been resolved, so the status page has been resolved.

thelucre

PROOP

23 days ago

then why can't my db connect externally?

brody

EMPLOYEE

23 days ago

That would be a separate issue that we are looking into.

justinrajdeveloper

HOBBY

23 days ago

App is up but throwing 502 for me.

shodty

PRO

23 days ago

That would be a separate issue that we are looking into.

sounds like all systems are not operational then?

subhanc

HOBBY

23 days ago

systems are down again for me. They went back up but now down again.

shodty

PRO

23 days ago

Attachments

image.png

zakii

PRO

23 days ago

It's been literally 24 hours with my system down, partially operating but main functions not. Can't deploy any new builds. That's fine and obviously the team is working on it, but why status page shows no issues!

brody

EMPLOYEE

23 days ago

There is no longer a widespread issue related to builds. If you are experiencing issues with builds, please open your own thread.

As for the 502s, we have resolved them. All network requests should work as expected now.

thelucre

PROOP

23 days ago

we're bak! thank you! if the feedback is welcome, please consider keeping that status page aligned with literal issues

zakii

PRO

23 days ago

Still failing. Same exact error after your fix:

DEADLINE_EXCEEDED: dial tcp 152.233.30.98:443: i/o timeout
Registry: production-dpiad.railway-registry.com

Build succeeds, image pushes fine, but container creation can't pull from the registry. This has been going on for 8+ hours now. Region: us-east4-eqdc4a. Can someone look at this specific registry endpoint?

zakii

Still failing. Same exact error after your fix: DEADLINE_EXCEEDED: dial tcp 152.233.30.98:443: i/o timeout Registry: production-dpiad.railway-registry.com Build succeeds, image pushes fine, but container creation can't pull from the registry. This has been going on for 8+ hours now. Region: us-east4-eqdc4a. Can someone look at this specific registry endpoint?

brody

EMPLOYEE

23 days ago

Unrelated, please use the thread you already have open.

Status changed to Solved brody • 23 days ago