Deploys hang at `creating containers` and our staging and prod have gone down

Deploys hang at

creating containers

and our staging and prod have gone down

thelucre

PROOP

4 months ago

https://status.railway.com/ shows all incidents resolved but our PRO plan environments are dead on the line. Builds are handing for over 10 mins or more at generating containers and no logs coming through. Serious issue. Third event this week and we're about to open public beta. Guys what can we do here?

I tried redploying all app services and restarting DBs, nothing... Deploys hang and restarts don't fix any problems.

Solved

18 Replies

thelucre

PROOP

4 months ago

Deploys hang at creating containers and our staging and prod have gone down

brody

EMPLOYEE

4 months ago

Please allow some time for new deployments to come online as our deployment pipeline works through the backlog.

thelucre

PROOP

4 months ago

but why are our previous containers, the live ones, failing?

thelucre

PROOP

4 months ago

502 from both apps regardless of deploy status, jsut downtime

brody

EMPLOYEE

4 months ago

We are looking into that.

thelucre

PROOP

4 months ago

please kkeep that status page active if there are known issues! 🙏

brody

EMPLOYEE

4 months ago

Build issues have been resolved, so the status page has been resolved.

thelucre

PROOP

4 months ago

then why can't my db connect externally?

brody

EMPLOYEE

4 months ago

That would be a separate issue that we are looking into.

justinrajdeveloper

HOBBY

4 months ago

App is up but throwing 502 for me.

shodty

PRO

4 months ago

That would be a separate issue that we are looking into.

sounds like all systems are not operational then?

subhanc

HOBBY

4 months ago

systems are down again for me. They went back up but now down again.

shodty

PRO

4 months ago

Attachments

image.png

zakii

PRO

4 months ago

It's been literally 24 hours with my system down, partially operating but main functions not. Can't deploy any new builds. That's fine and obviously the team is working on it, but why status page shows no issues!

brody

EMPLOYEE

4 months ago

There is no longer a widespread issue related to builds. If you are experiencing issues with builds, please open your own thread.

As for the 502s, we have resolved them. All network requests should work as expected now.

thelucre

PROOP

4 months ago

we're bak! thank you! if the feedback is welcome, please consider keeping that status page aligned with literal issues

zakii

PRO

4 months ago

Still failing. Same exact error after your fix:

DEADLINE_EXCEEDED: dial tcp 152.233.30.98:443: i/o timeout
Registry: production-dpiad.railway-registry.com

Build succeeds, image pushes fine, but container creation can't pull from the registry. This has been going on for 8+ hours now. Region: us-east4-eqdc4a. Can someone look at this specific registry endpoint?

zakii

``` Still failing. Same exact error after your fix: DEADLINE_EXCEEDED: dial tcp 152.233.30.98:443: i/o timeout Registry: production-dpiad.railway-registry.com Build succeeds, image pushes fine, but container creation can't pull from the registry. This has been going on for 8+ hours now. Region: us-east4-eqdc4a. Can someone look at this specific registry endpoint? ```

brody

EMPLOYEE

4 months ago

Unrelated, please use the thread you already have open.

Status changed to Solved brody • 4 months ago

Welcome!