19 days ago
Hi Railway team,
Since this morning, every deployment of our api-gateway service fails at the same step. Build and container creation finish normally, then CONFIGURE_NETWORK hangs for about 457 seconds and the deployment is marked FAILED. The application starts fine and its logs show no errors.
Project: adorable-intuition (bd296af4-9924-4767-865e-b1d26474ea27)
Environment: production (d5c84afe-e187-4310-bb34-4585ed547680)
Service: api-gateway (0b1674d8-68cb-400d-9983-c3c4d5ab17bd)
Replicas: 8
Failed deployments today (UTC). Every one shows BUILD_IMAGE 48–80 s, CREATE_CONTAINER 3 s, then CONFIGURE_NETWORK about 457 s:
3a0b19bd-049f-40d0-affd-76e197a69c4c, 14:04: a redeploy of a commit that deployed successfully yesterday
740f28fd, 13:20
946d5204, 13:10
110f3c3d, 12:58
63eae939, 09:27
The last successful deployment was at 08:02 UTC today. Even then, CONFIGURE_NETWORK took 153 s, which is much slower than usual.
So the failure does not depend on our code: a redeploy of a known-good commit fails the same way.
Impact: while one of these deployments was pending, the dashboard showed two ACTIVE deployments. When we removed the old one, all our sites returned 502 until we restored a previous version. We are now on a deployment that Railway reports as FAILED, but it is still serving traffic, so we can't safely ship any gateway changes.
Could you check what is blocking the network configuration for this service, and whether the 8 replicas are a factor? We're happy to reduce the replica count temporarily if that helps.
Thanks!
6 Replies
Status changed to Awaiting Railway Response Railway • 19 days ago
19 days ago
Same issue here, so this looks platform-wide rather than project-specific.
Project 6a49f552-ed0f-4781-9b30-94975aff0642, service api (c859ae85-5f25-45fa-bf83-80d4af4b2bc7), 1 replica. Every deploy since 12:27 UTC (11 in a row, across 10 commits) builds and creates the container fine, then sits in CONFIGURE_NETWORK for about 7m35s and fails. The last good deploy was at 12:11 and is still serving.
We also moved the service from europe-west4 to us-east4 to test it, and it failed the same way there (deployment 81f3108c), so it isn't regional. We run 1 replica, not 8, so the replica count isn't the cause either.
Example failed deployments: 81f3108c (us-east4), c617956c, 5dea4bd4 (europe-west4).
19 days ago
We are only keeping production alive by triggering a new redeploy every few minutes, because each deployment serves traffic for about 7 minutes before it is marked FAILED. Reducing replicas (down to 2) or changing regions makes no difference — CONFIGURE_NETWORK never completes. Is this related to your ongoing 'Recovery after October 21 Incident'?"
19 days ago
The recovery work after the October 21 incident is about restoring new signups. Its updates don't describe deployments hanging in network configuration, and we haven't found anything linking the two.
On api-gateway, the 14:04 UTC redeploy matches what you described: the build took 80 seconds, container creation took 3 seconds, then network configuration ran for 456 seconds before the deployment was marked failed. We haven't identified what is blocking that step. Since single-replica services and other regions fail the same way, the replica count doesn't look like the cause, so lowering it isn't likely to help.
It's still unconfirmed which deployment is serving your traffic. Avoid removing the one that's currently live, because that's what caused the 502s earlier. Redeploying the same commit again just repeats the same failure, so it won't tell us anything new.
Status changed to Awaiting User Response Railway • 19 days ago
Railway
The recovery work after the October 21 incident is about restoring new signups. Its updates don't describe deployments hanging in network configuration, and we haven't found anything linking the two. On api-gateway, the 14:04 UTC redeploy matches what you described: the build took 80 seconds, container creation took 3 seconds, then network configuration ran for 456 seconds before the deployment was marked failed. We haven't identified what is blocking that step. Since single-replica services and other regions fail the same way, the replica count doesn't look like the cause, so lowering it isn't likely to help. It's still unconfirmed which deployment is serving your traffic. Avoid removing the one that's currently live, because that's what caused the 502s earlier. Redeploying the same commit again just repeats the same failure, so it won't tell us anything new.
19 days ago
Unfortunately we rmoved manually the one tha was serving and now we have to deploy every 7 minutes to keep the production alive
Status changed to Awaiting User Response brody • 19 days ago
Status changed to Awaiting Railway Response brody • 19 days ago
19 days ago
I'm sorry you've had to keep redeploying to keep production online. We're looking into the rollout failures affecting api-gateway and the other services reported here. If you're still affected, could you let us know whether your next deployment completes successfully and stays running? We'll follow up here when we have more information to share.
Status changed to Awaiting User Response Railway • 19 days ago
Status changed to Solved delavegaopengl • 19 days ago
14 days ago
Hi,
We identified and fixed the root cause of the deployment failures you reported. This specific issue should not happen again.
I’m sorry for the disruption and inconvenience it caused. If you’re still seeing a deployment fail, please reply here with a recent deployment link so we can take a look.
Status changed to Open Railway • 14 days ago