a month ago
Project: acceptable-happiness (79d61780-d60d-4486-9769-1da50b6aa697)
Service: bandit-ads-generator (08f1dafe-f3b5-42a5-8006-cf29da0461df)
Environment: production (dd3ebbd0-b8ee-43ca-93da-55c6dc621add)
Two persistent issues, possibly related, both started ~2026-06-05:
-
ALL recent deploys FAIL at "Create container" stage (build OK every time):
- 89024f11-1ee0-48a3-a5ce-9f47e3abfeb6 (2026-06-06 20:58) FAILED
- e594a6b6-d454-4627-9ff1-6291d7fe3b17 (2026-06-06 20:35) FAILED
- a50455b8-6308-4dfe-a1c3-770b0997e179 (2026-06-05 22:01) FAILED
- d7f87ee1-ba9f-4dfc-a3b5-92146b9284ab (2026-06-05 21:54) FAILED
Diagnosis panel says: "Failed to create deployment. Infrastructure Error.
Build completed but deployment failed at container creation stage,
which is a platform-level issue."
Last SUCCESS: c02781db-d4b8-448b-a98d-4d5df67cfe91 (2026-06-04 16:13)
Service has a volume mounted at /app/persistent — possibly volume
lock conflict preventing new container creation?
-
Currently-running container (c02781db, uptime ~52h) experiencing
sustained ETIMEDOUT on outbound TCP to multiple destinations:
- api.anthropic.com:443
- www.french-bandit.com:443 (Shopify CDN)
From local shell both destinations are perfectly reachable.
Same code worked fine until ~2026-06-05.
Logs show: errno: 'ETIMEDOUT', code: 'ETIMEDOUT' on node-fetch calls.
Could you check if there's a regional networking issue affecting our
service or container orchestration on this project?
4 Replies
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
Hi Mathieu — thanks for the unusually detailed report, it made this much faster to pin down.
Good news: this isn't anything wrong with your service or your volume. Both issues you're seeing trace back to a platform-side networking problem in the us-west2 region where your service runs, and they're related — not a volume lock.
The failed deploys: Your builds are completing fine. The failure happens at the container-creation step, where our orchestrator pulls your built image from our internal registry — and those pulls are timing out at the network layer (network dial: i/o timeout) for services on the affected nodes in us-west2. We can see this is hitting multiple services in that region in the same window, not just yours, so it's a platform issue rather than your configuration. Your volume is healthy and mounted normally — it's not the cause.
The ETIMEDOUT errors from your running container (to api.anthropic.com, the Shopify CDN, etc.) are the other side of the same coin: the node your container is on is having intermittent outbound connectivity trouble, which is exactly why those destinations are reachable from your local shell but time out from inside the container. The timing matches the deploy failures — both started the evening of 2026-06-05 (UTC).
What we're doing / what you can do in the meantime:
- We've flagged this to the infrastructure team as a regional registry/networking issue and are tracking it alongside the other affected services.
- To try to get a fresh deploy out sooner, you can redeploy (the container sometimes lands on a healthy node), or temporarily move the service to another region (Service → Settings → Region) if your data residency allows. Note that a region move also relocates the service off the affected node, and your volume needs to be in the same region.
- I'd hold off on assuming any code changes are needed — the same code worked before and nothing in your setup changed.
I'll keep this thread updated as the platform issue is resolved. If you still see failures after we confirm recovery, reply here and I'll re-check your specific node right away.
Status changed to Awaiting User Response Railway • about 1 month ago
a month ago
Quick update, Mathieu — the underlying us-west2 registry/networking issue has recovered. We're no longer seeing image-pull timeouts on the platform, and your service's outbound timeouts stopped on 2026-06-05 as well. Please trigger a fresh deploy — it should now go through cleanly. Let me know if you hit anything further and I'll dig back in.
a month ago
Update : I just retried (deployment 35df3045-98d0-441d-ba74-1e4229cd9417,
2026-06-08 09:07 UTC+2) and it FAILED again with the same pattern (build OK,
"Failed to create deployment" at container creation). That's 5 consecutive
failures over the past 3 days (06-05, 06-06 ×3, 06-08).
The currently-running container (c02781db, uptime now ~89h) is STILL
experiencing the ETIMEDOUT outbound issues that started 06-05 — so the
networking issue on that node has not self-resolved either.
Two specific asks since waiting hasn't helped:
-
Can you manually pin our service to a healthy node within us-west2,
or trigger a node migration that bypasses the affected hosts?
-
If the issue persists on us-west2 broadly, we'd be open to a region
move — but only if you can handle the volume migration as part of it
(the volume contains app data and a region change without volume
migration would mean data loss).
Project / Service / Env IDs as before:
- acceptable-happiness (79d61780-d60d-4486-9769-1da50b6aa697)
- bandit-ads-generator (08f1dafe-f3b5-42a5-8006-cf29da0461df)
- production (dd3ebbd0-b8ee-43ca-93da-55c6dc621add)
Status changed to Awaiting Railway Response Railway • 29 days ago
a month ago
Thanks for your patience here — and apologies, my earlier "recovered, please redeploy" was premature. Digging back into the platform side with your latest failure (deploy 35df3045) in hand, I can now see the actual cause.
Your builds are succeeding, but container creation is failing because the us-west2 nodes can't pull your built image from our internal image registry — the pull is timing out at the network layer (network dial: i/o timeout) against the registry endpoint. This is not specific to your service: we're seeing the same image-pull timeouts hit multiple unrelated services in us-west2 in the same window, so it's a platform-level registry-reachability problem, not your config, your image, or your volume (which is healthy and not the cause). The outbound ETIMEDOUT from your still-running container on that node is consistent with the same underlying network-path degradation.
I've escalated this internally as an active platform incident with the evidence. A few honest notes on your two asks:
- Redeploying won't help until the registry path is restored — every retry hits the same broken pull, which is exactly what you've been seeing across the 5 attempts.
- We can't manually pin a service to a specific node from support, and I'd hold off on a region move for now. It wouldn't be a clean fix mid-incident, and a region change does require relocating your volume — we wouldn't do that without preserving your data, and I don't want to introduce that risk while this is in flight.
The fastest path is the infra fix, which is in progress. I'll update this thread the moment registry reachability is restored, and a fresh deploy should then go through cleanly. If you need to mitigate sooner and your data residency allows it, I can walk you through a safe region move with volume migration — but my recommendation is to hold.
Status changed to Awaiting User Response Railway • 29 days ago
22 days ago
This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!
Status changed to Solved Railway • 22 days ago