Hi Railway team, My Postgres service (service ID: `675bcc74-cc8f-4d79-992c-8e51cc3a25d1`, project: `zooming-quietude`) in **us-west2** has been down since \~10:30 UTC today (April 9). The deployment nodes in us-west2 cannot reach **any** container registry — including Railway's own internal registry. Every redeploy fails with: DEADLINE\_EXCEEDED: ctrd: failed to pull/unpack image: failed to resolve reference "[production-us-west2.railway-registry.com/675bcc74-cc8f-4d79-992c-8e51cc3a25d1](http://production-us-west2.railway-registry.com/675bcc74-cc8f-4d79-992c-8e51cc3a25d1):..." dial tcp 162.220.232.123:443: i/o timeout Key details: * **Region:** us-west2 * **Image:**`ghcr.io/railwayapp-templates/postgres-ssl:18` * **Last successful deployment:**`a30a6acd` on April 3 (same image digest `sha256:25ca4a...`) * **Failed deployments:** 9+ consecutive failures, all with the same `i/o timeout` to both `162.220.232.123` and `162.220.232.124` * I also tried deploying via Dockerfile (build succeeds, push to Railway registry succeeds, but the deployment node still can't pull from `production-us-west2.railway-registry.com`) * Switching to Docker Hub `postgres:18` also fails — same timeout * **My data is on a persistent volume** in us-west2, so I can't switch regions The build machines work fine — they can pull and push images. It's specifically the **deployment/runtime nodes** in us-west2 that have broken networking. Could you please investigate the us-west2 deployment cluster connectivity? Thank you.

us-west2 deployment nodes cannot pull images — i/o timeout to registry

adversoup

HOBBYOP

a month ago

Hi Railway team,

My Postgres service (service ID: 675bcc74-cc8f-4d79-992c-8e51cc3a25d1, project: zooming-quietude) in us-west2 has been down since ~10:30 UTC today (April 9).

The deployment nodes in us-west2 cannot reach any container registry — including Railway's own internal registry. Every redeploy fails with:

DEADLINE_EXCEEDED: ctrd: failed to pull/unpack image: failed to resolve reference

"production-us-west2.railway-registry.com/675bcc74-cc8f-4d79-992c-8e51cc3a25d1:..."

dial tcp 162.220.232.123:443: i/o timeout

Key details:

Region: us-west2
Image:ghcr.io/railwayapp-templates/postgres-ssl:18
Last successful deployment:a30a6acd on April 3 (same image digest sha256:25ca4a...)
Failed deployments: 9+ consecutive failures, all with the same i/o timeout to both 162.220.232.123 and 162.220.232.124
I also tried deploying via Dockerfile (build succeeds, push to Railway registry succeeds, but the deployment node still can't pull from production-us-west2.railway-registry.com)
Switching to Docker Hub postgres:18 also fails — same timeout
My data is on a persistent volume in us-west2, so I can't switch regions

The build machines work fine — they can pull and push images. It's specifically the deployment/runtime nodes in us-west2 that have broken networking.

Could you please investigate the us-west2 deployment cluster connectivity?

Thank you.

Solved

2 Replies

Status changed to Awaiting Railway Response Railway • about 1 month ago

noreal001

HOBBY

a month ago

Same issue here. My service Darks Factory News (ID: 371c0848-d582-4eb3-8f66-ea7877a94009, project BAHUR) is also stuck in us-west2 since April 9.

DNS resolution fails inside running containers: [Errno -3] Temporary failure in name resolution
New deploys fail: dial tcp 162.220.232.125:443: i/o timeout
railway run works fine (DNS resolves) — only runtime nodes are broken
Persistent volume attached — can't switch regions

Volume ID: afafbf1e-b4d4-4883-9f48-1226781abcdc

Please migrate or fix us-west2 deployment nodes.

sam-a

EMPLOYEE

a month ago

I'm really sorry for how long this took to get a response. You were right that this was an infrastructure issue on our side, and the delay here was unacceptable.

The us-west2 registry connectivity issue has since been resolved. @noreal001, I can see your Darks Factory News service is now deploying successfully as of April 13.

@adversoup, I see you deleted your Postgres service, but your volume with the data is still intact (5GB at /var/lib/postgresql/data). To recover:

1. Create a new Postgres service in your project

2. In the service settings, attach your existing volume postgres-volume) to the mount path /var/lib/postgresql/data

3. Deploy - your data should be there

If you run into any issues with this, let me know and I'll help sort it out.

Again, apologies for the radio silence here.

Status changed to Awaiting User Response Railway • about 1 month ago

Railway

BOT

a month ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway • 28 days ago

Welcome!