Good news first: your data is safe and reachable right now. worker-volume (about 105 MB, your SQLite database) is intact and healthy, and it's currently mounted at /data on your canario-nginx service, which is deployed and running. You can open a shell into canario-nginx with railway ssh and copy the database out of /data. This was a platform-side issue with volume operations in that project, not anything in your code or config, and it has since cleared for your service.

One caution: if you've got any uncommitted staged changes in this environment, one would mount worker-volume onto worker2. Don't apply that one, or it'll pull the volume off the service you're reading from. Copy your data out from canario-nginx first.

The Railway Team

Thanks - confirmed. The volume was mounted on canario-nginx, I copied the SQLite database out with railway ssh, and the data is fully recovered. I did NOT apply the pending worker2 mount. Much appreciated.

Reporting back on one thing, in case it helps: the same CREATE_CONTAINER signature came back TODAY in a DIFFERENT project - the new one I migrated to. I do see the 'Deployments are slow to progress' incident banner now, so this may simply be that incident; if so, ignore the rest.

Project odds-ev-prod (cc4b4e5e-69c5-425e-8e51-d9e2eae824fe), env production (f51b7827-1720-44cd-baaa-9b16db4ee7b5), service worker (549669fe-9ad1-4ca2-9411-6b1806c66edc), us-west, one volume at /data. It deployed fine on Aug 17 (f1fc4ef9) and has been running since. Today a config-only push failed twice, identical signature: build succeeds and the image is pushed, then it hangs at CREATE_CONTAINER with zero runtime logs and goes FAILED after ~15-20 min. Failed deployments: 9d97d4b1-2847-4147-aacb-31c9e01cbf8c and f883541e-0d7a-4607-9f07-49e8fcbec73a. Same Dockerfile, same start command, tests green.

Good news this time: the running container was NOT torn down, so the service stayed up throughout.

Question: is today's failure the ongoing deployments incident, or the same volume-path issue as before? If it is the volume path, the common factor across both projects is 'redeploy of a service that already has a volume mounted'. I will leave the two failed deployments in place in case they are useful to trace.

Glad the data is fully recovered. Today's failures in the new project are the active deployment incident, not the volume bug. The original issue was stuck container removals blocking every commit workflow in that project. Today's deployments are stalling earlier in the pipeline (during build/image stages), which matches the Deployments are slow to progress incident affecting US West. Your running container was not torn down because the failure happened before the deploy reached the container-swap stage. Once that incident resolves, your deploys should progress normally.

The container-removal bug struck AGAIN — third project affected, second forced migration in 6 days.

Timeline of this recurrence (project odds-ev-prod, cc4b4e5e-69c5-425e-8e51-d9e2eae824fe, service worker 549669fe):

• Aug 23 ~20:20 UTC: a routine deploy logged "Stopping Container", the running container was killed, and no new container ever started.

• 3 consecutive deployments FAILED with build OK and zero container logs (deployments 1b92bab5, 1bdbfab8, 03eaee4c) — the exact "stuck container removals" failure mode sam-a described above for our first dead project (vivacious-nurturing).

• status.railway.com showed "Fully Operational" the whole time (unlike the Aug 18 incident, which did NOT kill the running container).

• The same commit/Dockerfile deploys fine elsewhere: we migrated to a fresh project (odds-ev-prod2, dada9aed) and the identical image boots there (4 successful deploys today).

Two asks:

1. Root cause: this is now the 2nd project killed by stuck container removals in a week (vivacious-nurturing Aug 16 → migrated Aug 17 to odds-ev-prod → killed Aug 23). Migrating projects every few days is not sustainable — is there a fix or a flag for our account/workload?

2. Data recovery: volume worker-volume in odds-ev-prod is now detached and holds ~6h of Sunday data we need. Last time (Aug 18) you unstuck the volume and we recovered it by attaching to a live service. Please unstick this one the same way. We will not delete anything in that project until then.

Your data is reachable right now. worker-volume is mounted on worker-b, which is up, so railway ssh into worker-b and copy the SQLite database out of /data. Don't apply the pending change sitting in that environment, it would move the volume onto worker and pull it off the service you're reading from.

Same fault as before: a container from your Aug 19 deploy couldn't be stopped, so its deployment stayed marked as running and every new deploy spent about 16 minutes trying to swap it out. That stuck deployment has been cleared, and worker can deploy again.

On prevention there's no good answer to give you. There's no account or workload flag to set, and it isn't tied to your code or your project, your two cases landed on different machines, which is why migrating didn't protect you. A recurrence can be cleared from our side in minutes once the failed deployment IDs are known, so rebuilding a project isn't the remedy.

Confirmed: data fully recovered. I pulled both the SQLite database and the raw payload archive out of /data on worker-b via railway ssh.

One note for your telemetry: when the volume was mounted onto worker-b, that service (built from our production repo) booted a full second instance of our worker and spent ~20h polling providers and pushing alerts against the old database. No harm done on our side, and I have deleted worker-b now that the data is out.

Thanks for the quick unstick and for the prevention guidance. On any recurrence we will post the failed deployment IDs here immediately instead of migrating projects. Marking this solved.