**Project id:** `1cdc3514-8698-4242-86c8-ec2d6eadb105` **Environment:** `production`, id `4f597888-1a24-459a-916b-a802a503ac9b` ## Summary Three times in 18 days, our production service's config object has become corrupted: after days of normal operation, a routine GitHub-push deploy wedges at `CREATE_CONTAINER` (stuck `pending`, **zero container logs**) and times out to `FAILED` ~17–20 min later — and from that moment the service never deploys again by any means, while every other service in the same project, region, and environment deploys normally. Each time we have had to recreate the service and move the volume to recover. The corruption has now followed us across two rebuilt services, so we expect the current one (`trader4`) to be poisoned too. We are asking for a **root cause**, not a cleanup: what is corrupting these service objects, and is there anything in our configuration that triggers it? ## The three incidents All three have the identical signature: build + publish complete cleanly, then `CREATE_CONTAINER` sits `pending` with zero container log output until the deployment times out to `FAILED`; retries and even redeploys of a previously-SUCCESS image wedge the same way. | # | Service | Service id | Corrupted (UTC) | Age at corruption | Evidence deployment ids | |---|---|---|---|---|---| | 1 | `trader` | `6e0791d5-d3c6-4669-b289-6633471d83c6` (since hard-deleted) | 2026-07-04 18:45 | months | `9c29222d`, `f154eef3`, `aa50fbe8` (all FAILED, zero logs) | | 2 | `trader2` | `1e2ec4b3-3d94-4fcb-ac3f-b67bb6e77f96` | 2026-07-18 20:24 | 14 days | `c28ed2f0` (push), `266a8fa7` (retry) | | 3 | `trader3` | `06b17488-9d5b-4e86-ba7c-13b378fda819` | 2026-07-22 03:14 | 3 days | `25efeaaf` (push deploy), `f497ec54` (redeploy of last-SUCCESS image) | ## Why we believe it's the service config object time) - **Region exonerated:** a fresh throwaway nginx project + environment + region (`europe-west4-drams3a`) deploys SUCCESS in seconds during the wedge (7s on 07-22). On 07-1esh volume in the same region — SUCCESS <1 min, so the volume subsystem is fine. - **Image exonerated:** `deploymentRedeploy` of t last-SUCCESS deployment (an image that ran for days) also wedges at `CREATE_CONTAINER` with zero logs (07-18: on trn trader3). - **Replacement works immediately:** a newly crea same repo, same variables, and the same volume (moved via IaC) deploys and runs normally right away — trader4's first depl container in 7s and was SUCCESS in 102s. - **Corruption also wedged deletion** (incident 1 original `trader` returned HTTP 504 and took ~6 days to complete asynchronously (started 2026-07-06, verified go ## Pattern / possible triggers - Each corruption fired on a routine push deploy operation (14 days for trader2, 3 days for trader3 — trader3 had ~6 clean deploys before wedging). - The service runs a Dockerfile build with a mounted volume (`/data`, SQLite), `restartPolicyType = ON_FAILURE` (max 5), watch replica, `europe-west4-drams3a`. - In each incident the outgoing container's proce (`ELIFECYCLE` in logs) around the time the wedge began — unclear whether cause or casualty. - trader2 and trader3 are retained in the project, source-disconnected and credential-stripped, in case you want to inspec objects. (If they're of no diagnostic value, feel free to say so and we'll delete them.) - Possibly relevant: the environment's **staged-changes subsystem** has appeared near this bug repeatedly — on 07-04 Rans reported success but remained staged-only, and a 07-18 IaC probe cleanup (service delete + volume delete) sat as staged 4 days until committed manually in the dashboard on 07-22. Staged deletions did not block deploys, but if config-object sta share storage, that may be a lead. - Stronger staged-changes evidence (2026-07-22 ~1 staged changeset in the dashboard — which displayed only the two probe deletions — **also applied a stale environment the production volume to decommissioned service trader3 and created new deployments on trader2 (`eb323b7f`) and trader3 containers simultaneously attached to the same volume for ~9 minutes until we intervened. A further staged changeset still referenced the original `trader` service id (`6e0791d5-…`) — deleted since ~2026-07-12 — and failed to commit with "An unk environment's staged/committed config state appears to be internally inconsistent, which looks related to whatever c objects. ## Impact Each incident is a full production outage of a system with a strict uptime requirement (3.5h, 50 min, 30 min respectively — we've drilled the rebuild-and-move-volume recovery). The current service `trader4` (`cd030325-c668-49c2-8000-d3732a11fdb0` the observed pattern we expect it to corrupt within days, and we'd like to break the cycle rather than keep rebuilding. Happ deploy ids, IaC config, or run diagnostics.

Three consecutive services corrupted: deploys wedge at CREATE_CONTAINER

p-boaz

HOBBYOP

5 days ago

Project id: 1cdc3514-8698-4242-86c8-ec2d6eadb105

Environment: production, id 4f597888-1a24-459a-916b-a802a503ac9b

Summary

Three times in 18 days, our production service's config object has become

corrupted: after days of normal operation, a routine GitHub-push deploy wedges

at CREATE_CONTAINER (stuck pending, zero container logs) and times out

to FAILED ~17–20 min later — and from that moment the service never deploys

again by any means, while every other service in the same project, region, and

environment deploys normally. Each time we have had to recreate the service

and move the volume to recover. The corruption has now followed us across two

rebuilt services, so we expect the current one (trader4) to be poisoned too.

We are asking for a root cause, not a cleanup: what is corrupting these

service objects, and is there anything in our configuration that triggers it?

The three incidents

All three have the identical signature: build + publish complete cleanly, then

CREATE_CONTAINER sits pending with zero container log output until the

deployment times out to FAILED; retries and even redeploys of a

previously-SUCCESS image wedge the same way.

|---|---|---|---|---|---|

| 1 | trader | 6e0791d5-d3c6-4669-b289-6633471d83c6 (since hard-deleted) | 2026-07-04 18:45 | months | 9c29222d,

f154eef3, aa50fbe8 (all FAILED, zero logs) |

(redeploy of last-SUCCESS image) |

Why we believe it's the service config object time)

Region exonerated: a fresh throwaway nginx

project + environment + region (europe-west4-drams3a) deploys SUCCESS in

seconds during the wedge (7s on 07-22). On 07-1esh

volume in the same region — SUCCESS <1 min, so the volume subsystem is fine.
Image exonerated: deploymentRedeploy of t

last-SUCCESS deployment (an image that ran for days) also wedges at

CREATE_CONTAINER with zero logs (07-18: on trn

trader3).
Replacement works immediately: a newly crea

same repo, same variables, and the same volume (moved via IaC) deploys and

runs normally right away — trader4's first depl

container in 7s and was SUCCESS in 102s.
Corruption also wedged deletion (incident 1

original trader returned HTTP 504 and took ~6 days to complete

asynchronously (started 2026-07-06, verified go

Pattern / possible triggers

Each corruption fired on a routine push deploy

operation (14 days for trader2, 3 days for trader3 — trader3 had ~6 clean

deploys before wedging).
The service runs a Dockerfile build with a mounted volume (/data, SQLite),

restartPolicyType = ON_FAILURE (max 5), watch

replica, europe-west4-drams3a.
In each incident the outgoing container's proce

(ELIFECYCLE in logs) around the time the wedge began — unclear whether

cause or casualty.
trader2 and trader3 are retained in the project, source-disconnected and

credential-stripped, in case you want to inspec

objects. (If they're of no diagnostic value, feel free to say so and we'll

delete them.)
Possibly relevant: the environment's staged-changes subsystem has

appeared near this bug repeatedly — on 07-04 Rans

reported success but remained staged-only, and a 07-18 IaC probe cleanup

(service delete + volume delete) sat as staged

4 days until committed manually in the dashboard on 07-22. Staged deletions

did not block deploys, but if config-object sta

share storage, that may be a lead.
Stronger staged-changes evidence (2026-07-22 ~1

staged changeset in the dashboard — which displayed only the two probe

deletions — **also applied a stale environment

the production volume to decommissioned service trader3 and created new

deployments on trader2 (eb323b7f) and trader3

containers simultaneously attached to the same volume for ~9 minutes until

we intervened. A further staged changeset still

referenced the original trader service id (6e0791d5-…) — deleted since

~2026-07-12 — and failed to commit with "An unk

environment's staged/committed config state appears to be internally

inconsistent, which looks related to whatever c

objects.

Impact

Each incident is a full production outage of a system with a strict uptime

requirement (3.5h, 50 min, 30 min respectively —

we've drilled the rebuild-and-move-volume recovery). The current service

trader4 (cd030325-c668-49c2-8000-d3732a11fdb0

the observed pattern we expect it to corrupt within days, and we'd like to

break the cycle rather than keep rebuilding. Happ

deploy ids, IaC config, or run diagnostics.

Solved

4 Replies

Status changed to Awaiting Railway Response Railway • 5 days ago

sam-a

EMPLOYEE

5 days ago

Good news: nothing is corrupted, and you can stop rebuilding services.

We traced all three incidents. Each time, the previous container refused to die when the new deploy tried to stop it. The kill signal was retried for about 18 minutes, then the deploy gave up and failed. The stuck container cleared on its own within about 30 to 40 minutes, and after that the service was deployable again. Your trader3 actually proved this: the deployment created at 17:15 UTC today started its container normally (it exited only because the service's credentials had been stripped). So the fix for a future wedge is to wait about 45 minutes and redeploy, not to recreate the service.

On the trigger: the container gets stuck mid-shutdown, most likely while flushing writes to the SQLite volume, and the ELIFECYCLE crash you saw right before each wedge means the kill arrived while the process was crash-restarting mid-write. Two things that should shrink the window on your side: make sure the app handles SIGTERM by closing SQLite and exiting within a few seconds, and use WAL mode if you aren't already.

If trader4 (or anything else) wedges at CREATE_CONTAINER again, reply here immediately with the timestamp and deployment ID. So far we've only been able to investigate after the fact, and catching it live lets us see exactly where the process is stuck. And you can delete trader2 and trader3, the stuck containers are already gone so those services hold no diagnostic value.

The staged changes behavior you documented (stale changesets referencing the deleted trader service, the double volume attach) is a real, separate bug on our side. It didn't cause the deploy wedges, but we've logged it. Thanks for the writeup.

Status changed to Awaiting User Response Railway • 5 days ago

p-boaz

HOBBYOP

3 days ago

Thanks sam-a — really helpful reframe, and good to know the staged-changes behavior is tracked separately. We've deleted trader2 and trader3, and adopted "wait ~45 min, then redeploy" as our recovery playbook.

Two data points that may narrow the root cause, because both of your recommendations were already in place before the second and third wedges:

WAL mode has been enabled from the start.
Since 2026-07-11 (a week before the trader2 wedge), every daemon registers a SIGTERM handler that closes its SQLite handle and exits 0 immediately, and our supervisor forwards SIGTERM to each child's whole process group, then SIGKILLs stragglers and exits 0 after 30 s. The 07-18 and 07-22 wedges both happened with that shipped.

One question that might explain the gap: our container's PID 1 is pnpm (Dockerfile CMD pnpm run:supervisor, exec'd through an entrypoint), and in our testing pnpm does not forward SIGTERM to its child process — that's exactly why the supervisor signals process groups internally. If the platform's stop sequence delivers SIGTERM only to PID 1 and waits, our shutdown handler never runs, and the tree would only die on SIGKILL. Does the stop path signal PID 1 only or the whole cgroup, and at what point does it escalate to SIGKILL? If it's PID-1-only, we'll switch the CMD to exec the node process directly so PID 1 handles SIGTERM itself.

Next wedge, we'll post the timestamp and deployment ID here immediately. Thanks again.

Status changed to Awaiting Railway Response Railway • 3 days ago

nico

EMPLOYEE

3 days ago

SIGTERM goes to PID 1 only. We do not signal any other process or the cgroup directly. After the draining window, PID 1 gets SIGKILL, which takes the whole container down. The draining window defaults to 0 seconds, configurable via the RAILWAY_DEPLOYMENT_DRAINING_SECONDS service variable. Your diagnosis is correct: with pnpm as PID 1, SIGTERM never reaches your supervisor, so the graceful SQLite shutdown never runs, and with 0s draining the container is SIGKILL'd immediately. Switching CMD to exec the node supervisor directly as PID 1 is the right fix, and you'll want to set RAILWAY_DEPLOYMENT_DRAINING_SECONDS to give it time to flush and exit (your 30s internal timeout would be a reasonable value).

Status changed to Awaiting User Response Railway • 3 days ago

p-boaz

HOBBYOP

an hour ago

Closing the loop: this is resolved. We switched the container to exec node directly as PID 1 and set RAILWAY_DEPLOYMENT_DRAINING_SECONDS=35, and on the very next cutover the captured logs showed the full graceful shutdown for the first time — supervisor received SIGTERM, all three daemons closed their SQLite handles and exited, clean REMOVED. That confirms the whole story: with pnpm as PID 1 the SIGTERM never reached our handler, every stop was a SIGKILL mid-write, and the stuck outgoing containers were the deploy wedges. Thanks sam-a and nico — the PID-1-only + 0s-default-drain details were exactly what we needed.

Status changed to Awaiting Railway Response Railway • about 1 hour ago

Status changed to Solved p-boaz • about 1 hour ago

Welcome!