Project: Mola — 72bc754c-6745-48e8-ac25-917a5739ab61 Environment: production — 42ef47fb-e411-458f-9aeb-7f1b1525e0c4 Service: Redis (redis:8.2.1, volume-backed) — 1b6628f0-6933-4717-afe1-29c678281be6 Region: US East Running deployment: since 2026-05-25 (deployment shows "Online" throughout) Stuck replacement deployment: 18b0f527-59c8-49cc-84c7-45498bac9e89, created 2026-07-18T04:49:37Z, stuck at "Creating containers…" for 15+ minutes The host (or volume) backing this Redis service appears severely CPU/IO-starved since ~02:00 UTC on 2026-07-18. Evidence from the service's own logs (all UTC): - Baseline: RDB background save of the ~1MB dataset runs every ~60s and completes in <100ms. - 04:03:02 → 04:03:32 — save took 30 seconds. - 04:08:00 — save cron fired; fork didn't happen until 04:08:09 (9s to fork); save completed 04:23:15 — 15 minutes for ~1MB. - 04:25:17 → 04:29:51 — Redis's once-per-60s save check took ~4.5 minutes to fire (serverCron starvation). - 04:29:51 — background save started and had still not completed at 05:05+ (35+ minutes and counting). Client impact during the same windows: connections to redis.railway.internal:6379 returned "(104, Connection reset by peer)" and "Timeout connecting to server"; our API's p95 latency reached 200 seconds with requests stuck in multi-minute cohorts. At 04:49:37 we attempted `railway redeploy` to move off the host; the new container has been stuck at "Creating containers…" ever since (normally <1 minute), which we suspect is because the service is pinned to the volume's host. Request: please migrate this service and its volume to a healthy host, or advise on how to unblock the stuck deployment. Happy to provide more logs.

Redis Issue - Railway Central Station

Redis Issue

solrwarrior

PROOP

a month ago

Project: Mola — 72bc754c-6745-48e8-ac25-917a5739ab61

Environment: production — 42ef47fb-e411-458f-9aeb-7f1b1525e0c4

Service: Redis (redis:8.2.1, volume-backed) — 1b6628f0-6933-4717-afe1-29c678281be6

Region: US East

Running deployment: since 2026-05-25 (deployment shows "Online" throughout)

Stuck replacement deployment: 18b0f527-59c8-49cc-84c7-45498bac9e89,

created 2026-07-18T04:49:37Z, stuck at "Creating containers…" for 15+ minutes

The host (or volume) backing this Redis service appears severely CPU/IO-starved

since ~02:00 UTC on 2026-07-18. Evidence from the service's own logs (all UTC):

Baseline: RDB background save of the ~1MB dataset runs every ~60s and

completes in <100ms.
04:03:02 → 04:03:32 — save took 30 seconds.
04:08:00 — save cron fired; fork didn't happen until 04:08:09 (9s to fork);

save completed 04:23:15 — 15 minutes for ~1MB.
04:25:17 → 04:29:51 — Redis's once-per-60s save check took ~4.5 minutes to

fire (serverCron starvation).
04:29:51 — background save started and had still not completed at 05:05+

(35+ minutes and counting).

Client impact during the same windows: connections to

redis.railway.internal:6379 returned "(104, Connection reset by peer)" and

"Timeout connecting to server"; our API's p95 latency reached 200 seconds with

requests stuck in multi-minute cohorts.

At 04:49:37 we attempted railway redeploy to move off the host; the new

container has been stuck at "Creating containers…" ever since (normally <1

minute), which we suspect is because the service is pinned to the volume's

host.

Request: please migrate this service and its volume to a healthy host, or

advise on how to unblock the stuck deployment. Happy to provide more logs.

$20 Bounty

3 Replies

solrwarrior

PROOP

a month ago

Update: the replacement deployment ultimately FAILED. Railway's own

"Diagnosis" panel classified it as an Infrastructure Error: "The deployment

stalled at the container creation stage with no logs and zero resource

usage… The image and configuration are unchanged from the previous

successful deployment, so nothing in the code or config caused this."

We have since migrated the app to a new Redis service (new volume), which

provisioned in seconds — consistent with the original host being the

problem. Please still investigate/reclaim the affected host; the original

Redis service (1b6628f0-6933-4717-afe1-29c678281be6) and its volume are

left in place as evidence and can be deleted once you've taken a look.

Railway

BOT

a month ago

This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.

Status changed to Open Railway • about 1 month ago

mayori

MODERATOR

a month ago

Have you try Redeploy source image?

marlonwq

FREE

a month ago

Wasn't it the original host that experienced a massive I/O (or disk) bottleneck, which stalled the RDB snapshot and locked the persistent volume, preventing the redeployment process from detaching it?

Welcome!