Skip to content
Production down: volume stuck, every deploy hangs at container creation since 03:08 UTC
junkify-xavier
OP

2 months ago

Production has been down since 03:08 UTC on Oct 5 (502/503 on every route). Project modest-healing (a033fd4c-1c66-42db-9a54-bd17660a084f), environment production, service junk-ify-platform (fd27d617-e7f0-40f8-82c0-96577a638680), volume junk-ify-platform-volume (aea20fda-4eb8-4403-ad8a-7003d3c5f82c, /data, us-west2). Deployment bf279a74 got SIGTERM at 03:08 UTC and its teardown never finished; it still shows ACTIVE. Since then every deployment hangs at container creation: a0e1e64e and ada93af2 (cancelled), ca1b3f33 and 4d725a08 (FAILED, diagnosis 'Infrastructure Error, no code or config change needed'), 2059b6b9 (deploying now). The volume seems held by the old instance. Please release it and get the service running. Do NOT delete or wipe the volume: it holds our production database.

Solved

4 Replies

Status changed to Awaiting Railway Response Railway • about 2 months ago


junkify-xavier
OP

a month ago

Hi Railway team. Your support AI asked us to open a new thread (earlier one: https://station.railway.com/support/force-release-volume-aea20fda-stuck-on-f-91c14252).

Project a033fd4c-1c66-42db-9a54-bd17660a084f, service fd27d617-e7f0-40f8-82c0-96577a638680, production environment 0a7718d4-4e69-4cf9-afd9-408cc581849c, us-west2.

  1. On Oct 4 from 22:04 CT, 5 deployments hung at CREATE_CONTAINER (a0e1e64e, ada93af2, ca1b3f33, 4d725a08, 2059b6b9). Our container had exited cleanly and volume aea20fda showed READY. We were down 3 h 12 min until we restored a backup to a new volume. What caused the hang, and how do we avoid it? Please keep the old volume.

  2. Manual backups fail with "Plan limit of 10 backups per volume exceeded" (13 of our 15 are your scheduled ones). Can you raise the limit or stop counting scheduled backups?

  3. Volume I/O is capped at 70 MB/s and 3,000 IOPS. Can it be raised?

  4. Growing the volume from 100 to 200 GB with skipDeploys still caused a redeploy and about 25 seconds of errors. Is that expected?

Thank you.


a month ago

Your app exited cleanly, but the previous deployment's container never finished shutting down, so each new deployment waiting to take over the volume hung at container creation. That container was finally removed at about 06:24 UTC. Your original volume, junk-ify-platform-volume, is still in the project, detached, with its data intact.

On backups, Pro allows 10 backups per volume, and scheduled backups count toward that limit. Deleting backups you no longer need frees room for manual ones.

The per-volume I/O limit can't be raised on Pro.

The 100 to 200 GB change ran as a Grow rather than a live resize. Grow restarts the service to apply the new size, so a short burst of errors is expected on that path. A live resize grows the volume without a restart.


Status changed to Awaiting User Response Railway • about 2 months ago


junkify-xavier
OP

a month ago

New problem on the same project (Pro plan). Since 2026-10-11 05:13 UTC our production deployment is REMOVED about once an hour, always between minute 13 and minute 21 past the hour: 15 times so far. Each time the container gets a clean SIGTERM and no newer deployment is queued, so the site is down until we redeploy (about 2 minutes).

What we checked: our workspace audit log shows no remove or stop action; none of our people or scripts ran one (our only Railway calls before each removal were read-only); no config change since Oct 9 22:06 UTC; memory was under 4 GB of the 24 GB limit and CPU under 0.3 core before the removals we measured.

Project a033fd4c-1c66-42db-9a54-bd17660a084f, environment production 0a7718d4-4e69-4cf9-afd9-408cc581849c, service fd27d617-e7f0-40f8-82c0-96577a638680.

Removed deployments (UTC, Oct 11): 28195fcf 05:13:38, 6c14b1ec 06:18:05, a5c84b67 07:16:10, 147534fe 08:16:55, 8977b83a 09:15:50, c29c3bd7 10:16:58, 6bf25859 11:15:08, 133c33e2 12:18:42, b40e8f55 13:19:21, a0d74fb9 14:18:12, 9bde1d78 15:16:53, c4c84b03 16:18:28, d1e50301 17:16:47, eea7950c 18:21:17, 84cdb0c7 19:17:56.

Questions:

  1. What on Railway's side removes these deployments, and why?
  2. How do we stop it?
  3. Can we be notified when Railway stops or removes a deployment?

Thank you.


Status changed to Awaiting Railway Response Railway • about 1 month ago


sam-a
EMPLOYEE

a month ago

The hourly removals were triggered by your workspace’s $75 hard spending limit. Each redeployment was stopped again by the next usage-limit check.

You raised the limit to $150 at 22:50 UTC on October 11, and we haven’t seen another billing-triggered stop since. Your current deployment is running successfully, with usage at about $81.

For notifications when a deployment is removed, you can configure a project webhook under Project Settings → Webhooks, including delivery to Slack or Discord.


Status changed to Awaiting User Response Railway • about 1 month ago


Railway
BOT

a month ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway • about 1 month ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...