Skip to content
Production down: successful builds stuck at Creating containers in us-west2
angejamal10-afk
OP

a month ago

Production service cannot start after successful builds in us-west2 (20 October 2026, all times UTC).

The previously working build (commit f57d66e) was healthy at 14:34 UTC. The old process received SIGTERM around 14:50:45. Several subsequent builds pushed their image successfully but never produced runtime logs; the UI stays at Creating containers and public HTTP returns 502 Application failed to respond. SSH is closed by the remote host.

An authorized restart of the old container at 15:10:47 returned INVALID_ARGUMENT: unable to restart a container in a paused or unknown state: container state improper.

The current restoration was created at 15:29:23 UTC and targets ONLY the previously working commit f57d66e. It is still DEPLOYING with no runtime logs. There is one replica and an existing persistent /data volume. No environment variables, region, replica count or volume mounts were changed during this restoration.

Please investigate container provisioning and the paused/unknown state, and restore the existing known-working build. Please preserve the existing /data volume and all data; do not detach, recreate, reset or delete the volume, change environment variables, or run application/business recovery commands. No replay of expired jobs is requested.

I can provide project/service/deployment identifiers directly to Railway staff if needed. No credentials or private application data are included in this public report.

Solved

2 Replies

Status changed to Awaiting Railway Response Railway • about 1 month ago


angejamal10-afk
OP

a month ago

Update, 20 October 2026, all times UTC:

Production remains unreachable. Both restorations of f57d66e failed without runtime logs. We then tried the earlier morning version 21617f5: its image built successfully, but the restoration failed at 16:25 UTC without any runtime logs. The latest public health check at 16:52 UTC still returns HTTP 502. Proxy logs report connection refused.

The owner's current recovery target is now ONLY 21617f5, not f57d66e and not the newer configured source. Please do not deploy the latest source automatically.

Earlier restart attempts returned "unable to restart a container in a paused or unknown state: container state improper". This persists across two previously working code versions, so please inspect the host/container state and the existing volume attachment. The dashboard's Active/Online label is not consistent with the failing public endpoint.

Please preserve the existing /data volume and its current contents, variables, and service configuration. Do not reset, detach, recreate, delete, restore over, or migrate the volume without a separate approved plan. Do not run any application recovery or expired publication jobs. Please advise the least disruptive way to clear the stuck runtime state and recover the requested version.

The observed four-hour memory maximum is approximately 1.73 GB against an 8 GB limit; we have no evidence of an application OOM. We can provide the deployment/instance identifiers and affected request ID directly to Railway staff if needed.


Status changed to Awaiting Railway Response Railway • about 1 month ago


chandrika
EMPLOYEE

a month ago

We've cleared the stale state left by the earlier deployment that wouldn't shut down. That leftover state is why your 21617f5 and f57d66e restorations failed before any runtime logs appeared, even though their images built fine.

When you're ready, you can redeploy 21617f5 from that deployment's menu in the dashboard. That redeploys only that version, not the latest source. We haven't changed your /data volume, variables or service configuration.


Status changed to Awaiting User Response chandrika • about 1 month ago


Railway
BOT

24 days ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway • 24 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...