6 hours ago
Subject: Deployments stuck in INITIALIZING — app container never starts (staging)
Project: ATHENATS
Environment: staging
Service: athenats
Region: (as configured for this service)
SUMMARY
Since 2026-09-21 ~12:30 UTC, every deployment of this service builds
successfully, runs its preDeployCommand successfully, and then never starts
the application container. The instance stays in INITIALIZING indefinitely.
The previous deployment keeps RUNNING and serving traffic normally, so the
site is up, but no new code can be deployed.
This service deployed normally in under a minute until this morning, with
the same configuration.
THE EXACT PATTERN
A healthy deployment shows two container starts in its deploy logs:
Starting Container <- preDeploy container
INFO Nothing to migrate.
INFO Configuration cache cleared successfully.Stopping Container
Starting Container <- application container
INFO Configuration cache cleared successfully.
INFO Server running on [http://0.0.0.0:8080].The stuck deployments only ever produce the first block:
Starting Container <- preDeploy container
INFO Configuration cache cleared successfully.
INFO Nothing to migrate.Stopping Container
(nothing further — the application container never starts and never
logs a single line)
No error is emitted anywhere. The build log ends with a successful
"image push".
AFFECTED DEPLOYMENTS (all 2026-09-21, all with this pattern)
c6fed185-58f8-47aa-8cd3-3e9689ea2546 created 12:33:26 UTC
Build OK (image created 12:33:55Z). Sat in INITIALIZING for ~40 min
until we cancelled it.012a5b50-92ac-4dd6-8f7a-e83e0644523c created 13:00:18 UTC
Build OK (image created 13:01:14Z). Queued behind the above, then
INITIALIZING, then disappeared from active deployments on its own.cc83d4e5-d435-4221-872b-4ac0d9768b27 created 15:15:42 UTC
Build OK. Launched via `railway redeploy --from-source` with NO other
deployment in progress and an empty queue. Same result: INITIALIZING,
no application container.LAST HEALTHY DEPLOYMENT (still running right now)
c8767072-7352-4573-97d4-22a42ff55e37 created 05:18:53 UTC
SUCCESS, instance RUNNING, serving all traffic. Same service, same
railway.json, same nixpacks.toml.WHAT WE HAVE ALREADY RULED OUT
-
Volume: athenats-volume (/app/storage/app) reports status=Ready,
isPendingDeletion=false, deletedAt=null, 83 MB of 5000 MB. Checked both
while a deployment was stuck and with nothing deploying.
-
Replicas: numReplicas is null for this service instance.
-
Queue contention: cc83d4e5 was launched with an empty queue and no other
active deployment, and failed identically.
-
Our configuration: railway.json and nixpacks.toml are unchanged since
2026-09-06; the commits deployed today touch only PHP, Blade templates,
tests and compiled assets.
-
Migrations: the preDeployCommand completes successfully every time
("Nothing to migrate." / a migration that reports DONE).
-
A platform incident: status.railway.com shows 100% uptime for Builds,
Deployments and Compute with no open incidents.
POSSIBLY RELATED PLATFORM CHANGE THE SAME DAY
Separately, the Nixpacks base image changed between 12:33 and 12:50 UTC
today: commit c8218b0 built successfully at 12:33:55Z, and the same commit
failed to build at 12:50 because ext-intl, ext-bcmath and ext-zip were no
longer present in the build environment. We fixed that on our side by
declaring those extensions in composer.json, and builds pass again.
We mention it only as context: the container-start problem began BEFORE that
image change (c6fed185 built with the old image and still failed to start),
so we do not believe they are the same issue.
RELEVANT CONFIG
railway.json:
healthcheckPath: "/"
preDeployCommand: php artisan config:cache && php artisan route:cache
&& php artisan view:cache && php artisan migrate --force
startCommand: php artisan config:clear
&& (php artisan schedule:work >> /tmp/schedule.log 2>&1 &)
&& php artisan serve --no-reload --host=0.0.0.0 --port=$PORTWHAT WE NEED
Why is the application container not being created after preDeploy completes,
and what can we do to get deployments flowing again? We have deliberately left
cc83d4e5 in place so you can inspect it.
4 Replies
Status changed to Awaiting Railway Response Railway • about 6 hours ago
5 hours ago
We are seeing a similar intermittent issue in Singapore (Hobby, Docker with a persistent volume).
Since approximately 2026-09-21 14:56 UTC, multiple Docker builds and image pushes succeeded, including different image digests, but CREATE_CONTAINER stayed incomplete and instances remained INITIALIZING without application logs. Several attempts were left for 20–26 minutes; one failed with only “Failed to create deployment.”
The previous deployment continued serving traffic and restarted successfully at 15:44 UTC, mounting the existing volume. The volume reports Ready and uses approximately 56 MB of 500 MB. Cancelling stale attempts and retrying with an empty queue did not consistently fix this.
One explicit-commit deployment finally succeeded at 16:08:45 UTC without a service configuration change. The next release built and pushed successfully, then stalled again at CREATE_CONTAINER from 16:37:06 UTC. We cancelled that attempt around 16:42 UTC and retried the same commit; that retry has also remained INITIALIZING since 16:43:13 UTC. The last successful deployment remains healthy. No application migration or pre-deploy command is involved.
For comparison, a temporary nginx:stable-alpine service in the same project and Singapore region started successfully at 15:57 UTC; CREATE_CONTAINER took approximately 7 seconds. The same application then started successfully both without a volume and with a new empty diagnostic volume. This service had no public domain or production data.
Is there a known issue affecting container replacement or volume placement? We would appreciate a recovery procedure that preserves the running deployment and persistent data.
Project, service, and deployment identifiers can be provided directly to Railway staff if a private channel is available. This public report intentionally omits identifiers, site URLs, credentials, environment values, code, database contents, and user messages.
5 hours ago
Update from our case: we also tested a bounded cold replacement. We confirmed the previously healthy instance had exited before requesting a fresh deployment of the same commit. The build completed, but the new instance still remained INITIALIZING for about 100 seconds. We cancelled that attempt and successfully restored the previous instance; health is normal again. Simply removing live connections or stopping the previous container did not immediately resolve the problem. We have left a normal deployment attempt pending for investigation. The root cause remains unconfirmed.
4 hours ago
Further read-only evidence from our case: at 17:41 UTC the deployment started at 17:10 UTC was still INITIALIZING, with no application logs. Comparing the healthy and pending deployments found identical environment-variable snapshots and service configuration. The intervening commits only change frontend/mobile assets and documentation; backend code, Dockerfile and dependency locks are unchanged.
Could the platform team please check these two previously confirmed failure modes?
- A persistent volume pinned to an affected host, preventing container creation despite successful build/image publication: https://station.railway.com/questions/deployments-stuck-failing-at-create-con-a464c0b7
- A volume-integrity protection workflow leaving deployments blocked: https://station.railway.com/questions/service-deployment-stuck-in-queued-wai-7922ebe8
These are historical precedents, not a claim that our root cause is confirmed. Our earlier control service could start the application with a new empty volume, while the original service intermittently cannot create replacement containers. Please inspect host placement, volume handoff/protection state, and image-pull errors while preserving the healthy instance and existing volume. We can provide identifiers privately to Railway staff. No identifiers, application URLs, secret values, code or user data are included here.
4 hours ago
Recovery update from our case: the previously pending deployment completed at 17:48:28 UTC on 2026-09-21. We discovered this during the next live status check. We made no further deployment request, application/configuration change, volume reassignment or region change. CREATE_CONTAINER took 37 minutes 26.9 seconds; the subsequent healthcheck took 2.2 seconds. The original persistent volume mounted successfully. We have now independently verified the new production assets, published APK and the actual Android update/overlay-install flow. The underlying cause remains unknown to us, including whether any platform-side intervention occurred. A trace of the delayed container creation would still help prevent recurrence.