铁路部门的各位同事，大家好！

我们的生产服务目前不可用，并返回 HTTP 502 错误。

无论是新建镜像还是使用之前已知良好的镜像，都会在 CREATE_CONTAINER 阶段停滞。SNAPSHOT_CODE、BUILD_IMAGE 和 PUBLISH_IMAGE 阶段均成功完成，但容器始终无法启动，并且没有生成任何应用程序启动日志。

回滚到之前运行正常的版本后，出现了同样的故障模式，服务未能恢复。

由于新镜像和已知良好的回滚镜像在应用程序启动之前都失败了，因此这似乎是容器配置、调度程序或编排问题，而不是应用程序启动失败。

请检查该服务的容器配置和调度程序状态，并协助恢复该服务。

Additional evidence for this ongoing production outage:

The build itself completes successfully:

• Railpack detects Node 22.23.2.
• npm install completes successfully.
• The application files are copied successfully.
• An OCI/Docker image is exported with a valid image digest.
• The image is pushed successfully.

There are no application startup logs because the deployment never reaches the point where the application process can start.

Railway's deployment diagnostics report:

“The build, image push and release all completed successfully, but the container never managed to start. The deployment stopped without recording any specific error.”

The deployment details show:

• Initialization: completed
• Build: completed
• Deploy: completed / 0:00
• Post-deploy: never started
• No application startup logs

The event sequence confirms that SNAPSHOT_CODE, BUILD_IMAGE, and PUBLISH_IMAGE completed, while CREATE_CONTAINER never completed. The only deployment log recorded afterward is “Stopping Container”.

A rollback to a previously successful production image failed at the same container creation stage.

This further confirms that the failure occurs after image build and publish but before application startup. Could Railway staff please inspect the internal CREATE_CONTAINER, compute scheduling, and volume attachment path for this service?

Try set a Custom Start Command on the Service Settings

We completed the suggested Custom Start Command test.

Before the test:

• Custom Start Command was unset
• Railpack already auto-detected npm run start
• package.json has "start": "node server.js"

We then explicitly set:

Custom Start Command = npm run start

and performed one controlled deployment.

Result:

• SNAPSHOT_CODE completed
• BUILD_IMAGE completed
• PUBLISH_IMAGE completed
• CREATE_CONTAINER failed
• application startup was never reached
• npm run start was never executed
• node server.js was never executed
• no application startup logs were produced
• instance was ultimately REMOVED

So the missing Custom Start Command hypothesis can now be ruled out.

A known-good rollback image and the current image both fail before application startup.

Could Railway staff please inspect the service at the container provisioning / compute scheduling / persistent volume attachment layer?

We are intentionally not triggering further deployments or touching the persistent volume.

Production impact update: existing users are now unable to log in because the backend service remains unavailable. This is affecting active production users, not only health checks or test endpoints. The frontend is still reachable, but authentication and all backend-dependent features are unavailable.

Update: this incident is ongoing and NOT resolved. If the thread has

been marked as Solved, please reopen it.

Current impact: active production users have been unable to log in

for more than 7.5 hours and counting. This is a live production

service, not a test or staging environment.

Summary for reference:

• The original deployment failed at CREATE_CONTAINER.
• A rollback to the previously stable production version failed at the same CREATE_CONTAINER stage.
• The previously stable version had been running successfully in production before this incident.
• No application startup logs were produced in either case.
• The application process never reached npm run start or node server.js.
• We also explicitly configured the Custom Start Command as npm run start, following the earlier suggestion. The deployment still failed before application startup.
• Build, image creation, image publishing, and release completed successfully before the container creation failure.
• The service is still configured in the Amsterdam / EU West region.
• The existing persistent volume mounted at /data has not been modified, detached, recreated, migrated, or replaced.

The repeated failure of both the current version and a previously

known-good production version before application startup is

consistent with an issue in the platform-side container

provisioning, compute scheduling, or persistent-volume attachment

path rather than an application runtime failure.

We have intentionally stopped making further deployment attempts in

order to preserve the current production state and avoid any risk to

the persistent volume.

Could someone from the Railway team please inspect this service's

internal provisioning / scheduling / volume-attachment status

directly?

Given the ongoing impact on active production users, an ETA or

confirmation that this has been escalated would be greatly

appreciated.

Both deployments are completing their build stages successfully but failing at the container provisioning step before any application code runs. This is not something in your application or configuration, and there is nothing to change on your side. We're checking what is preventing containers from being created for this service and will update you in this thread.

Please note that we can only provide support in English. If you could reply in English going forward, that will help us assist you as quickly as possible.

Thank you for confirming this is happening before the application code runs and that no changes are required on our side.

We will keep the service unchanged while you investigate.

One important note: this production service has an existing persistent volume mounted at /data containing our SQLite production database. Please preserve the current volume and its data during any internal troubleshooting, re-provisioning, or container recovery work.

We will not trigger any further deployments, restarts, region changes, or volume operations from our side.

Please let us know when it is safe to retry or when the service has been recovered. Thank you for looking into this.

We received another automated “Deployment Failed” email about an hour ago for the production environment.

We have not triggered any new deployment, restart, region change, or volume operation from our side.

The service is still being left unchanged as requested. Could you please confirm whether this failed deployment was part of Railway’s internal investigation or recovery attempt?

Please continue to preserve the existing /data volume during troubleshooting.

Thank you.

We’ve redeployed the service successfully, and the application is now running with your original /data volume mounted. We haven’t deleted or replaced the volume.

We found no new deployment today before this recovery attempt. The earlier deployment was marked as crashed around the time you reported receiving the email, which may explain that notification.

Could you check whether users can log in and access their existing data now?

Thank you for the update.

We can confirm that the service is now running again and users are able to log in and access their existing data successfully.

We also verified that the original /data volume is mounted and the historical production data is still intact.

We appreciate your help in restoring the service and preserving the existing volume.

For now, we will keep the Railway environment unchanged while we continue operating our current production setup separately.

Thank you again for your support.