Skip to content
Production Service Down — Build Outage Prevents Hotfix Deployment or Rollback
mendelb-blip
OP

a month ago

Hello Railway Support,

Our production application is currently down, and Railway's ongoing build outage is preventing us from restoring service.

Service details:

What happened:

Our deployment of commit 2053a5c completed successfully but subsequently crashed during startup with the following error:

Error: WheelPilot identity middleware registration could not be located.

We identified the application startup issue, corrected it, and merged the fix into GitHub main:

Fixed commit: a0bce93

Pull request: https://github.com/mendelb-blip/wheelpilot/pull/134

The corrected code has passed our backend build and automated tests. However, Railway is refusing to deploy the fix because of the ongoing platform incident:

https://status.railway.com/incident/JUA1PC5H

Attempts to trigger a new deployment return:

Deploys have been paused temporarily.

We also attempted to roll back to a previously successful deployment (2b72392d / commit 55c738b). Although the interface briefly displayed "Rollback successful," it also returned the deployment-paused error, and our production service remained crashed.

Our urgent request:

Is there any way to restore our previously working deployment or deploy the already-committed startup hotfix while the Railway build system is paused?

Our primary concern is restoring production availability as quickly as possible. We understand Railway is actively addressing the broader incident, but we would appreciate any available recovery workaround or an estimated time for builds to resume.

Thank you for your assistance.

Solved

1 Reply

Railway
BOT

a month ago

We're sorry for the trouble. The "Deploys have been paused temporarily" message comes from the active incident New builds are temporarily paused. It affects builds in all regions, including US West, and new builds are paused for every workspace while the fix is in progress.

The incident updates say deploys started during the pause will need to be triggered again once builds resume. So once the pause lifts, you can redeploy your fixed commit or retry the rollback. We can't give a time for builds to resume yet. Updates are posted on the incident page above.


Status changed to Awaiting User Response Railway • about 1 month ago


Railway
BOT

22 days ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway • 22 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...