10-day old commit randomly promoted to production
obsidiantide
HOBBYOP

2 months ago

Hi Community,

Yesterday at 08:15 UTC, a 9-day-old commit was unexpectedly deployed to production, causing data loss due to schema incompatibility. I had to redeploy the current version to recover.

What I've ruled out:

  • I did not manually trigger this deployment
  • No one else has repository access
  • No cron schedule is configured
  • No GitHub Actions or Railway automation configs exist
  • The deployment was not the latest commit on main

This appears to be an unexplained automatic deployment from Railway's infrastructure. Anyone else has experienced something similar?

Solved

6 Replies

Status changed to Awaiting Railway Response Railway about 2 months ago


2 months ago

Do you have a specific deployment ID we can look into?


Status changed to Awaiting User Response Railway about 2 months ago


brody

Do you have a specific deployment ID we can look into?

obsidiantide
HOBBYOP

2 months ago

Sure,

this was the correct deployment ID: 280abc25-d12c-47c5-84f2-82966f628ca7,

this is the old commit that was randomly deployed: ed8b3774-41dd-4444-a537-2bf3c4d6c0d0,

and then I redeployed the correct one again: e0663d5f-c4f3-479a-a63a-7292767537e9


Status changed to Awaiting Railway Response Railway about 2 months ago


Thanks for those IDs. We're checking what triggered deployment ed8b3774 — specifically whether it was a GitHub webhook, a scheduled redeploy, or an internal event. Will follow up with what we find.


Status changed to Awaiting User Response Railway about 1 month ago


angelo-railway

Thanks for those IDs. We're checking what triggered deployment `ed8b3774` — specifically whether it was a GitHub webhook, a scheduled redeploy, or an internal event. Will follow up with what we find.

obsidiantide
HOBBYOP

a month ago

Thank you!

Since this directly affected my production database (deleted data since they were inside nonexistent fields), what is the best way to handle such cases? On my hobby plan, there is no backup, so I had to manually recreate some of the data. This would be impossible if the scale of the project was bigger.


Status changed to Awaiting Railway Response Railway about 1 month ago


a month ago

This is currently a single report, so we're unable to investigate the root cause further at this time. We are tracking this issue organically and will look into it when additional reports surface.

Regarding your question about handling such cases, application-level concerns like database schema compatibility and data protection strategies fall outside the scope of what we can assist with here. We're unable to provide application architecture suggestions.


Status changed to Awaiting User Response Railway about 1 month ago


obsidiantide

Thank you! Since this directly affected my production database (deleted data since they were inside nonexistent fields), what is the best way to handle such cases? On my hobby plan, there is no backup, so I had to manually recreate some of the data. This would be impossible if the scale of the project was bigger.

rickynotaro
HOBBY

a month ago

There's no automated backups but you can always backup your database yourself. 🙂

I'm personally backing up my DB outside of Railsway every day.


Status changed to Awaiting Railway Response Railway about 1 month ago


Status changed to Solved sam-a about 1 month ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...