25 days ago
I saw a troubling thread on X today indicating that the Railway CLI was used by Cursor (in plan mode) and deleted all production data due to backups being contained inside of the volumes... and the volume was deleted which destroyed the backups intrinsicly.
If this is indeed a real incident or concern, I would like to prepare for this possibility and ensure I have a backup plan.
What are good ways to back up production environments here?
I have a PostGres DB, and some storage buckets.
This would be a great topic to cover in future newsletter / tutorial posts (if that does not already exist).
Pinned Solution
25 days ago
Hello ,
so yes wiping a volume deletes all its native backups and if a cli agent deletes your volume, the backups go with it
for your postgres db deploy the official railway template railwayapp-templates/postgres-s3-backups on github , it runs pg_dump on a schedule and pushes backups to an external s3-compatible bucket like cloudflare r2 or backblaze b2 and since these backups live outside railway, a volume deletion can't touch them
railway also has a native backup feature with daily/weekly/monthly schedules under your service backups tab worth enabling but it won't survive a volume wipe so treat it as a secondary layer only
one more thing is that when a volume is deleted it's not immediately permanent railway gives you a 48 hour window to restore it via a link sent to your email
for your storage buckets, i think railway's don't currently document an official automated backup solution for them, so worth raising that directly with railway support or on the roadmap
Hope this help you :)
8 Replies
25 days ago
I wanted to post a suggestion in the roadmap relating to requesting CLI/MCP controls be brought in at the account / project level. it won't let me because this post is too similar - it wont even let me override it.. lost the post a few times. If anyone wants to suggest that, let me know. I will support that thread.
25 days ago
Well pretty much the only way to back up production data is through Railway backups, which can be managed in the database configuration panel.
If an AI agent (eg, Cursor) has access to Railway CLI or any access to production environments, you should always be mindful and supervise the commands it's executing.
25 days ago
Hello ,
so yes wiping a volume deletes all its native backups and if a cli agent deletes your volume, the backups go with it
for your postgres db deploy the official railway template railwayapp-templates/postgres-s3-backups on github , it runs pg_dump on a schedule and pushes backups to an external s3-compatible bucket like cloudflare r2 or backblaze b2 and since these backups live outside railway, a volume deletion can't touch them
railway also has a native backup feature with daily/weekly/monthly schedules under your service backups tab worth enabling but it won't survive a volume wipe so treat it as a secondary layer only
one more thing is that when a volume is deleted it's not immediately permanent railway gives you a 48 hour window to restore it via a link sent to your email
for your storage buckets, i think railway's don't currently document an official automated backup solution for them, so worth raising that directly with railway support or on the roadmap
Hope this help you :)
0x5b62656e5d
Well pretty much the only way to back up production data is through Railway backups, which can be managed in the database configuration panel. If an AI agent (eg, Cursor) has access to Railway CLI or any access to production environments, you should always be mindful and supervise the commands it's executing.
25 days ago
It is being indicated that even that would not be enough to contain the blast radius of this potential issue.
https://x.com/lifeof%5Fjer/status/2048103471019434248
They did indicate the situation is resolved. I am just wondering how to avoid it.
document8n
It is being indicated that even that would not be enough to contain the blast radius of this potential issue. <https://x.com/lifeof%5Fjer/status/2048103471019434248> They did indicate the situation is resolved. I am just wondering how to avoid it.
25 days ago
You can create backups to S3-compatible providers as Domehane suggested above.
25 days ago
>for your postgres db deploy the official railway template railwayapp-templates/postgres-s3-backups on github , it runs pg_dump on a schedule and pushes backups to an external s3-compatible bucket like cloudflare r2 or backblaze b2 and since these backups live outside railway, a volume deletion can't touch them
So I have to setup and monitor an additional backend provider to backup my backend provider? This is a hard idea to break, but that seems excessive, and I have a number of different files to backup ( I made a 2nd db just for logs )...
Maybe I just start with a backup bucket inside of the same project to keep it together...
It feels like there is a better way to approach this.
document8n
\>for your postgres db deploy the official railway template railwayapp-templates/postgres-s3-backups on github , it runs pg\_dump on a schedule and pushes backups to an external s3-compatible bucket like cloudflare r2 or backblaze b2 and since these backups live outside railway, a volume deletion can't touch them So I have to setup and monitor an additional backend provider to backup my backend provider? This is a hard idea to break, but that seems excessive, and I have a number of different files to backup ( I made a 2nd db just for logs )... Maybe I just start with a backup bucket inside of the same project to keep it together... It feels like there is a better way to approach this.
25 days ago
you can actually use a railway bucket as your backup destination , railway's own docs list backups as a valid use case for buckets and since they're fully s3-compatible and the backup template supports it via the aws_s3_endpoint variable. so no external provider needed everything stays inside railway
and as i mentionned before if the whole project gets deleted, the bucket goes with it since buckets are services inside the project. railway does give you a 48 hour window to recover a deleted project though
Hope i help you find the answer :)
Status changed to Solved ray-chen • 24 days ago
Status changed to Open ray-chen • 24 days ago
24 days ago
Quick context on the agent incident: we're rolling out additional safeguards around agentic workflows. One already live: delayed deletion when the API is used to drop a volume. We're also decoupling backups from volumes so a deletion can't take them with it. More coming on both safety and usability for agentic workflows 🙂
On backups generally, for anything important I'd keep an off-site copy. A Railway bucket inside the same project is convenient, but it's not a real backup if something takes the whole project down. The 3-2-1 Backup Strategy is a solid write-up if you're curious.
Status changed to Solved Railway • 24 days ago
