21 days ago
Hi all,
I have to download my data so I can get service back up for my customers. Is it possible if the DB is down? I have backups but I can't seem to download them.
12 Replies
EU West is down. How do I download my railway data dump or backup if the database is down?
status.railway.com is not reflecting the actual status. Besides the earlier 5+ hour outage, this is another 5+ hour outage. I've now migrated to planetscale, will monitor when railway is back online so I can recover the missing data
20 days ago
What's happening for you? Everything is working fine as I'm aware
Is your database showing the crashed state? have you tried redeploying?
20 days ago
im facing the same issue,. It doses show any error, but the site is down
20 days ago
I have 20+ services in EU-West they are all fine and live ATM
Figured out what happened. The May 19–20 outage left our MySQL volume with a torn InnoDB page, mysqld crash-loops on startup with MY-013546 and MY-013540 (doublewrite buffer also unrecoverable). This morning's automatic daily backup snapshotted the already-corrupt state and restores into the same crash loop. Yesterday's daily backup restores cleanly, so we're back up, but we've lost ~a day of writes.
Two asks:
-
Do you retain any snapshots of my MySQL volume from between yesterday's daily and ~22:43 UTC May 19 that aren't shown in the dashboard?
-
Can you start the corrupt volume with --innodb-force-recovery=3 (or give me shell access) so I can mysqldump whatever's still readable?
The central station thread is also throwing an error loading the page and posting a message
19 days ago
Hi all, it's been over 2 days, no response yet?
16 days ago
So much for 72 hour response time. Fortunately, Planetscale is running stable. But unfortunate that you do not assist with data recovery based on your own service issues.
16 days ago
Discord threads don't get put into the teams queue as they are considered community support, so aren't likely to be seen by the support team. The 72 hour response time is for when you open a private thread here on the station (as outlined here: https://docs.railway.com/platform/support#pro).
Answers to your questions:
- They only back up what you tell them to back up, so no
- You can change the start command in your service settings (to
sleep 999999if you would like to ssh in without it crashing)
I'm going to guess that this was caused by the service and/or volume being suddenly killed by GCP, causing a corrupt state.
15 days ago
Thanks for your input yellowmacaroni!