24 days ago
"URGENT: Database corruption after your Kernel update.
Hi Railway Team,
Yesterday at around 6:00 AM my MongoDB service crashed. It seems the host OS update (Kernel 6.19 incompatibility with my Mongo version) combined with a full disk caused an unclean shutdown.
My volume (mongodb-volume_aEN) now has a corrupted WiredTiger metadata file and throws: Fatal assertion: UnsupportedFormat: Application metadata.
I have 200 real customer accounts on this disk and I cannot lose this data. Since I don't have UI backups enabled, do you have a block-level volume snapshot from yesterday BEFORE the crash (before 6:00 AM) that you can restore to save my business? Please help."
4 Replies
Status changed to Awaiting Railway Response Railway • 24 days ago
24 days ago
Your MongoDB data is almost certainly not corrupted. The logs from your current deployment show WiredTiger completing validation on every collection with "No corruption found" - the service is crash-looping for a different reason.
This is a known issue with mongo:8.x images where glibc's rseq support conflicts with an internal memory allocator, causing a fatal assertion on startup. The fix is to add the environment variable GLIBC_TUNABLES=glibc.pthread.rseq=1 to your MongoDB service and redeploy. Once set, MongoDB should start cleanly and your data will be accessible.
Regarding block-level snapshots, we don't maintain internal snapshots, WAL recovery, or PITR for volumes. Databases on Railway are unmanaged at the storage layer. Going forward, enable volume backups on your MongoDB service from the service settings page so you can self-restore if something like this happens again.
Status changed to Awaiting User Response Railway • 24 days ago
Status changed to Solved mykal • 24 days ago
mykal
Your MongoDB data is almost certainly not corrupted. The logs from your current deployment show WiredTiger completing validation on every collection with "No corruption found" - the service is crash-looping for a different reason. This is a known issue with mongo:8.x images where glibc's rseq support conflicts with an internal memory allocator, causing a fatal assertion on startup. The fix is to add the environment variable `GLIBC_TUNABLES=glibc.pthread.rseq=1` to your MongoDB service and redeploy. Once set, MongoDB should start cleanly and your data will be accessible. Regarding block-level snapshots, we don't maintain internal snapshots, WAL recovery, or PITR for volumes. Databases on Railway are unmanaged at the storage layer. Going forward, enable volume backups on your MongoDB service from the service settings page so you can self-restore if something like this happens again.
24 days ago
I added glibc.pthread.rseq=1 and it still crashes
Status changed to Awaiting Railway Response Railway • 24 days ago
23 days ago
Hi support, however, I had to change the Docker to a different one and my database no longer crashes, thanks for the help, but now I can't even get data from it, I tried using the old MongoDB password from a few days ago and nothing helps, can you help me?
Attachments
23 days ago
This thread has been marked as public for community involvement, as it does not contain any sensitive or personal information. Any further activity in this thread will be visible to everyone.
Status changed to Open mykal • 23 days ago
22 days ago
Since changing the Docker image stopped the crash, I would treat the current problem as an auth or data-directory mismatch first, not confirmed corruption.
The common trap is that Mongo credentials are not recreated from MONGO_INITDB_ROOT_USERNAME and MONGO_INITDB_ROOT_PASSWORD when a data directory already exists. Those variables only initialize an empty database directory. If your old volume is mounted, Mongo keeps the users that already exist in the admin database. Changing Railway variables later will not change that existing user's password.
I would check this in order:
-
Confirm the original volume is mounted to the same dbPath the new image uses. For the official Mongo image, that is usually /data/db. If the new image uses a different path, Mongo can start cleanly but read a different empty database directory.
-
Confirm the connection string uses the right auth database. For the root user created by the official Mongo image, the URI usually needs authSource=admin.
-
Do not keep switching images until you know which dbPath is being used. First preserve the current volume, then verify Mongo is reading the same WiredTiger files from the original volume.
-
If the old password is truly unknown, isolate the database first: remove public access, stop app traffic, take a backup or snapshot if available, start Mongo temporarily without auth against the same data directory, reset the admin user's password in mongosh, then restart with auth enabled again.
I would verify whether the new container is mounted on the same Mongo data directory as the old one, and whether the app is authenticating against admin instead of the application database.