mongodb
velorie
PROOP

24 days ago

"URGENT: Database corruption after your Kernel update.

Hi Railway Team,

Yesterday at around 6:00 AM my MongoDB service crashed. It seems the host OS update (Kernel 6.19 incompatibility with my Mongo version) combined with a full disk caused an unclean shutdown.

My volume (mongodb-volume_aEN) now has a corrupted WiredTiger metadata file and throws: Fatal assertion: UnsupportedFormat: Application metadata.

I have 200 real customer accounts on this disk and I cannot lose this data. Since I don't have UI backups enabled, do you have a block-level volume snapshot from yesterday BEFORE the crash (before 6:00 AM) that you can restore to save my business? Please help."

$20 Bounty

4 Replies

Status changed to Awaiting Railway Response Railway 24 days ago


24 days ago

Your MongoDB data is almost certainly not corrupted. The logs from your current deployment show WiredTiger completing validation on every collection with "No corruption found" - the service is crash-looping for a different reason.

This is a known issue with mongo:8.x images where glibc's rseq support conflicts with an internal memory allocator, causing a fatal assertion on startup. The fix is to add the environment variable GLIBC_TUNABLES=glibc.pthread.rseq=1 to your MongoDB service and redeploy. Once set, MongoDB should start cleanly and your data will be accessible.

Regarding block-level snapshots, we don't maintain internal snapshots, WAL recovery, or PITR for volumes. Databases on Railway are unmanaged at the storage layer. Going forward, enable volume backups on your MongoDB service from the service settings page so you can self-restore if something like this happens again.


Status changed to Awaiting User Response Railway 24 days ago


Status changed to Solved mykal 24 days ago


mykal

Your MongoDB data is almost certainly not corrupted. The logs from your current deployment show WiredTiger completing validation on every collection with "No corruption found" - the service is crash-looping for a different reason. This is a known issue with mongo:8.x images where glibc's rseq support conflicts with an internal memory allocator, causing a fatal assertion on startup. The fix is to add the environment variable `GLIBC_TUNABLES=glibc.pthread.rseq=1` to your MongoDB service and redeploy. Once set, MongoDB should start cleanly and your data will be accessible. Regarding block-level snapshots, we don't maintain internal snapshots, WAL recovery, or PITR for volumes. Databases on Railway are unmanaged at the storage layer. Going forward, enable volume backups on your MongoDB service from the service settings page so you can self-restore if something like this happens again.

velorie
PROOP

24 days ago

I added glibc.pthread.rseq=1 and it still crashes


Status changed to Awaiting Railway Response Railway 24 days ago


velorie
PROOP

23 days ago

Hi support, however, I had to change the Docker to a different one and my database no longer crashes, thanks for the help, but now I can't even get data from it, I tried using the old MongoDB password from a few days ago and nothing helps, can you help me?

obraz_2026-05-22_093114552.png

Attachments


23 days ago

This thread has been marked as public for community involvement, as it does not contain any sensitive or personal information. Any further activity in this thread will be visible to everyone.

Status changed to Open mykal 23 days ago


sheeki03
FREE

22 days ago

Since changing the Docker image stopped the crash, I would treat the current problem as an auth or data-directory mismatch first, not confirmed corruption.

The common trap is that Mongo credentials are not recreated from MONGO_INITDB_ROOT_USERNAME and MONGO_INITDB_ROOT_PASSWORD when a data directory already exists. Those variables only initialize an empty database directory. If your old volume is mounted, Mongo keeps the users that already exist in the admin database. Changing Railway variables later will not change that existing user's password.

I would check this in order:

  1. Confirm the original volume is mounted to the same dbPath the new image uses. For the official Mongo image, that is usually /data/db. If the new image uses a different path, Mongo can start cleanly but read a different empty database directory.

  2. Confirm the connection string uses the right auth database. For the root user created by the official Mongo image, the URI usually needs authSource=admin.

  3. Do not keep switching images until you know which dbPath is being used. First preserve the current volume, then verify Mongo is reading the same WiredTiger files from the original volume.

  4. If the old password is truly unknown, isolate the database first: remove public access, stop app traffic, take a backup or snapshot if available, start Mongo temporarily without auth against the same data directory, reset the admin user's password in mongosh, then restart with auth enabled again.

I would verify whether the new container is mounted on the same Mongo data directory as the old one, and whether the app is authenticating against admin instead of the application database.


Welcome!

Sign in to your Railway account to join the conversation.

Loading...