MongoDB volume crashes every ~2 minutes in production (disk reported full inside container)
clujball
HOBBYOP

20 days ago

We are experiencing critical instability on our MongoDB service in production.
The database becomes completely unreachable every 1–2 minutes, and no client can connect during these periods.

Connectivity impact:

  • x emoji Application cannot connect

  • x emojimongosh cannot connect (internal or public URL)

  • x emojimongodump cannot run

  • x emoji TCP connections are reset (ECONNRESET)

At the moment, there is no reliable way to access the database from anywhere.

Key observations

  • Inside the MongoDB container, the mounted volume (/data/db) is reported as 100% full:

    df -h /data/db
    /dev/zd22208  4.4G  4.4G  0  100% /data/db
    
  • In contrast, the Railway UI reports the same volume (industrious-volume) as using only ~300–500 MB.

  • MongoDB repeatedly crashes or restarts, but crash logs are not visible:

    • No relevant crash logs appear in Railway logs

    • No clear shutdown or panic messages are exposed

  • This makes it difficult to diagnose the exact failure point while the service continues to flap.

This suggests a filesystem-level disk exhaustion not properly reflected in Railway’s volume metrics, potentially combined with missing or inaccessible runtime logs.

Impact

  • rotating_light emoji Production database completely inaccessible

  • rotating_light emoji Application downtime

  • rotating_light emoji No backups possible

  • rotating_light emoji No visibility into crash root cause (logs missing)

  • rotating_light emoji Risk of data corruption if instability continues

Environment

  • Service: MongoDB

  • Environment: Production

  • Volume: industrious-volume

  • MongoDB version: 7.0.x

  • Region: europe-west4

Request

We need urgent help to:

  1. Restore stable access to the MongoDB database

  2. Understand why the volume appears full inside the container but not in Railway metrics

  3. Investigate why MongoDB crash/restart logs are not visible

  4. Ensure the database can be stabilized without data loss

This issue is production-blocking and critical.

Solved$10 Bounty

Pinned Solution

darseen
HOBBYTop 1% Contributor

20 days ago

Have you tried to increase the volume size in Volume > Settings > Volume Size ?

1 Replies

darseen
HOBBYTop 1% Contributor

20 days ago

Have you tried to increase the volume size in Volume > Settings > Volume Size ?


Status changed to Solved brody 20 days ago


Loading...