MongoDB reads hang indefinitely, writes succeed — deployment stuck 24+ hours
7scgb4t8vc-cpu
HOBBYOP

21 days ago

My Flask app (Gunicorn) connects to MongoDB Atlas. For ~24 hours, all cursor-based read queries (collection.find()) hang indefinitely and never return — even with socketTimeoutMS=None (no timeout at all). Write operations (insert_one) succeed normally and quickly. I've ruled out: credentials (verified via MongoDB Compass from my own machine), connection string format (tried SRV and standard), connection pool size, IPv6 egress (tested both on/off, no difference), and Gunicorn worker timeout (raised to 120s, no difference). This looks like a network-level stall between Railway and Atlas specifically affecting read traffic.

Deployment ID: f5545f50-753a-407c-892c-5138fa87b14c

$10 Bounty

6 Replies

Railway
BOT

21 days ago

This thread has been opened as a bounty so the community can help solve it.

Status changed to Open Railway 21 days ago


manuproject
HOBBY

21 days ago

this matches a known gunicorn + pymongo issue, not a railway/atlas network problem. if your MongoClient is created at module level (import time) rather than inside each worker after fork, every gunicorn worker inherits the same connection pool and sockets from the parent process. writes often slip through because theyre one quick round trip, but reads do multiple round trips over the same socket (find then getMore), and that's exactly what hangs forever on a shared/corrupted pooled connection.

fix: move MongoClient() creation out of any module-level/global code and either instantiate it lazily on first use per worker, or create it inside gunicorn's post_fork hook in gunicorn.conf.py. also check if youre using --preload in your gunicorn start command, if you are, thats almost certainly the trigger, since preload creates the app (and your client) before forking. removing --preload or moving client creation to post_fork should fix it immediately

reference: pymongo docs explicitly warn against sharing a MongoClient across forked processes



manuproject

this matches a known gunicorn + pymongo issue, not a railway/atlas network problem. if your MongoClient is created at module level (import time) rather than inside each worker after fork, every gunicorn worker inherits the same connection pool and sockets from the parent process. writes often slip through because theyre one quick round trip, but reads do multiple round trips over the same socket (find then getMore), and that's exactly what hangs forever on a shared/corrupted pooled connection. fix: move MongoClient() creation out of any module-level/global code and either instantiate it lazily on first use per worker, or create it inside gunicorn's post_fork hook in gunicorn.conf.py. also check if youre using --preload in your gunicorn start command, if you are, thats almost certainly the trigger, since preload creates the app (and your client) before forking. removing --preload or moving client creation to post_fork should fix it immediately reference: pymongo docs explicitly warn against sharing a MongoClient across forked processes

7scgb4t8vc-cpu
HOBBYOP

21 days ago

Thanks for the suggestion — I implemented this exactly (post_fork hook creating a fresh MongoClient per worker, with verbose logging to confirm). Logs show it working correctly:

[post_fork] worker 4: creating MongoClient

[post_fork] worker 4: MongoClient created successfully

[post_fork] worker 4: creating indexes

[post_fork] worker 4: indexes created successfully

[post_fork] worker 4: post_fork hook completed successfully

Despite this, the exact same hang still occurs a few minutes later on the very next find() read — writes succeed, reads hang indefinitely, same as before. This rules out the fork/shared-connection-pool theory, since each worker now provably has its own fresh MongoClient created after fork.

I've also already ruled out: credentials, connection string format (SRV vs standard), pool size, IPv6 egress, Gunicorn worker timeout, and MongoDB Atlas whitelist/collection size (only 5.6k docs). Direct connection via MongoDB Compass from my own machine works instantly with the same credentials/query.

At this point it really does look like something in Railway's network path specifically affecting long-lived read sockets to Atlas, rather than an application-level config issue. Any chance this could get escalated to someone with visibility into Railway's actual network infrastructure?


7scgb4t8vc-cpu

Thanks for the suggestion — I implemented this exactly (post_fork hook creating a fresh MongoClient per worker, with verbose logging to confirm). Logs show it working correctly: [post_fork] worker 4: creating MongoClient [post_fork] worker 4: MongoClient created successfully [post_fork] worker 4: creating indexes [post_fork] worker 4: indexes created successfully [post_fork] worker 4: post_fork hook completed successfully Despite this, the exact same hang still occurs a few minutes later on the very next find() read — writes succeed, reads hang indefinitely, same as before. This rules out the fork/shared-connection-pool theory, since each worker now provably has its own fresh MongoClient created after fork. I've also already ruled out: credentials, connection string format (SRV vs standard), pool size, IPv6 egress, Gunicorn worker timeout, and MongoDB Atlas whitelist/collection size (only 5.6k docs). Direct connection via MongoDB Compass from my own machine works instantly with the same credentials/query. At this point it really does look like something in Railway's network path specifically affecting long-lived read sockets to Atlas, rather than an application-level config issue. Any chance this could get escalated to someone with visibility into Railway's actual network infrastructure?

manuproject
HOBBY

21 days ago

good test, that actually narrows it down rather than ruling everything out. small operations (write acks) succeeding while read responses hang forever, combined with your own machine working fine directly, is the classic signature of an mtu/pmtud black hole - something in the path between railways network and atlas is dropping the icmp packet that tells both sides to fragment large responses, so anything bigger than one packet just vanishes and the socket hangs waiting.

two ways to confirm it fast:

add compression to your connection string: &compressors=zstd (or zlib). if reads suddenly start working, that's confirmation, compression shrinks responses below the point where they need fragmenting.

from a railway shell (railway run bash or the console tab), try curl -v --max-time 10 https:// - if that hangs too on anything beyond the initial connection, its the same path issue, not a mongo-specific one.

if compression fixes it, that's your workaround without needing infra access. if it doesnt, the curl test result is exactly what railways team will want to see when you escalate, since it isolates it from your app code entirely


manuproject

good test, that actually narrows it down rather than ruling everything out. small operations (write acks) succeeding while read responses hang forever, combined with your own machine working fine directly, is the classic signature of an mtu/pmtud black hole - something in the path between railways network and atlas is dropping the icmp packet that tells both sides to fragment large responses, so anything bigger than one packet just vanishes and the socket hangs waiting. two ways to confirm it fast: add compression to your connection string: &compressors=zstd (or zlib). if reads suddenly start working, that's confirmation, compression shrinks responses below the point where they need fragmenting. from a railway shell (railway run bash or the console tab), try curl -v --max-time 10 https://<your-atlas-hostname> - if that hangs too on anything beyond the initial connection, its the same path issue, not a mongo-specific one. if compression fixes it, that's your workaround without needing infra access. if it doesnt, the curl test result is exactly what railways team will want to see when you escalate, since it isolates it from your app code entirely

7scgb4t8vc-cpu
HOBBYOP

20 days ago

Thanks — just to clarify, I already tried both of these yesterday:

Compression — added &compressors=zstd to the connection string, redeployed. Same hang, no change.

curl test from Railway console — ran curl -v --max-time 10 https://project1.fslkmc3.mongodb.net, got:

  • Could not resolve host: project1.fslkmc3.mongodb.net

curl: (6) Could not resolve host: project1.fslkmc3.mongodb.net

This looked like a DNS resolution failure specific to Railway's container.

Since both of these were already tested with no fix, is there a different next step, or does the DNS failure result change your read on this?


marlonwq
FREE

19 days ago

That DNS error is actually a red herring. Atlas SRV clusters don't have standard A records, so standard HTTP tools like curl will always fail to resolve them. Since your app was able to write data and create indexes, your DNS and network routing between Railway and Atlas are working perfectly fine.

The real culprit here is the combination of socketTimeoutMS=None and the fact that the hang happens "a few minutes later".

You are dealing with a silent TCP connection drop. Cloud environments like Railway aggressively prune idle connections from their NAT tables after a few minutes to free up resources. They do this silently, without sending a TCP RST packet back to your app.

When your next find() query runs, PyMongo blindly pulls one of these dead sockets from the pool. Because you set socketTimeoutMS=None, you've instructed the driver to never give up. It just sits there waiting indefinitely for data on a connection that the network firewall has already destroyed. Writes sometimes bypass this because they are quick, single-packet acks that might happen on fresh sockets or trigger different driver behavior.

To fix this, you need to stop the pool from holding onto stale connections.

First, get rid of socketTimeoutMS=None (or set it to a normal value like 60000). If a socket dies, you want the driver to throw a timeout, discard it, and reconnect. Second, append maxIdleTimeMS=120000 to your connection URI. This instructs PyMongo to automatically recycle any connection that has been idle for 2 minutes, ensuring your app closes them gracefully before Railway's NAT firewall has a chance to quietly kill them.


Welcome!

Sign in to your Railway account to join the conversation.

Loading...