a month ago
Our team has hardened security after the Vercel incident and accidentally demoted me from Admin to Deployer instead of Member. Since then, I have been promoted to Member, but 1 of our 3 Mongo deployments in BWR-TOOLS Services have stopped working. BWR-TOOLS-2 just loads infinitely on the frontend and does not connect via the public URI. Railway frontend shows: "Attempting to connect to the database...". Each time I restart, logs show: "Railway rate limit reached for deployment, update your application to reduce the logging rate. Messages dropped: 301"
Pinned Solution
a month ago
because the pthread_create failed error means your mongodb container is running out of threads/resources because too many connections are hitting it at once so the apps connecting to bwr-tools-2 need to limit their connection pool size
6 Replies
a month ago
This thread has been marked as public for community involvement, as it does not contain any sensitive or personal information. Any further activity in this thread will be visible to everyone.
Status changed to Open Railway • about 1 month ago
a month ago
Hello meta-light,
two things to check , first go to bwr-tools-2 variables tab in the railway dashboard and confirm your mongodb connection string is still there and correct i think the security changes ur team made might have affected that specific service's variables without touching the other two
and second the messages dropped 301 means your app is hitting railway's 500 log lines sec rate limit, which is hiding the real error you need to temporarily reduce your connection retry logging so u can see what error is being thrown before the logs get dropped and once you see the real error it'll be much easier to fix
so redeploy mongodb first wait for it to be fully online then redeploy bwr-tools-2
a month ago
MONGO_PUBLIC_URL is still there but has a new warning about egress charges, which I assume we can ignore. On your 2nd flag here, BWR-TOOLS-2 is the MongoDB container, Are you saying the applications connecting to this DB are causing too much logging?
meta-light
MONGO\_PUBLIC\_URL is still there but has a new warning about egress charges, which I assume we can ignore. On your 2nd flag here, BWR-TOOLS-2 is the MongoDB container, Are you saying the applications connecting to this DB are causing too much logging?
a month ago
can you check the raw logs of bwr-tools-2 and tell me what appears before the messages dropped warning? that'll show the real error since railway drops everything after the limit hits
a month ago
Prior to the warning is it a loop of logs like this:
```
ERROR(4850900): pthread_create failed
Connection accepted
Connection ended
ERROR(4850900): pthread_create failed
Connection accepted
ERROR(4850900): pthread_create failed
Unable to schedule a new loop for the session workflow
```
a month ago
okay, can you check how many app services are connecting to bwr-tools-2 and whether they're opening a new db connection on every request instead of reusing one?
domehane
okay, can you check how many app services are connecting to bwr-tools-2 and whether they're opening a new db connection on every request instead of reusing one?
a month ago
because the pthread_create failed error means your mongodb container is running out of threads/resources because too many connections are hitting it at once so the apps connecting to bwr-tools-2 need to limit their connection pool size
Status changed to Solved brody • about 1 month ago