a month ago
We currently have a messaging application and we are experiencing service interruptions (including API, Redis, and RabbitMQ shutdowns) triggered by logging rate limits on our hosting platform.
The Root Cause
Currently, we are testing the application's "offline mode." When a batch of users transitions from offline to online, the system processes all pending messages simultaneously. This surge generates a high volume of logs that exceeds Railway's platform-wide limit of 500 lines per second.
Impact
When this threshold is crossed, Railway’s security protocol triggers:
Service Termination: The platform may force-shutdown services or drop logs.
Manual Intervention is Required: Because these are platform-enforced shutdowns, our automated restart policies are bypassed, requiring a manual restart of the environment.
Can you please assist with what we can do to either turn off these logs on a conditional basis or some other solution from Railway? Otherwise, it seems Railway may not be the platform for our solution.
Thank you.
2 Replies
a month ago
Under no circumstances do we terminate or shutdown an application in anyway when a logging rate limit is reached.
That log is purely informational, nothing else happens besides us not saving the excess logs to our log store.
Your application is crashing, printing a lot of logs in the process, and you are misinterpreting that error log as us shutting your service down.
Status changed to Awaiting User Response Railway • 27 days ago
a month ago
This thread has been marked as public for community involvement, as it does not contain any sensitive or personal information. Any further activity in this thread will be visible to everyone.
Status changed to Open Railway • 27 days ago
24 days ago
Hey,
it does seem to be an application-level issue. I'd try removing logging, or sending it in batch. Or even using an external logging tool.