Hey there! We've found the following might help you get unblocked faster:

• 🧵 Railway logging is barely usable, with significant errors and limitations

• 📚 Public Networking

If you find the answer from one of these, please let us know by solving the thread!

Hello,

This means your application is logging too frequently - 500 log lines per second equates to logging every 2ms or more. To maintain quality of service for all users, we drop logs that exceed the threshold.

To reduce your logging rate, it looks like your application is pretty-printing whole JSON documents. Minifying those into single lines will likely solve this issue entirely.

Best,

Brody

ok i'll stop using json.stringify(obj, null, 2) makes for hard to read logs though. i wish you could just pay to up the limit

We have great support for structured logging, so it should actually make it easier.

https://docs.railway.com/guides/logs#structured-logs

hmm, ok. i can try that. i replaced all my stringify calls in console logs with a smartStringify function that does not do multi-line stringify output if running on railway (but does use it on local so i can develop). Unfortunately, despite using it throughout my codebase, the logs are still being dropped hugely. It seems like it's the speed of processing that gets me in trouble. But I have to run batch jobs where I do have to watch for problems and am processing a ton of data. If I use the structured logs will that apply the log rates differently to the different classes of log e.g. info, alert etc?

There will not be different rates depending on the log type; the threshold is on the number of log lines printed per second per replica.

So with structured logs, you print your JSON minified so it will be a single line, versus pretty-printing JSON, where the entire document is printed out line by line.

Please see our docs for more information on this topic.

https://docs.railway.com/reference/logging#logging-throughput

ok. however, i did modify things so that each console log can only output the json on a single line. yet we still have situations where a lot of single lines are printed out in a tight loop. I think even with the structured logs, that would hit the limits, yeah? in that case I'll find other ways to reduce those unless strictly debugging something...

Yep, you can always hit limits regardless of how you are logging, so please do look into alternative methods to reduce your logging rate.