0b97a174-a572-4b03-887b-98a62c42f368

Does this persist even after redeploying?

yes

we have the problems since days

we log a lot

there is a ratelimit in logs, perhaps you're exceeding the rate limit

where can i find it?

I'm not sure what the rate limit exactly is 🤔

there is actually nothing written about limit, just about 30 day history

Nevermind, I swear I remember one of the conductors mention a log rate limit, but I can't seem to find anything like that

yes i have the same problem, if i hit the limit, it would be nice if i would get nottified about it

I have brought this up to the team, there is definitely a limit and the limit should be listed in the technical specifications section of the docs, but it's not right now.

however, you would still see some logs, it's just that the logs that go over the threshold are discarded, it's also a very reasonable limit, so I'm confident in saying that you are not running into any logging limits.

what I do think is happening is that some of the logging infrastructure was shut down yesterday during the incident and unfortunately yours never got reconnected.

Thank goodness, thought I was schizo or something

<@1060856209332260864> - missing all deployment logs

Looking

<@778644535948869724> can you link to your service please?

That's not a valid project ID

deployment id

7e6d9a13-b546-4b51-a2bb-12af6682a7d5

this is the service

sorry

open your service and then copy the url, paste it here

ray works for railway and is the only one who would be able to look at the project

https://railway.app/project/f1c50ffb-536c-4274-8f82-c57e126a10dc/service/7e6d9a13-b546-4b51-a2bb-12af6682a7d5/settings

7e6d9a13-b546-4b51-a2bb-12af6682a7d5
this is the service id i think

the linked was the deployment id

all good

This is the testing env right? You have a deployment from 2hrs ago and the logs appear to be fine

Is it just a single inactive deployment missing logs?

it is on dev and prod

sometimes they are there, and then they hide again

any updates on logging limit?

there's nothing in the docs yet, and tbh I've found that most people would never hit the log per second limit unless they where absolutely spamming console.log, exactly what the limit is there to prevent, from my experience, it's a very sensible limit.
but limits should of course always be documented, so I will bump this to the right person!

So I reckon streaming to a .log file is the way to go if I were to stress test?

Or is also writing to a stream restricted? Dk how this limits work or apply, didnt test yet

you could stream as fast as you want to a file, but that wouldn't cause logs to show on the deployment logs.

the limit is simply just how many log lines per second railways logging infrastructure will process before it drops log lines, and for railway to even parse logs you need to log unbuffered to stdout / stderr.