5 months ago
I've discussed most of these issues with support and they've indicated it's not on their roadmap to resolve them so I thought I'd post a thread to gather support or suggestions from others as I love Railway otherwise (
)
For example: we have a simple Node server that runs a CRON job every hour to process documents. This prints a one-line report for each document, which can mean a few hundred logs within one second.
Railway instantly deletes many of these logs due to its 500 log/sec limit. This limit still applies if exporting logs to a different provider via locomotive.
Logs appear in the wrong order. If you pretty print a JSON object, the lines of this JSON object can appear in a mess with each other and with other logs. (See screenshot showing spliced JSON object mixed with out-of-order progress counters)
Support indicated that to get around the above issue, you should print JSON objects minified with a message attribute. However, these JSON objects then become invisible - you can't see them straight away when browsing logs, and they don't appear in searches. If an email address appears in a JSON object or an error, and you search that email address - it won't appear. (Unless you already know the exact error or attribute in which it's going to appear - which you won't, if you're debugging)
In fact, searching for any email address e.g. "tom@example.com" breaks the search because of the @. You have to search "tom example.com", which might give undesireable matches.
When scrolling through logs, it can jump around in a glitchy way. To reproduce: enter a search term, select a line and click "View in context", scroll down and click "Load more". If you were at 20:30, instead of letting you continue scrolling from 20:30 onwards - it jumps you all the way to 20:40, i.e. the bottom of the loaded results. You have to go back up to 20:30 to find where you were and scroll from there.
Attachments
0 Threads mention this feature
15 Replies
5 months ago
I didn’t know about the 500/sec logs limit. Is that global or per service?
I also reported a bug with filters and yet to receive a response.
5 months ago
The error that appears in logs is:
Railway rate limit of 500 logs/sec reached for replica, update your application to reduce the logging rate. (100 logs dropped)
Not completely sure what replica refers to but sounds like it's a service, most likely
5 months ago
In addition to reordering, the entries also tend to disappear transiently. Some entries simply never show up unless you reload the entire web page. You never know for sure if a log entry is missing because the code path wasn't hit, or if it's a Railway bug - so you constantly refresh the page.
5 months ago
Railway rate limit of 500 logs/sec reached for replica, update your application to reduce the logging rate. (100 logs dropped)
This actually crashes my services sometimes
5 months ago
Has anyone found a fix for this? My system keeps looping when scraping with Browserless.
it occurred when this issue came up:
[Railway rate limit of 500 logs/sec reached for replica, update your application to reduce the logging rate. Messages dropped: 5456]
stefanflorea00
Railway rate limit of 500 logs/sec reached for replica, update your application to reduce the logging rate. (100 logs dropped)This actually crashes my services sometimes
5 months ago
Your service crashing would be completely unrelated to dropping logs.
You would want to add more observability in order to find out why your application crashes.
5 months ago
Logs appear in the wrong order. If you pretty print a JSON object, the lines of this JSON object can appear in a mess with each other and with other logs. (See screenshot showing spliced JSON object mixed with out-of-order progress counters)
I have this issue as well. I'm logging JSON objects with pino (in a node app), and log strings and json objects gets mixed into each other so its impossible to figure out which log is which. Can't seem to find anything about this in the docs either.
5 months ago
I use to see the logs like this with winston and json.pretty, changing to json fixed them
5 months ago
The button "See in context" was removed when you released the new logging?
I'm having use issues to search for stuff, specifically if I enter an Until date it can't never find an event I know it's there.
Another thing that is quite annoying is that I can't use browser's search for finding some text that is not in view (below the fold).
4 months ago
When scrolling through logs, it can jump around in a glitchy way. To reproduce: enter a search term, select a line and click "View in context", scroll down and click "Load more". If you were at 20:30, instead of letting you continue scrolling from 20:30 onwards - it jumps you all the way to 20:40, i.e. the bottom of the loaded results. You have to go back up to 20:30 to find where you were and scroll from there.
Is there a fix for this bug planned? It's still present with the recently updated observability interface and is pretty painful. It also happens when scrolling up through logs or e.g. errors.
When scrolling through logs, it can jump around in a glitchy way. To reproduce: enter a search term, select a line and click "View in context", scroll down and click "Load more". If you were at 20:30, instead of letting you continue scrolling from 20:30 onwards - it jumps you all the way to 20:40, i.e. the bottom of the loaded results. You have to go back up to 20:30 to find where you were and scroll from there.Is there a fix for this bug planned? It's still present with the recently updated observability interface and is pretty painful. It also happens when scrolling up through logs or e.g. errors.
4 months ago
Hey there, sorry about this! I see what you mean and I'm working on a fix.
wahmad
The button "See in context" was removed when you released the new logging?I'm having use issues to search for stuff, specifically if I enter an Until date it can't never find an event I know it's there.Another thing that is quite annoying is that I can't use browser's search for finding some text that is not in view (below the fold).
4 months ago
The button "See in context" was removed when you released the new logging?
We didn't remove this but we have since made it more easy to find. Hopefully this is fixed for you as a result!
> specifically if I enter an Until date it can't never find an event
I think we've also fixed this one since, but if not please give me a simple reproduction and we will fix it for you.
> Another thing that is quite annoying is that I can't use browser's search for finding some text that is not in view (below the fold).
We only render 100 logs on either side of the logs you see in the visible window. This is called "virtualization" and we use it to make sure the page renders quickly, doesn't crash. I can see how that is annoying though and maybe there's another solution (e.g. "find" box that mimics the browser behavior)
4 months ago
I noticed you can no longer right click and open in a new tab for "See in context". And if you do click it in the same tab, then when you click "Back" to get back to where you were - it's completely broken and makes you lose your place in the logs. So the "See in context" button is now unuseable 
I noticed you can no longer right click and open in a new tab for "See in context". And if you do click it in the same tab, then when you click "Back" to get back to where you were - it's completely broken and makes you lose your place in the logs. So the "See in context" button is now unuseable
4 months ago
Yep, losing the right click is a mistake. Will fix that. Going back I think will almost always scroll you to the end of the stream which is unfortunate but challenging to workaround for every case. Will think on it!
