log viewer is missing entries since adding multiple services

23 days ago

My service sends email to alert me of certain log messages as they are generated. Since yesterday when I added two services to my environment (1 redis, 1 tailscale-forwarder), I am missing entries from the log viewer for my main service. I know they should be there, because I got mail about them. And there are others that I know should be there (from the pattern of messages, which is consistent) but are also not showing in the viewer.

Restricting the viewer just to the one service makes no difference.

I need help seeing these logs; they are critical.

$10 Bounty

4 Replies

Status changed to Awaiting Railway Response Railway 23 days ago


Railway
BOT

22 days ago

This thread has been opened as a bounty so the community can help solve it.

Status changed to Open Railway 22 days ago


bamboochristopher
FREE

22 days ago

In the Railway dashboard, click the Observability tab at the top instead of the per-service log panel. Set the time range to cover when your alert emails were sent, and use a search term from those alerts (e.g., a unique substring in the log message) so you know you’re matching the right events.

Filter by environment and service

In Log Explorer, filter to your environment first, then select only your main service so you’re not mixing logs from Redis and the Tailscale forwarder.

If you still don’t see entries, briefly clear service filters and check whether those same messages show up tagged as coming from another process, which would indicate mislabeling in the UI rather than true loss.

Use railway logs from the CLI

Install or open the Railway CLI and run something like: railway logs --service --since 24h.

Cross‑check timestamps and message content against your email alerts; if they appear in CLI but not the web viewer, you’re dealing with a viewer issue, not missing storage.

If logs truly aren’t stored

If the logs don’t appear in the CLI at all:

Confirm log level and logger config in your main service (e.g., ensure you didn’t accidentally move those critical alerts to a different logger or level when adding Redis/Tailscale). This kind of misconfiguration would cause alerts to fire but structured logs to go elsewhere or be dropped.

Check any new environment variables or log-related settings you added for Redis or Tailscale forwarder so you’re not redirecting stdout/stderr away from Railway’s log capture (for instance, wrapping your app in another process that doesn’t forward logs properly).


21 days ago

I need help from the Railway team on this. The deployment info for the service that has these logs is:

Project: AAF Organizational Pipeline

Environment: production

Service: aaf-tool

Deployment ID: 17a74a1b-0893-4cb8-adee-55c79221b8f0

I have email that says that at or just before 6:07:25am PT there was a log entry containing "Warning: timeout before entering data". There would be other log entries just before and after that entry containing related info. But when I use the CLI to get those logs, I see no entries with timestamps in the entire 6:07 minute:

$ railway logs 17a74a1b-0893-4cb8-adee-55c79221b8f0 --since 2026-07-29T13:06:00Z --until 2026-07-29T13:09:00Z --json

{"message":"Processing home page request","level":"error","timestamp":"2026-07-29T13:06:15.221636585Z"}

{"timestamp":"2026-07-29T13:06:15.932873744Z","message":"Processing job page request for job:EX2bNX4BHX (org:RvvRk4JAXX)","level":"error"}

{"message":"Processing rerun page request for job:EX2bNX4BHX (org:RvvRk4JAXX)","timestamp":"2026-07-29T13:06:18.031847450Z","level":"error"}

{"timestamp":"2026-07-29T13:06:18.031853710Z","level":"error","message":"Rerunning job:EX2bNX4BHX"}

{"message":"Submitted from front end: job:yt4BjFtQpd (org:yt4BjFtQpd): 20 cases","level":"error","timestamp":"2026-07-29T13:06:18.039376638Z"}

{"message":"Reloaded proxy for job:EX2bNX4BHX: Completed, 20/20 cases in 56.581331948s","timestamp":"2026-07-29T13:06:18.069440808Z","level":"error"}

{"message":"Processing job page request for job:yt4BjFtQpd (org:RvvRk4JAXX)","timestamp":"2026-07-29T13:06:18.175028513Z","level":"error"}

{"level":"error","message":"Grabbed job:yt4BjFtQpd from the waiting queue and put it in the processing queue","timestamp":"2026-07-29T13:06:21.497597780Z"}

{"timestamp":"2026-07-29T13:06:21.497603760Z","level":"error","message":"There are 0 jobs left in the waiting queue"}

{"message":"Starting processing of job:yt4BjFtQpd: Submitted, 0/20 cases in 0s","timestamp":"2026-07-29T13:06:21.500182775Z","level":"error"}

{"timestamp":"2026-07-29T13:06:21.500188515Z","message":"Reloaded proxy for job:yt4BjFtQpd: Processing, 0/20 cases in 0s","level":"error"}

{"message":"Starting web session Web test session for job:yt4BjFtQpd with driver timeout 8s","timestamp":"2026-07-29T13:06:22.547684231Z","level":"error"}

{"level":"error","timestamp":"2026-07-29T13:06:22.547689461Z","message":"Starting check session Web test session for job:yt4BjFtQpd separating checks by 1.75s"}

{"level":"error","message":"Processing job page request for job:yt4BjFtQpd (org:RvvRk4JAXX)","timestamp":"2026-07-29T13:08:03.539321222Z"}

What I do see at that time, when I look in the log viewer, is a bunch of messages from the Redis service. I need the Railway team to determine whether this is an log egress bandwidth problem and advise me how I can avoid this happening in future.


azashi
HOBBY

21 days ago

Are Redis/Tailscale running as separate services, or is Tailscale wrapped into your main service's start command/entrypoint?


azashi

Are Redis/Tailscale running as separate services, or is Tailscale wrapped into your main service's start command/entrypoint?

21 days ago

Separate services.


Welcome!

Sign in to your Railway account to join the conversation.

Loading...