Runtime logs not captured after recent outage — only startup logs visible
sapshah
HOBBYOP

17 days ago

Description:

After the May 19, 2026 GCP account suspension incident, runtime/application logs are no longer being captured for our services. Only the initial startup burst

appears in both the Railway dashboard log viewer and railway logs CLI command.

Project: svantic

Affected services: All services (gateway, mesh, cortex, bff, dashboard, api-edge)

Environment: production

Symptoms:

• Container startup logs (first ~20 lines) appear correctly

• After startup, zero runtime logs appear — no request logs, no session logs, no error logs

• Services ARE processing requests successfully (verified via API responses and trace data)

• The logging library (log4js) writes structured JSON to stdout, which worked before the outage

• We have tried: redeploying from the dashboard, redeploying via railway up CLI, deleting and recreating services entirely — none resolved the issue

Expected: Runtime stdout should be captured continuously, not just during startup.

Environment details:

• All services use Dockerfile builder

• Node.js 20 (Alpine)

• log4js with console appender writing structured JSON to stdout

• Services were recreated fresh after the outage (not restored from pre-outage state)

Solved$10 Bounty

1 Replies

Status changed to Open Railway 17 days ago


sheeki03
FREE

17 days ago

This has to be split into two cases: platform log ingestion versus application logging. Because it affects multiple services, survived service recreation, and appears in both the dashboard and railway logs, do not start by rewriting log4js. First prove whether plain stdout/stderr is still being captured after startup.

Add this temporary heartbeat to one affected service, deploy it, then watch both Log Explorer and railway logs for 2-3 minutes:

setInterval(() => {
  console.log(JSON.stringify({
    marker: "railway-log-probe",
    stream: "stdout",
    ts: new Date().toISOString(),
  }));
  console.error(JSON.stringify({
    marker: "railway-log-probe",
    stream: "stderr",
    ts: new Date().toISOString(),
  }));
}, 10000);

Then split the result:

  1. If the probe also only appears during startup, treat this as a runtime log ingestion issue, not a log4js configuration problem. Escalate it with:

    • project id
    • environment id
    • affected service ids
    • deployment ids
    • replica region
    • the exact marker string above
    • UTC timestamp range where the marker was emitted but absent
    • confirmation that both Log Explorer and railway logs miss the same lines
  2. If the probe appears continuously, stdout/stderr capture is working and the issue is inside the app logging path. Check whether log4js still uses a console appender after startup, whether runtime config switches it to file-only appenders, and whether the Docker CMD keeps the Node process in the foreground instead of redirecting output to a file or detached child process.

Also compare the Deployment Logs panel against Observability/Log Explorer. If the deploy/startup panel shows the first burst but Log Explorer never receives runtime markers, include that distinction. It narrows the issue to the runtime log pipeline rather than the container process.

The fastest minimal reproduction is a fresh Railway service in the same project/environment using this command:

node -e 'setInterval(() => { console.log("stdout railway-log-probe " + new Date().toISOString()); console.error("stderr railway-log-probe " + new Date().toISOString()); }, 10000)'

If that minimal service also stops logging after the startup burst, it gives support a clean platform-side repro without your application code. If it logs continuously, compare the minimal service against the affected Dockerfiles and log4js runtime config.


Status changed to Awaiting User Response Railway 17 days ago


Railway
BOT

9 days ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway 9 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...