17 days ago
Description:
After the May 19, 2026 GCP account suspension incident, runtime/application logs are no longer being captured for our services. Only the initial startup burst
appears in both the Railway dashboard log viewer and railway logs CLI command.
Project: svantic
Affected services: All services (gateway, mesh, cortex, bff, dashboard, api-edge)
Environment: production
Symptoms:
• Container startup logs (first ~20 lines) appear correctly
• After startup, zero runtime logs appear — no request logs, no session logs, no error logs
• Services ARE processing requests successfully (verified via API responses and trace data)
• The logging library (log4js) writes structured JSON to stdout, which worked before the outage
• We have tried: redeploying from the dashboard, redeploying via railway up CLI, deleting and recreating services entirely — none resolved the issue
Expected: Runtime stdout should be captured continuously, not just during startup.
Environment details:
• All services use Dockerfile builder
• Node.js 20 (Alpine)
• log4js with console appender writing structured JSON to stdout
• Services were recreated fresh after the outage (not restored from pre-outage state)
1 Replies
Status changed to Open Railway • 17 days ago
17 days ago
This has to be split into two cases: platform log ingestion versus application logging. Because it affects multiple services, survived service recreation, and appears in both the dashboard and railway logs, do not start by rewriting log4js. First prove whether plain stdout/stderr is still being captured after startup.
Add this temporary heartbeat to one affected service, deploy it, then watch both Log Explorer and railway logs for 2-3 minutes:
setInterval(() => {
console.log(JSON.stringify({
marker: "railway-log-probe",
stream: "stdout",
ts: new Date().toISOString(),
}));
console.error(JSON.stringify({
marker: "railway-log-probe",
stream: "stderr",
ts: new Date().toISOString(),
}));
}, 10000);Then split the result:
-
If the probe also only appears during startup, treat this as a runtime log ingestion issue, not a
log4jsconfiguration problem. Escalate it with:- project id
- environment id
- affected service ids
- deployment ids
- replica region
- the exact marker string above
- UTC timestamp range where the marker was emitted but absent
- confirmation that both Log Explorer and
railway logsmiss the same lines
-
If the probe appears continuously, stdout/stderr capture is working and the issue is inside the app logging path. Check whether
log4jsstill uses a console appender after startup, whether runtime config switches it to file-only appenders, and whether the DockerCMDkeeps the Node process in the foreground instead of redirecting output to a file or detached child process.
Also compare the Deployment Logs panel against Observability/Log Explorer. If the deploy/startup panel shows the first burst but Log Explorer never receives runtime markers, include that distinction. It narrows the issue to the runtime log pipeline rather than the container process.
The fastest minimal reproduction is a fresh Railway service in the same project/environment using this command:
node -e 'setInterval(() => { console.log("stdout railway-log-probe " + new Date().toISOString()); console.error("stderr railway-log-probe " + new Date().toISOString()); }, 10000)'If that minimal service also stops logging after the startup burst, it gives support a clean platform-side repro without your application code. If it logs continuously, compare the minimal service against the affected Dockerfiles and log4js runtime config.
Status changed to Awaiting User Response Railway • 17 days ago
9 days ago
This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!
Status changed to Solved Railway • 9 days ago