16 days ago
Project fantasy-cricket-pros · env production · service Web.
Runtime V2 (RAILWAY_BETA_ENABLE_RUNTIME_V2=1) injects empty-string values for recently-added variables into the running process, while SSH / rails runner sessions in the same container get
the correct values:
- Puma has the key SENTRY_ORG but ENV["SENTRY_ORG"] is "". Same for SENTRY_PROJECT/AUTH_TOKEN/DSN, SIDEKIQ_WEB_USERNAME/PASSWORD, FIREBASE_*.
- SSH/rails runner sees the real values (SENTRY_ORG = 18 chars, etc.).
- Old variables (DATABASE_URL, REDIS_URL) have correct values — only recently-added ones are blank.
- Active deployment metadata = runtime: V2.
I cannot disable it: removing RAILWAY_BETA_ENABLE_RUNTIME_V2 returns "not found" (managed setting, not a variable), and a service-settings "switch to V1" + redeploy did not take — the deployment still boots runtime: V2. No runtime/beta toggle is visible in my dashboard.
Please disable Runtime V2 for this service/project (boot on V1), or fix the V2 variable-value resolution. It's breaking production error reporting and admin auth.
1 Replies
Status changed to Awaiting Railway Response Railway • 16 days ago
15 days ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • 15 days ago
6 days ago
@Railway - this one needs the Runtime V2 team; a community fix can't resolve it. Sharing a definitive diagnosis and a reproducible test below.
Confirmed: Runtime V2 is injecting recently-added variables into the running process as present-but-empty keys, and it survives a full rebuild-deploy.
Ruling out the env-precedence / shared-shadow theory (thanks @chriscybersec76, good hypothesis to check): all affected keys are plain literal service variables - no references, no shared vars, no blank duplicates (verified with railway variable list --service Web --json). If a blank service-level dupe were shadowing a shared value, SSH and rails runner would resolve blank too - but they resolve the correct values. So precedence isn't it.
Ground truth, read from inside the running process (not from an SSH shell):
- ENV.key?("SENTRY_ORG") is true, but ENV["SENTRY_ORG"].present? is false (empty string).
- Old variables (SENTRY_DSN, DATABASE_URL) have correct values; only the recently-added ones (SENTRY_ORG / SENTRY_PROJECT / SENTRY_AUTH_TOKEN) are empty.
- SSH and rails runner in the same container resolve the correct values (they read config fresh), which is exactly why this looks fine from a shell but breaks the actual web process.
Note for anyone reproducing: /proc//environ can't be used to verify this on Railway - it's permission-denied to the SSH session's root and reads empty as the process owner. We read the value through an authenticated app endpoint that checks .present?.
Controlled test - does a redeploy fix it? No.
- Before: deployment b681a02e (git commit ccc902f8) - the three keys present, values empty.
- Action: railway redeploy --service Web (same commit, no --from-source, so env re-injection is the only variable).
- After: fresh SUCCESS deployment 1924bc96 (same commit ccc902f8) - byte-identical: keys present, values still empty.
A clean rebuild re-injects the empty values, so "just redeploy" does not resolve this.
Ask (same as the original report): please disable Runtime V2 for this service/environment (V1 injects values correctly), or fix the V2 variable-value resolution so recently-added literals are injected with their values. We can't toggle it ourselves - RAILWAY_BETA_ENABLE_RUNTIME_V2 isn't a service variable in our project, and removing it returns "not found."
Environment details for lookup:
- Project: fantasy-cricket-pros (ID d965fdcb-2083-4eeb-a619-3fa50faaef70)
- Environment: production (ID ce29d55f-d635-4504-95a3-695503959e38)
- Service: Web (ID a0cbb2e8-2eec-4c87-b210-33b0598234bb)
- Current deployment: 1924bc96-ebde-4da0-9409-7690683d45b7 (running commit ccc902f8)
- Prior deployment referenced above: b681a02e-aa64-4ba6-9747-8af1a08f5600
Happy to run any diagnostic you'd like, or grant support access.