5 months ago
Project fantasy-cricket-pros · env production · service Web.
Runtime V2 (RAILWAY_BETA_ENABLE_RUNTIME_V2=1) injects empty-string values for recently-added variables into the running process, while SSH / rails runner sessions in the same container get
the correct values:
- Puma has the key SENTRY_ORG but ENV["SENTRY_ORG"] is "". Same for SENTRY_PROJECT/AUTH_TOKEN/DSN, SIDEKIQ_WEB_USERNAME/PASSWORD, FIREBASE_*.
- SSH/rails runner sees the real values (SENTRY_ORG = 18 chars, etc.).
- Old variables (DATABASE_URL, REDIS_URL) have correct values — only recently-added ones are blank.
- Active deployment metadata = runtime: V2.
I cannot disable it: removing RAILWAY_BETA_ENABLE_RUNTIME_V2 returns "not found" (managed setting, not a variable), and a service-settings "switch to V1" + redeploy did not take — the deployment still boots runtime: V2. No runtime/beta toggle is visible in my dashboard.
Please disable Runtime V2 for this service/project (boot on V1), or fix the V2 variable-value resolution. It's breaking production error reporting and admin auth.
2 Replies
Status changed to Awaiting Railway Response Railway • 5 months ago
5 months ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • 5 months ago
5 months ago
@Railway - this one needs the Runtime V2 team; a community fix can't resolve it. Sharing a definitive diagnosis and a reproducible test below.
Confirmed: Runtime V2 is injecting recently-added variables into the running process as present-but-empty keys, and it survives a full rebuild-deploy.
Ruling out the env-precedence / shared-shadow theory (thanks @chriscybersec76, good hypothesis to check): all affected keys are plain literal service variables - no references, no shared vars, no blank duplicates (verified with railway variable list --service Web --json). If a blank service-level dupe were shadowing a shared value, SSH and rails runner would resolve blank too - but they resolve the correct values. So precedence isn't it.
Ground truth, read from inside the running process (not from an SSH shell):
- ENV.key?("SENTRY_ORG") is true, but ENV["SENTRY_ORG"].present? is false (empty string).
- Old variables (SENTRY_DSN, DATABASE_URL) have correct values; only the recently-added ones (SENTRY_ORG / SENTRY_PROJECT / SENTRY_AUTH_TOKEN) are empty.
- SSH and rails runner in the same container resolve the correct values (they read config fresh), which is exactly why this looks fine from a shell but breaks the actual web process.
Note for anyone reproducing: /proc//environ can't be used to verify this on Railway - it's permission-denied to the SSH session's root and reads empty as the process owner. We read the value through an authenticated app endpoint that checks .present?.
Controlled test - does a redeploy fix it? No.
- Before: deployment b681a02e (git commit ccc902f8) - the three keys present, values empty.
- Action: railway redeploy --service Web (same commit, no --from-source, so env re-injection is the only variable).
- After: fresh SUCCESS deployment 1924bc96 (same commit ccc902f8) - byte-identical: keys present, values still empty.
A clean rebuild re-injects the empty values, so "just redeploy" does not resolve this.
Ask (same as the original report): please disable Runtime V2 for this service/environment (V1 injects values correctly), or fix the V2 variable-value resolution so recently-added literals are injected with their values. We can't toggle it ourselves - RAILWAY_BETA_ENABLE_RUNTIME_V2 isn't a service variable in our project, and removing it returns "not found."
Environment details for lookup:
- Project: fantasy-cricket-pros (ID d965fdcb-2083-4eeb-a619-3fa50faaef70)
- Environment: production (ID ce29d55f-d635-4504-95a3-695503959e38)
- Service: Web (ID a0cbb2e8-2eec-4c87-b210-33b0598234bb)
- Current deployment: 1924bc96-ebde-4da0-9409-7690683d45b7 (running commit ccc902f8)
- Prior deployment referenced above: b681a02e-aa64-4ba6-9747-8af1a08f5600
Happy to run any diagnostic you'd like, or grant support access.
a month ago
This is a different Railway production issue from the PostgreSQL storage problem. Here, the key symptom is that the same environment variables have different values depending on how the process starts.
Your strongest evidence is:
- Puma receives empty strings for recently added variables such as SENTRY_ORG, SENTRY_PROJECT, and SENTRY_AUTH_TOKEN.
- SSH or rails runner in the same container sees the expected values.
- Older variables, including DATABASE_URL and REDIS_URL, continue to work.
- The active deployment explicitly reports Runtime V2, and attempts to switch to V1 have not changed that.
This points toward a possible Runtime V2 deployment-time environment injection or process-launch issue. It does not yet prove which component is responsible.
What I recommend:
- Keep a record of the affected deployment ID and its runtime metadata.
- Compare the environment variables inside Puma with those in a fresh SSH session, without printing secret values.
- Ask Railway to investigate Runtime V2 variable resolution and confirm whether Runtime V1 can be restored for your service.
- Avoid posting actual Firebase credentials, Sentry tokens, passwords, or DSNs containing secrets in a public thread.
Important: Railway has already addressed the Runtime V2 theory
I found a Railway Central Station discussion that matches your report exactly: Runtime V2 injecting empty variable values into the deployment process.
There is an important clarification in related Railway discussions: a Railway employee stated that Runtime V2 had become the standard runtime and that switching back to the legacy runtime was no longer an available solution. Another similar report was resolved by correcting a stuck staged-variable deployment state.
So I would not make disabling Runtime V2 your only requested fix. Instead, ask Railway to investigate why the running Puma process receives empty values while a newly launched Rails process sees the correct ones.
Next diagnostic steps
- Open the service's Variables tab and check for pending staged changes.
- Confirm that the affected variables are saved under the correct production environment and Web service.
- If changes are pending, review and deploy them through Railway's staged-changes workflow.
- If no changes are pending, ask Railway to inspect the actual Puma process's startup environment and compare it with a fresh Rails process.
- If the problem persists after a confirmed fresh deployment, provide the deployment ID and request an engineering investigation.
My recommendation: Continue in the existing public thread rather than opening a duplicate. Your report already contains useful reproduction evidence. Focus the follow-up on the discrepancy between Puma and rails runner, and request a concrete fix for the production environment-variable injection issue.