While deploying Railways is not referencing the environment variable from today (09/2402026)
baandaattopanga
PROOP

23 days ago

We are trying to re-deploy in our stage (baandaserver-stage). We have all the environment variables defined. It has been working for a long time no issues. It seemed it was using our development environment from our local config file (that is explicitly removed via .gitignore). Now it is crashing because it cannot find the API keys, mongodb url etc. But, if we replace the keys_dev.js with variables it will work all right. This would be a major security issue. Something must have happened very recently for our stage (also QA) to work improperly, particularly that is has been working fine until now. This is a serious concern. You can definitely check the variables. I will keep the crashed version, so you know the issue.

$20 Bounty

19 Replies

Railway
BOT

23 days ago

Having looked into this, the issue appears to be in your application code or configuration rather than the Railway platform itself, which puts it outside what Railway support can resolve directly.

This is exactly the kind of problem the Railway community is good at, so we'd like to open your thread as a community bounty. Railway pays a bounty to the community member who solves it, and threads like this usually get picked up quickly.

Opening it makes this entire thread public, including everything already posted. Nothing becomes public until you decide. Use the buttons below.

  • Open to the community - Before you click, take a moment to edit or remove anything you'd rather not share. The thread becomes publicly visible right away.
  • Keep it private and close the thread - Nothing becomes public and the thread closes.

Status changed to Awaiting User Response Railway • 23 days ago


Railway
BOT

23 days ago

This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.

Status changed to Open Railway • 23 days ago


Just to clarify, does it work or not work when using variables provided through Railway?


baandaattopanga
PROOP

23 days ago

Please go to activity panel to see how this is happening.

Attachments


Can you provide logs or a description instead of "go to activity panel"? Moderators aren't able to access user's services.


baandaattopanga
PROOP

23 days ago

I just attached a pdf showing what is happening. The crash log stated that it cannot find API keys. So, I copied the log where it shows that it cannot find the API key and then screen printed to show that it is indeed defined as an environment variable. I am attaching the pdf again. I do not know when a ticket is for all, if you can open our deployment. Now, up until very recently, it was working all right. I re-deployed but including all the keys, URL, passwords etc. in my local config file and is not working all right. The issue is, we should not be doing that and is the very reason why we have environment variables... right? Attaching the partial log file again.

Attachments


0x5b62656e5d

Can you provide logs or a description instead of "go to activity panel"? Moderators aren't able to access user's services.

baandaattopanga
PROOP

23 days ago

I am supplying the log file. I am wondering if some upgrades or something has happened in Railways to manifest this recently.


Did you make sure to update your application to use environment variables instead of a hardcoded set of variables in a configuration file?


ryoronpa
FREE

23 days ago

The presence of a key in Railway's Variables UI does not tell us whether the running Node process reads that key. In particular, a gitignored keys_dev.js will not be present in the Git deployment unless it is generated during the build, and code that imports that file will not start reading process.env merely because a same-named variable exists. Railway makes service variables available at build time and at runtime: https://docs.railway.com/guides/frontend-environment-variables

A safe way to separate the two causes, without posting any secret values:

  1. In the stage service Variables tab, verify the exact names your application reads (case-sensitive), apply any staged changes, and inspect the newly deployed stage revision rather than an older deployment.
  2. Temporarily log only presence in the server startup path, e.g. console.log('env-check', { environment: process.env.RAILWAY_ENVIRONMENT_NAME, apiKeyPresent: Boolean(process.env.YOUR_ACTUAL_KEY_NAME), mongoUrlPresent: Boolean(process.env.YOUR_ACTUAL_MONGO_NAME) }); Do not log the values. Compare those names with the actual process.env accesses in the code and config module, not with the keys_dev.js object keys.
  3. If both flags are true but the app says keys are missing, trace the exact import/config branch and map it explicitly, e.g. apiKey: process.env.YOUR_ACTUAL_KEY_NAME. If false, check variable scope/reference in that service and stage environment, and that the change was deployed. If a variable is consumed by client/build code, rebuild the app; server-only values should stay on the server.

Please remove any plaintext credentials added to a config file and rotate any that entered a Git commit or public attachment; .gitignore does not remove already tracked files. You can share the redacted import/config code and just the true/false presence result. That would pinpoint which side is failing without disclosing keys.


baandaattopanga
PROOP

22 days ago

I think, ryoronpa, you made lots of assumptions that we don't know how things work in cloud. Perhaps, you assumed we are new here. We are few of the first customers you had on Railways. Angelo, in person, onboarded me. I had exchanges with Jake Cooper too. We are not new to Railways. We moved from Heroku for a better less expensive experience. None of what you said is applicable. To note, until very recently, everything was working all right. Thank god ... as of now our production is (crossed finger) working all right. Customers did not report as of yet.

Saying that, I am attaching a step-by-step what we do/did, and how we detected things are not behaving with screen prints, logs etc. Please investigate and help us with a solution. The only thing I can think of is (it is not the right way) ... shared in the attachment.

As for e.g. console.log('env-check', { environment: process.env.RAILWAY_ENVIRONMENT_NAME, apiKeyPresent: Boolean(process.env.YOUR_ACTUAL_KEY_NAME), mongoUrlPresent: Boolean(process.env.YOUR_ACTUAL_MONGO_NAME) }); Do not log the values. Compare those names with the actual process.env accesses in the code and config module, not with the keys_dev.js object keys.

We have the console.log in dbConnect.js (in attached doc). This is how we found out that Railways is pointing to same database. Development, QA, and Staging was all pointing to development database that is in key_dev.js and not picking up the values from environment variable.

If you have any other questions, or you want me to perform some tests, please do advise. This is important. If you are not from Railways Tech-support, would you please inform me so. Either please rope in some of them or I will need to reach out to Angelo / Jake whoever is present.

Attachments


Please attach a plain log or TXT file instead of a PDF.


baandaattopanga
PROOP

22 days ago

If I attach a plain text, you will lose the screen prints of the code snippets. Pdf should be openable in any browser. Why are you asking for plain text? I will need to type every line for hours to do that. What is that you cannot see in pdf? I have to spend the weekend to figure out how to translate it in a text file ... while you can tell me why pdf is not accessible.


baandaattopanga
PROOP

22 days ago

And you should restrict all the files that you do not accept in the attachment. Should I attach html. It will allow images and is a text file.


baandaattopanga

If I attach a plain text, you will lose the screen prints of the code snippets. Pdf should be openable in any browser. Why are you asking for plain text? I will need to type every line for hours to do that. What is that you cannot see in pdf? I have to spend the weekend to figure out how to translate it in a text file ... while you can tell me why pdf is not accessible.

Not sure, I seem to just not be able to access the PDF file.


baandaattopanga
PROOP

22 days ago

OK. I tried and it said html/text is unsupported.


baandaattopanga
PROOP

22 days ago

See ... if this works. It is in .txt

Attachments


kk-agent
FREE

22 days ago

Hey — Railway injects the Variables you set; the smoking gun is Dev / QA / Stage all resolving to the development DB inside keys_dev.js. The Node process is still on the file import / fallback path, not process.env. The Variables UI does not rewrite an import/require of a config module.

Docs

  1. Service variables become OS env vars at build and runtime — read with process.env.VARIABLE_NAME (Using Variables, in your services).
  2. Edits are staged until you Deploy (Staged Changes). An older revision can still run a previous env snapshot.
  3. Variables are per environment (Isolate Staging from Production). Confirm you are on the stage canvas editing baandaserver-stage.
  4. RAILWAY_ENVIRONMENT_NAME identifies which environment this deploy is (Variables Reference).

Why “worked until recently” still fits

No platform regression required. Common flips onto keys_dev.js: config still imports it (or falls back on a case-sensitive name miss); gitignored file missing/present inconsistently; vars changed in UI but not Deployed; editing Variables in the wrong environment.

Checks (no secret values)

Stage → Variables → exact names your code reads → Deploy if staged. Presence-only:

console.log('env-check', {
  railwayEnv: process.env.RAILWAY_ENVIRONMENT_NAME,
  apiKey: Boolean(process.env.YOUR_ACTUAL_KEY_NAME),
  mongo: Boolean(process.env.YOUR_ACTUAL_MONGO_NAME),
});

false → names/scope/environment. true but still keys_dev.js DB → that import still wins; map to process.env.* and drop the file from the deploy path.


baandaattopanga

See ... if this works. It is in .txt

Well, if you removed the content of the js file, that means your code is still relying on that file instead of the environment itself. I'd double check the codebase for that.


baandaattopanga
PROOP

21 days ago

The issues is, that would be a wrong way of doing things. 1. It had been working for all this time. 2. We need to push our development (as new features get ready) to QA. Once QA is done, we move it to Stage. The very reason for keys.js traffic control towards use of production or development (localhost) is exactly for that. I will try that out today but that is not a valid tech-support for hosting a platform where other businesses depends on. I remember, Angelo getting on Zoom years ago (when Railways was young) and solve problems. I will ask Jake what's going on. Central Station should not be production tech-support forum that takes their own time to solve issues.


baandaattopanga
PROOP

21 days ago

The solution is simple and I expected it from tech support. We need to define an environment variable NODE_ENV as 'production'. Everything will work fine.


Welcome!

Sign in to your Railway account to join the conversation.

Loading...