Production .env Variables replaced with those of Dev Staging??
deadliftsandprofits
HOBBYOP

a month ago

Hi Railway team/community,

I need help understanding how to investigate a production environment variable incident.

In my Railway project, I selected the production environment and confirmed that at least one production service has environment variables that appear to match my development values. This is not limited to DATABASE_URL; it appears to include unrelated configuration such as object storage values and other app credentials/flags.

Example, with no secrets shown:

  • Production service: @maaate/app
  • Production variable observed: R2_BUCKET=maaate-dev

I did not intentionally update production variables to these development values.

A local read-only audit has already ruled out the following on my side:

  • no Railway CLI installed locally
  • no local Railway credential files found
  • no Railway CLI commands in shell history
  • no repo scripts found that write Railway variables
  • no RAILWAY_TOKEN / RAILWAY_API_TOKEN references in repo scripts
  • GitHub Actions does not deploy to Railway or modify Railway variables
  • recent local commits were not pushed and did not modify Railway service configs

This does not look like a normal database cutover issue, because a DB cutover might explain DATABASE_URL, but should not affect unrelated values like R2/storage, payment, email, analytics, or feature flags.

What I need help with:

  1. Where can I view production environment variable change history?

  2. Can Railway show:

    • timestamp of variable changes
    • actor/user/API token identity
    • whether a bulk variable import occurred
    • whether variables came from UI, CLI, API, GraphQL, shared variables, duplication, rollback, or another source
  3. Is there a way to confirm whether any of these occurred:

    • raw .env paste/import
    • bulk variable update
    • environment duplication from dev/staging into production
    • shared variable group attachment/change
    • service variable rollback
    • API token action
    • platform-side restore/rollback
  4. Is there any way to roll back production variables to a previous timestamp/state?

  5. If audit logs are only available on certain plans, what is the correct path to get Railway staff to privately review this?

I am not posting secrets publicly. I can provide masked screenshots, deploy timestamps, and affected variable names if Railway staff can advise the correct private channel.

Also WTF is up with Railway price-gating bloody support requests?? Are us users on lower tier paid plans simply forced into crowd-sourcing answers to our platform issues?

Solved

3 Replies

Railway
BOT

a month ago

Audit logs are available on all plans and record variable changes with timestamps, actor identity (user, API token, or Railway system), and event details. You can access them at railway.com/workspace/audit-logs. On the Hobby plan, retention is 48 hours, so if the change happened more than 48 hours ago, the log entry will no longer be available. For variable rollback, a deployment rollback restores both the Docker image and the custom variables from that deployment, but there is no standalone "restore variables to a previous timestamp" feature.


Status changed to Awaiting User Response Railway 30 days ago


deadliftsandprofits
HOBBYOP

a month ago

On Vercel, the Activity Log is available on ALL PLANS (incl free) and includes events like environment variable add/edit/delete/read, shared environment variable updates, and CLI env reads.

I did not make these changes. I have no integrations in place that allow any changes to occur to my Railway outside of CD. It's also mighty suspicious that this occurred after your platform outage.

It gives me pause to deploy any productions apps if the production variables can simply be replaced with dev-staging environment values, without warning, and with no suitable logs - despite being on a paid plan.


Status changed to Awaiting Railway Response Railway 30 days ago


Railway
BOT

a month ago

The May 19 outage postmortem is published at this link and does not describe environment variable overwrites as a side effect. Your audit logs are the only place to check whether a variable change was recorded, including the actor and event type, but on the Hobby plan those entries expire after 48 hours. If you have a previous successful deployment within your plan's image retention window, rolling it back will restore both the image and the variables from that deployment.


Status changed to Awaiting User Response Railway 30 days ago


Railway
BOT

23 days ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway 23 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...