17 hours ago
Workspace: 1000-bananas's Projects (project 1000-bananas-platform, also "1000 Bananas - Muted").
What happened on our side:
-
Oct 21: "builds paused in all regions" blocked a production deploy for hours. We had no way to deploy a fix because Railway was our only builder. We moved image builds to GitHub Actions the same day.
-
Oct 22: your security email said service variables may have been exposed. We treated every variable in every project as leaked (roughly 40 secrets across Amazon SP-API, Amazon Ads, OpenAI, Anthropic, Keepa, Walmart, Jira, Figma, SMTP, GitHub, Postgres, Django). Within hours an AWS access key that lived only in Railway variables was used from an IP we do not own to make ~209,000 Bedrock calls, about 13,000 USD in one day. That key had no other copy anywhere, so the Railway exposure is the only plausible source.
-
The fallout took our whole week: rotating every provider credential, moving the 33 GB Postgres and all services to AWS, and dealing with AWS suspending the account over the abuse. We have left Railway for production.
What would have changed the outcome:
- Tell customers exactly which variables were exposed, when, and whether any were read. "May have been exposed" with no scope forces a full rotation and gives no way to tell abuse from noise.
- An audit log of variable reads and exports (who, when, from where), visible to the workspace owner.
- Secret references (a vault or integration with AWS Secrets Manager / 1Password) so a Railway incident does not expose the raw value.
- A way to deploy a prebuilt image when the builder is down, clearly documented, with the outage on the status page from the first minutes.
- Faster, specific incident comms. We learned the scope from our own CloudTrail, not from Railway.
0 Threads mention this feature
0 Replies