Skip to content
Request for Clarification: Oct 21 Incident Scope
kazwolfe
OP

25 days ago

Hi,

In reviewing communications from Railway in response to the incident, it's not exactly clear whether any given project/workspace was affected or not. To quote from the relevant pages:

Railway’s system is built so that if one set of keys is compromised, they should not compromise the entire system. As a result, we can validate that at most 1.85% of all workspaces had some environment variables potentially compromised. This represents, numerically, about 100,000 workspaces.

(from the Preliminary Report)

Workspaces outside those higher-risk groups are not confirmed safe from compromise. If you have not already done so, please rotate sensitive credentials stored in Railway environment variables during the incident, revoke the old credentials, update your variables, and redeploy dependent services.

(from the incident statuspage)

While I absolutely do agree that rotating all credentials is best-practice in this case, it would be helpful to know if our workspace was part of the 1.85% that actually was affected by this incident. This affects both incident response management and risk assessment, especially when dealing with harder-to-rotate credentials such as cryptographic encryption keys.

Solved

6 Replies

Railway
BOT

25 days ago

We can't confirm from this thread whether your workspace was in the higher-risk group described in the preliminary report, so we can't give you a per-workspace exposure answer here.

Workspaces outside those higher-risk groups are not confirmed safe from compromise. For incident response and risk assessment, treat any sensitive value stored in your Railway environment variables during the incident as potentially exposed. That includes harder-to-rotate material like encryption keys. The remediation is the same for everyone: rotate the credential, revoke the old one, update the variable, and redeploy the services that depend on it.


Status changed to Awaiting User Response Railway • 25 days ago


Railway

We can't confirm from this thread whether your workspace was in the higher-risk group described in the preliminary report, so we can't give you a per-workspace exposure answer here. Workspaces outside those higher-risk groups are not confirmed safe from compromise. For incident response and risk assessment, treat any sensitive value stored in your Railway environment variables during the incident as potentially exposed. That includes harder-to-rotate material like encryption keys. The remediation is the same for everyone: rotate the credential, revoke the old one, update the variable, and redeploy the services that depend on it.

kazwolfe
OP

25 days ago

Case in point with the AI response - the preliminary report suggests that the majority of workspaces were "safe," but communications otherwise say that they may not have been anyways.

(Note: this thread was also submitted to Railway via support; this is a duplicate of that for public visibility purposes.)


Status changed to Awaiting Railway Response Railway • 25 days ago


Railway
BOT

25 days ago

We understand why the two statements read as contradictory. The 1.85% figure in the preliminary report is the upper bound of workspaces that the report could validate as higher-risk. The later incident notice says that falling outside that group does not confirm a workspace was safe. So the most recent guidance is that being outside the higher-risk group is not a guarantee.

We can't tell you from this thread whether your workspace was in that higher-risk group. For risk assessment, including hard-to-rotate material like encryption keys, the current guidance applies to every workspace: treat sensitive values stored in environment variables during the incident as potentially exposed.


Status changed to Awaiting User Response Railway • 25 days ago


Railway

We understand why the two statements read as contradictory. The 1.85% figure in the preliminary report is the upper bound of workspaces that the report could validate as higher-risk. The later incident notice says that falling outside that group does not confirm a workspace was safe. So the most recent guidance is that being outside the higher-risk group is not a guarantee. We can't tell you from this thread whether your workspace was in that higher-risk group. For risk assessment, including hard-to-rotate material like encryption keys, the current guidance applies to every workspace: treat sensitive values stored in environment variables during the incident as potentially exposed.

kazwolfe
OP

22 days ago

As a result, we can validate that at most 1.85% of all workspaces had some environment variables potentially compromised.

The messaging here implies that only a subset of credentials were decrypted, seemingly for one or a handful of partitions. Therefore, it'd stand that any credentials not in this 1.85% of workspaces would not have been decrypted as the backing keys were never pulled from the key management service (assuming each workspace is in a single "bucket" that uses a specific key).

I've rotated the credentials I can, but this point remains frustratingly unclear: is the higher-risk group a subset of the 1.85% of "potentially compromised" workspaces, is it the entire 1.85% (in which case, why is this number "at most" if users outside this band could still have been compromised)? Please flag this thread as needing a reply from a support human if possible.


Status changed to Awaiting Railway Response Railway • 22 days ago


dizzydes90
EMPLOYEE

22 days ago

We're sorry this has been so unclear, and for the extra work it's put on you, especially with keys that are hard to rotate.

The 1.85% figure is the share of workspaces we could validate as having environment variables potentially compromised. It isn't a determination that every workspace outside that share was safe. That's why the guidance doesn't depend on which group a workspace falls into.

We can't give a workspace-specific answer on whether your variables were accessed or decrypted, and we can't confirm or rule out the per-key partitioning you describe. We know that doesn't settle the risk assessment, and we're sorry it can't be more precise.

Rotating what you could was the right step. For anything that's hard to rotate, like encryption keys, the safest assumption is that any value stored before the incident was exposed. Your provider logs may also show misuse that provider recorded, though they can't establish the full extent of exposure. Replacement secrets are safe to store now, because the attacker's access to our internal systems ended at 03:32 UTC on October 22. The preliminary report has the full timeline.


Status changed to Awaiting User Response Railway • 22 days ago


Status changed to Solved kazwolfe • 22 days ago


kazwolfe
OP

22 days ago

Noted, thank you for the response. I don't think there's going to be a clearer answer here so I'll just close this out for now.

I will note that there still is quite a bit of frustration to be told "at most" 1.85% of workspaces were (potentially) compromised, only for further communications to turn around and say "well actually, that's not the case." Even within the scope of the same message, it seems odd that a workspace that doesn't fall into that "potentially compromised" bucket doesn't count as "safe" but I guess that's all the information we can get right now.


Status changed to Awaiting Railway Response Railway • 22 days ago


Status changed to Solved kazwolfe • 22 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...