2 months ago
im getitng logged in as a user right now, every login goes to that user, its very urgent if this is railway
103 Replies
2 months ago
Also seeing this
2 months ago
issue is on railway, multiple people are having issues with the mongoDb and possibly more including me
2 months ago
We accidentally enabled CDN capabilities for domains. It should be fixed.
Also, please make sure you don't respond with Cache-Control public headers for logged-in/authenticated content! This will result in unwanted data being stored on the CDN. I have issued a full purge for all cached assets.
we are still in the progress of downing our entire infra right now to avoid potentially leaking sensitive PII.
2 months ago
I can confirm it's railway, we're having the same issues, we just browght down our infrastructure due to this problem.
2 months ago
Also happened to me and all my users!
We restarted the server and solved it. But it should not happen
We need to know urgently whether only GET or also POST responses were affected. The latter would mean people could've gotten other's access tokens.
luismingati
Also happened to me and all my users! We restarted the server and solved it. But it should not happen
2 months ago
we restarted several times and it did not make effect. This is absurd, it's a huge problem.
2 months ago
- we do not cache text/html by default
- we do not cache most assets unless “Cache-Control” headers are set explicitly to make the resource public
- for certain mime types that we don’t expect to be dynamic (images/videos/etc), we apply a default 2 hour TTL to them unless a cache-control header is set
2 months ago
So, most resources are not cached. Only those with appropriate Cache-Control headers
2 months ago
Note that the cache was purged at the time of my original message, :36.
I am currently about 90% sure that we saw some authenticated responses that did not have a Cache-Control header being served from cache, at least briefly
2 months ago
We added surrogate keys to our requests with Fastly, which apparently bypasses our default “disable cache on domain” rule.
2 months ago
I have a cached session seeing another user here. We restarted the service and now it seems working again but this mean I can still see this user's session.
yeah i cant lie this is outrageous, i can stand the outages but come on, this literally just leaked so much shit
98% sure now that at least some responses without Cache-Control header were served from cache for us - is this a reasonable assumption?
2 months ago
What is the mime type/Content-Type of the asset?
2 months ago
Can you show me the response headers of the content that was getting cached?
this is the same endpoint, no longer getting cached now, but I saw it being cached earlier and have a lot of users reporting seeing other people's data returned by this particular endpoint
Attachments
we handle cache-control headers on Cloudflare, our server does not set them ever. We only very specifically cache certain assets on cloudflare, and the last rule is explicitly "do not cache json"
Also - We did not know that Railway would cache anything, and we don't see a way to disable it now. Before going back up we would like to 100% disable all caching from Railway. Is this possible?
2 months ago
Yes, if you have the CDN disabled, all caching capabilities will be disabled now.
2 months ago
Thank you for the headers, I’ll look into what happened to your responses
it was never enabled
Attachments
2 months ago
CDN headers yes, but they should all be MISS right?
2 months ago
Will wait for a full incident report and know what happened because it had a serious impact on our business.
2 months ago
Hi, could I also see your response headers? JSON without cache control shouldn’t be cached, but I’m still verifying the scope of the impact here.
2 months ago
Please note, we are also writing a retroactive incident.
100% - urgently need to know exactly what the conditions were for resources to be served from cache over what time frames. Right now I'm hopeful that nothing sensitive was leaked - but it depends on whether certain POST requests may have been cached. If so we may have to urgently report a data breach to authorities. What an absolute headache.
It is a miss right now but during the incident, it served content from the cache that resulted in data breach!
Attachments
Yes, we had the same issue. Our maintenance API call was being cached and always return false for isMaintenance. But the value in the database was true.
It is a content-type: application/json
Sadly I only have that screenshot.
I 'll also give you what we have right now regarding the same request.
@Phineas I can also now with 100% certainty say the incident affected endpoints WITHOUT any cache-control header set.
Still uncertain whether authenticated POST requests may have been cached. This is by far the most urgent piece of information.
2 months ago
I will verify everything shortly. I’m pushing the same erroneous code to a staging service and will test behavior against that.
The user experience is awful for us: students logged in on the wrong account, panicked and deleted the data of others. We must now investigate which account were impacted and how to restore it. Such a failure !
2 months ago
I’m really sorry, we will first determine impact and then make it right for you guys.
Who will be responsible for the financial loss of the other user's session they used?
2 months ago
Honestly. There is no other way than migrating away from railway. Cant be worse than that.
2 months ago
We really need compensation for this, major security breach. Unhappy customers. My whole day ruined.
2 months ago
guys is this 100% resolved now? can we re-enable our infra?
2 months ago
They resolved it 2 hours ago
2 months ago
Hello, I can verify that POSTs were not cached.
2 months ago
If you want to test your domains against the erroneous version, you can override your /etc/hosts file with:
151.101.2.51 yourdomain.com
Should be available for a few hours while we determine impact here.
2 months ago
Happy that this seems fixed.
But we have received multiple complaints from our customers and this situation is raising serious concerns.
As a platform serving professionals in a highly regulated field, we explicitly guarantee our users that their data is safe and handled with the utmost care. You can imagine the impact when they logged in and found themselves viewing another user's data.
This is not just a technical inconvenience, it is a breach of trust that directly undermines the credibility we have built with our customers. We need a clear explanation of what happened, what data was exposed, how do you think to compensate us for this mistake and what steps are being taken to ensure this never occurs again.
2 months ago
Can we get logs for affected cached requests to determine our customer impact?
2 months ago
Today we lost customers and revenue, and we are now at risk of being sued for leaking medical data. I know you are also extremely disappointed with what happened, but we need support from Railway at this moment. The least you can do is:
- A public announcement about the incident that I can share with customers
- Some form of compensation
niek-hdas
We really need compensation for this, major security breach. Unhappy customers. My whole day ruined.
2 months ago
whole day? whole fucking year. i think i lost years of my life today. i need a vacation
2 months ago
Can you confirm this only affects requests/responses with Cache-Control public headers?
We're running a PNPM monorepo with a next.js web app with custom Node/Express server, and separate Node/Express server. No explicit caching set up. Are we likely to be affected by this from a security perspective?
Can you confirm this only affects requests/responses with Cache-Control public headers? We're running a PNPM monorepo with a next.js web app with custom Node/Express server, and separate Node/Express server. No explicit caching set up. Are we likely to be affected by this from a security perspective?
2 months ago
I think there is problem with some machine and connections, i cannot connect my mongodb one of my machines.
2 months ago
i think this is a case for the lawyers. confidential data has been exposed to other users, we had to completely shut down our service temporarily. this has REAL legal and business implications.
also - why was there no MAJOR alert, maybe even via email since you apparently new who was affected? we were informed about the issue by our users who reported seing others' private data.
vpro
i think this is a case for the lawyers. confidential data has been exposed to other users, we had to completely shut down our service temporarily. this has REAL legal and business implications. also - why was there no MAJOR alert, maybe even via email since you apparently new who was affected? we were informed about the issue by our users who reported seing others' private data.
2 months ago
Agreed. Confidential user data got exposed to other users on our platform. One user's private data (name, relationship details, personal content) was cached and served to completely unrelated people. Multiple support tickets, lost paying subscribers, and zero notification from Railway. We found out from our own users. A status page banner doesn't cut it for a data breach.
2 months ago
Please note that we will be emailing and following up with all affected users.
2 months ago
Affected all users and leaked data, you are such a immature. Will switch back to AWS.
2 months ago
@phin
Too late. We found out from our own users, not from Railway. Had to shut down our service, investigate for hours thinking it was our bug, and handle support tickets from paying subscribers who saw another user's private data.
This is a PII exposure incident that triggers breach notification obligations depending on jurisdiction.
2 months ago
Tenants in app were seeing other tenants data. Personal information, contact details, financial info, this should NEVER happen, EVER!!!! It is very serious indeed. My customers are very upset and worried. I now need to report to ICO as Railway is data processor under GDPR regulations. I am shocked this could happen, it is something that should never happen, ever!
atspro2
Tenants in app were seeing other tenants data. Personal information, contact details, financial info, this should NEVER happen, EVER!!!! It is very serious indeed. My customers are very upset and worried. I now need to report to ICO as Railway is data processor under GDPR regulations. I am shocked this could happen, it is something that should never happen, ever!
2 months ago
Same here! I am also up for that
2 months ago
what's the ETA for that, has it been already send out?
2 months ago
Yes, we're sending them in batches. Last batch is going out now, est completion in 10mins
2 months ago
Is that all now? A lousy e-mail as apology?
Honestly, in every incident report I read "less than 2%" or this time "less than 0.5%" of users were affected. How come it ALWAYS hits us? Bad luck I guess.
What about compensation for lawyer fees? Migration costs? Potential customer losses? Hope you spend some of your 100M funding on compensation at least.
2 months ago
All this is just pathetic. You guys at least got an apology email? I had to drop everything I was doing just to try to wrap my head around this whole mess while my client sat on the phone, pissed, because their whole business was stuck and couldn't even sell anything in the middle of a shift change. And all I can show them is this little blog post you called a postmortem. There is no way this was just 0.05%. But sure, let's just ship another new feature with another 3rd-party service. What could go wrong? How can you even ship a CDN without any way of doing a cache purge?? Hell, the CDN is not even in the docs.
Hey team - while the public information has been helpful already to fully understand the impact on our end, I requested additional information on a private help thread nine hours ago, and still have not received a response. Please respond soon – I need to get the logs specifically forwarded to our lawyers ASAP. Thank you.
ultimatedoge5
All this is just pathetic. You guys at least got an apology email? I had to drop everything I was doing just to try to wrap my head around this whole mess while my client sat on the phone, pissed, because their whole business was stuck and couldn't even sell anything in the middle of a shift change. And all I can show them is this little blog post you called a postmortem. There is no way this was just 0.05%. But sure, let's just ship another new feature with another 3rd-party service. What could go wrong? How can you even ship a CDN without any way of doing a cache purge?? Hell, the CDN is not even in the docs.
2 months ago
Don't trust this number, percentage should be much higher. Some of people don't yet received any email such as me. Even though I am affected.
2 months ago
Our app became unuseable for hours during this ordeal last night - normal requests that should have taken a few seconds were taking 1-10+ minutes or hanging completely, affecting enterprise customers. Nothing on the status page describing the CDN situation indicated this could be related so I was debugging from my end. I posted a support thread where the AI support agent indicated this WAS caused by the CDN issue (still no human response) so I urgently moved ourselved to another provider and pulled an all nighter to get ourselves back online and to do a damage control audit, investigating for potential security and data breaches due to Railway's error. I have not received an email from Railway although they said they have emailed all affected users hours ago.
2 months ago
https://blog.railway.com/p/incident-report-march-30-2026-authenticated-user-data-cached
"Impacted customers were notified before releasing this press release.
If you have no email, you are not impacted."
We were VERY impacted, yet no emails. Care to explain?
I'v e also opened a thread during the incident: https://station.railway.com/community/railway-caching-all-http-responses-inclu-30bbbbb4
fr
<https://blog.railway.com/p/incident-report-march-30-2026-authenticated-user-data-cached> _"Impacted customers were notified before releasing this press release._ _If you have no email, you are not impacted."_ We were **VERY** impacted, yet no emails. Care to explain? I'v e also opened a thread during the incident: <https://station.railway.com/community/railway-caching-all-http-responses-inclu-30bbbbb4>
2 months ago
We are seriously concerned. I work with @Fr, and not only did he write a thread during the incident, but both of us continuously commented on these blog posts while the incident was unfolding, explicitly reporting our issue.
What concerns us even more is that we were undeniably one of the impacted companies, yet you do not recognize us as such.
What kind of systems do you use to diagnose these events?
We experienced SERIOUS issues. When I was personally logged into my account, all requests were being cached and my data wasn't updating. We then discovered the problem was much more severe: other people were logging in WITH MY ACCOUNT. Since it is an ADMIN account, they could potentially see and do anything!
We had users logging in as other users, gaining access to extremely sensitive data. And now they have been complaining to us since yesterday morning.
The damage we have suffered is beyond unacceptable, and you haven't even included us among those affected by the incident? We are shocked.
We demand immediate action from your team,. We expect an email or direct contact from you regarding this matter as soon as possible. We are truly appalled.
2 months ago
"Impacted customers were notified before releasing this press release.
If you have no email, you are not impacted."
We have been 100% impacted and still haven't received any communication.
To be honest at this point a "we are sorry email" makes no difference to me. The email that I would have wanted is the "we have messed up, you need to take a look immediately" instead of noticing the issue thanks to reports of our own users.
2 months ago
Excuse me, but where is my email? My users have had their sessions mixed up - how the hell do you even determine "who was affected"? By checking the thread here and emailing the loud ones?
I have 25+ projects with each of domain at least 2 domains being used, stating some of my domains are affected is unsufficient to do any retroactive checks, can I get detailed info or should I presume all of them were affected?
2 months ago
Mind if you send us a direct thread in https://station.railway.com, we have a bridge with our CDN provider, we're doing another pass to get that data for you.
a month ago
We were definitely affected and have evidence of it, and have not got any email or response so far from Railway. We all urgently need the logs from their CDN to do our own forensics ASAP. This is a massive problem.
How long can we expect to receive a response, it's now been almost 2 days with nothing.
a month ago
We got logs from the upstream now, working through the list now.




