data getting cached or something
andremaytorena
PROOP

2 months ago

im getitng logged in as a user right now, every login goes to that user, its very urgent if this is railway

103 Replies


andremaytorena
PROOP

2 months ago

anyone


2 months ago

Also seeing this


directsyndikat
PRO

2 months ago

issue is on railway, multiple people are having issues with the mongoDb and possibly more including me


2 months ago

We accidentally enabled CDN capabilities for domains. It should be fixed.

Also, please make sure you don't respond with Cache-Control public headers for logged-in/authenticated content! This will result in unwanted data being stored on the CDN. I have issued a full purge for all cached assets.


devenbhooshan
PRO

2 months ago

what is wrong with you folks


efstajas
PRO

2 months ago

can you confirm 100% that the inadverted caching is fixed now?


devenbhooshan
PRO

2 months ago

this is such a stupid issue


devenbhooshan
PRO

2 months ago

are you out of your mind


devenbhooshan
PRO

2 months ago

who asked you to enable caching for my endpoints


efstajas
PRO

2 months ago

we are still in the progress of downing our entire infra right now to avoid potentially leaking sensitive PII.


efstajas
PRO

2 months ago

can we stop?


devenbhooshan
PRO

2 months ago

Stupid


devenbhooshan
PRO

2 months ago

have you stopped it ?


riki95
PRO

2 months ago

I can confirm it's railway, we're having the same issues, we just browght down our infrastructure due to this problem.


luismingati
PRO

2 months ago

Also happened to me and all my users!

We restarted the server and solved it. But it should not happen


efstajas
PRO

2 months ago

We need to know urgently whether only GET or also POST responses were affected. The latter would mean people could've gotten other's access tokens.


luismingati

Also happened to me and all my users! We restarted the server and solved it. But it should not happen

riki95
PRO

2 months ago

we restarted several times and it did not make effect. This is absurd, it's a huge problem.


2 months ago

  • we do not cache text/html by default
  • we do not cache most assets unless “Cache-Control” headers are set explicitly to make the resource public
  • for certain mime types that we don’t expect to be dynamic (images/videos/etc), we apply a default 2 hour TTL to them unless a cache-control header is set

2 months ago

So, most resources are not cached. Only those with appropriate Cache-Control headers


2 months ago

Note that the cache was purged at the time of my original message, :36.


efstajas
PRO

2 months ago

what changed exactly when this current incident started?


efstajas
PRO

2 months ago

I


efstajas
PRO

2 months ago

I am currently about 90% sure that we saw some authenticated responses that did not have a Cache-Control header being served from cache, at least briefly


2 months ago

We added surrogate keys to our requests with Fastly, which apparently bypasses our default “disable cache on domain” rule.


riki95
PRO

2 months ago

I have a cached session seeing another user here. We restarted the service and now it seems working again but this mean I can still see this user's session.


andremaytorena
PROOP

2 months ago

yeah i cant lie this is outrageous, i can stand the outages but come on, this literally just leaked so much shit


efstajas
PRO

2 months ago

98% sure now that at least some responses without Cache-Control header were served from cache for us - is this a reasonable assumption?


2 months ago

What is the mime type/Content-Type of the asset?


efstajas
PRO

2 months ago

JSON


2 months ago

Can you show me the response headers of the content that was getting cached?


efstajas
PRO

2 months ago

this is the same endpoint, no longer getting cached now, but I saw it being cached earlier and have a lot of users reporting seeing other people's data returned by this particular endpoint

image.png

Attachments


pokepulse
PRO

2 months ago

We need a full incident report


pokepulse
PRO

2 months ago

I need to be able to share this with my customers..


efstajas
PRO

2 months ago

we handle cache-control headers on Cloudflare, our server does not set them ever. We only very specifically cache certain assets on cloudflare, and the last rule is explicitly "do not cache json"


efstajas
PRO

2 months ago

Also - We did not know that Railway would cache anything, and we don't see a way to disable it now. Before going back up we would like to 100% disable all caching from Railway. Is this possible?


2 months ago

Yes, if you have the CDN disabled, all caching capabilities will be disabled now.


2 months ago

Thank you for the headers, I’ll look into what happened to your responses


efstajas
PRO

2 months ago

it was never enabled

Screenshot_2026-03-30_at_14.02.33.png

Attachments


efstajas
PRO

2 months ago

but we still see railway cdn headers on responses right now


2 months ago

CDN headers yes, but they should all be MISS right?


efstajas
PRO

2 months ago

well i hope so


efstajas
PRO

2 months ago

but seems so, yeah


riki95
PRO

2 months ago

Will wait for a full incident report and know what happened because it had a serious impact on our business.


Anonymous
PRO

2 months ago

That was the final straw, WE ARE MOVING OUT ASAP


cs0041
PRO

2 months ago

This icon means we enable or disable the CDN cache ? i so confuse ux/ui


devenbhooshan
PRO

2 months ago

Why was JSON Data cached? This exposed one customer's data to others.


2 months ago

Hi, could I also see your response headers? JSON without cache control shouldn’t be cached, but I’m still verifying the scope of the impact here.


2 months ago

Please note, we are also writing a retroactive incident.


jclaveau
PRO

2 months ago

we had the same issue here


efstajas
PRO

2 months ago

100% - urgently need to know exactly what the conditions were for resources to be served from cache over what time frames. Right now I'm hopeful that nothing sensitive was leaked - but it depends on whether certain POST requests may have been cached. If so we may have to urgently report a data breach to authorities. What an absolute headache.


devenbhooshan
PRO

2 months ago

But I never enabled CDN, so why did that happen?


devenbhooshan
PRO

2 months ago

It is a miss right now but during the incident, it served content from the cache that resulted in data breach!

Screenshot_2026-03-30_at_5.50.00_PM.png

Attachments


thomas-ndlss
PRO

2 months ago

Yes, we had the same issue. Our maintenance API call was being cached and always return false for isMaintenance. But the value in the database was true.

It is a content-type: application/json

Sadly I only have that screenshot.

I 'll also give you what we have right now regarding the same request.

image.png

image.png


thomas-ndlss
PRO

2 months ago

CDN was never enabled.


efstajas
PRO

2 months ago

@Phineas I can also now with 100% certainty say the incident affected endpoints WITHOUT any cache-control header set.


efstajas
PRO

2 months ago

Still uncertain whether authenticated POST requests may have been cached. This is by far the most urgent piece of information.


2 months ago

I will verify everything shortly. I’m pushing the same erroneous code to a staging service and will test behavior against that.


jclaveau
PRO

2 months ago

The user experience is awful for us: students logged in on the wrong account, panicked and deleted the data of others. We must now investigate which account were impacted and how to restore it. Such a failure !


afsalmadathingal
PRO

2 months ago

This is something shouldn't have happened and never happened


pokepulse
PRO

2 months ago

A never-event just occured.


2 months ago

I’m really sorry, we will first determine impact and then make it right for you guys.


afsalmadathingal
PRO

2 months ago

Who will be responsible for the financial loss of the other user's session they used?


afsalmadathingal
PRO

2 months ago

I was panicking with the email from users


nikolandgraf
PRO

2 months ago

Honestly. There is no other way than migrating away from railway. Cant be worse than that.


niek-hdas
PRO

2 months ago

We really need compensation for this, major security breach. Unhappy customers. My whole day ruined.


haksonzvakson
HOBBY

2 months ago

guys is this 100% resolved now? can we re-enable our infra?


2 months ago

They resolved it 2 hours ago


2 months ago

Hello, I can verify that POSTs were not cached.


2 months ago

If you want to test your domains against the erroneous version, you can override your /etc/hosts file with:

151.101.2.51 yourdomain.com

Should be available for a few hours while we determine impact here.


efstajas
PRO

2 months ago

the relief i felt reading this is immense, thank you.


riki95
PRO

2 months ago

Happy that this seems fixed.

But we have received multiple complaints from our customers and this situation is raising serious concerns.

As a platform serving professionals in a highly regulated field, we explicitly guarantee our users that their data is safe and handled with the utmost care. You can imagine the impact when they logged in and found themselves viewing another user's data.

This is not just a technical inconvenience, it is a breach of trust that directly undermines the credibility we have built with our customers. We need a clear explanation of what happened, what data was exposed, how do you think to compensate us for this mistake and what steps are being taken to ensure this never occurs again.


haayhappen
PRO

2 months ago

Can we get logs for affected cached requests to determine our customer impact?


pedrograaff
PRO

2 months ago

Today we lost customers and revenue, and we are now at risk of being sued for leaking medical data. I know you are also extremely disappointed with what happened, but we need support from Railway at this moment. The least you can do is:

  • A public announcement about the incident that I can share with customers
  • Some form of compensation

niek-hdas

We really need compensation for this, major security breach. Unhappy customers. My whole day ruined.

yuvalreg
PRO

2 months ago

whole day? whole fucking year. i think i lost years of my life today. i need a vacation


Anonymous
PRO

2 months ago

Can you confirm this only affects requests/responses with Cache-Control public headers?

We're running a PNPM monorepo with a next.js web app with custom Node/Express server, and separate Node/Express server. No explicit caching set up. Are we likely to be affected by this from a security perspective?


Can you confirm this only affects requests/responses with Cache-Control public headers? We're running a PNPM monorepo with a next.js web app with custom Node/Express server, and separate Node/Express server. No explicit caching set up. Are we likely to be affected by this from a security perspective?

hikieadmin
PRO

2 months ago

I think there is problem with some machine and connections, i cannot connect my mongodb one of my machines.


vpro
FREE

2 months ago

i think this is a case for the lawyers. confidential data has been exposed to other users, we had to completely shut down our service temporarily. this has REAL legal and business implications.

also - why was there no MAJOR alert, maybe even via email since you apparently new who was affected? we were informed about the issue by our users who reported seing others' private data.


vpro

i think this is a case for the lawyers. confidential data has been exposed to other users, we had to completely shut down our service temporarily. this has REAL legal and business implications. also - why was there no MAJOR alert, maybe even via email since you apparently new who was affected? we were informed about the issue by our users who reported seing others' private data.

reezusp
PRO

2 months ago

Agreed. Confidential user data got exposed to other users on our platform. One user's private data (name, relationship details, personal content) was cached and served to completely unrelated people. Multiple support tickets, lost paying subscribers, and zero notification from Railway. We found out from our own users. A status page banner doesn't cut it for a data breach.


2 months ago

Please note that we will be emailing and following up with all affected users.


harshjk
HOBBY

2 months ago

Affected all users and leaked data, you are such a immature. Will switch back to AWS.


reezusp
PRO

2 months ago

@phin

Too late. We found out from our own users, not from Railway. Had to shut down our service, investigate for hours thinking it was our bug, and handle support tickets from paying subscribers who saw another user's private data.

This is a PII exposure incident that triggers breach notification obligations depending on jurisdiction.


atspro2
PRO

2 months ago

Tenants in app were seeing other tenants data. Personal information, contact details, financial info, this should NEVER happen, EVER!!!! It is very serious indeed. My customers are very upset and worried. I now need to report to ICO as Railway is data processor under GDPR regulations. I am shocked this could happen, it is something that should never happen, ever!


atspro2

Tenants in app were seeing other tenants data. Personal information, contact details, financial info, this should NEVER happen, EVER!!!! It is very serious indeed. My customers are very upset and worried. I now need to report to ICO as Railway is data processor under GDPR regulations. I am shocked this could happen, it is something that should never happen, ever!

harshjk
HOBBY

2 months ago

Same here! I am also up for that


2 months ago

what's the ETA for that, has it been already send out?


2 months ago

Yes, we're sending them in batches. Last batch is going out now, est completion in 10mins


nikolandgraf
PRO

2 months ago

Is that all now? A lousy e-mail as apology?

Honestly, in every incident report I read "less than 2%" or this time "less than 0.5%" of users were affected. How come it ALWAYS hits us? Bad luck I guess.

What about compensation for lawyer fees? Migration costs? Potential customer losses? Hope you spend some of your 100M funding on compensation at least.


ultimatedoge5
HOBBY

2 months ago

All this is just pathetic. You guys at least got an apology email? I had to drop everything I was doing just to try to wrap my head around this whole mess while my client sat on the phone, pissed, because their whole business was stuck and couldn't even sell anything in the middle of a shift change. And all I can show them is this little blog post you called a postmortem. There is no way this was just 0.05%. But sure, let's just ship another new feature with another 3rd-party service. What could go wrong? How can you even ship a CDN without any way of doing a cache purge?? Hell, the CDN is not even in the docs.


efstajas
PRO

2 months ago

Hey team - while the public information has been helpful already to fully understand the impact on our end, I requested additional information on a private help thread nine hours ago, and still have not received a response. Please respond soon – I need to get the logs specifically forwarded to our lawyers ASAP. Thank you.


ultimatedoge5

All this is just pathetic. You guys at least got an apology email? I had to drop everything I was doing just to try to wrap my head around this whole mess while my client sat on the phone, pissed, because their whole business was stuck and couldn't even sell anything in the middle of a shift change. And all I can show them is this little blog post you called a postmortem. There is no way this was just 0.05%. But sure, let's just ship another new feature with another 3rd-party service. What could go wrong? How can you even ship a CDN without any way of doing a cache purge?? Hell, the CDN is not even in the docs.

harshjk
HOBBY

2 months ago

Don't trust this number, percentage should be much higher. Some of people don't yet received any email such as me. Even though I am affected.


Anonymous
PRO

2 months ago

Our app became unuseable for hours during this ordeal last night - normal requests that should have taken a few seconds were taking 1-10+ minutes or hanging completely, affecting enterprise customers. Nothing on the status page describing the CDN situation indicated this could be related so I was debugging from my end. I posted a support thread where the AI support agent indicated this WAS caused by the CDN issue (still no human response) so I urgently moved ourselved to another provider and pulled an all nighter to get ourselves back online and to do a damage control audit, investigating for potential security and data breaches due to Railway's error. I have not received an email from Railway although they said they have emailed all affected users hours ago.


fr
PRO

2 months ago

https://blog.railway.com/p/incident-report-march-30-2026-authenticated-user-data-cached

"Impacted customers were notified before releasing this press release.

If you have no email, you are not impacted."

We were VERY impacted, yet no emails. Care to explain?

I'v e also opened a thread during the incident: https://station.railway.com/community/railway-caching-all-http-responses-inclu-30bbbbb4


fr

<https://blog.railway.com/p/incident-report-march-30-2026-authenticated-user-data-cached> _"Impacted customers were notified before releasing this press release._ _If you have no email, you are not impacted."_ We were **VERY** impacted, yet no emails. Care to explain? I'v e also opened a thread during the incident: <https://station.railway.com/community/railway-caching-all-http-responses-inclu-30bbbbb4>

riki95
PRO

2 months ago

We are seriously concerned. I work with @Fr, and not only did he write a thread during the incident, but both of us continuously commented on these blog posts while the incident was unfolding, explicitly reporting our issue.

What concerns us even more is that we were undeniably one of the impacted companies, yet you do not recognize us as such.

What kind of systems do you use to diagnose these events?

We experienced SERIOUS issues. When I was personally logged into my account, all requests were being cached and my data wasn't updating. We then discovered the problem was much more severe: other people were logging in WITH MY ACCOUNT. Since it is an ADMIN account, they could potentially see and do anything!

We had users logging in as other users, gaining access to extremely sensitive data. And now they have been complaining to us since yesterday morning.

The damage we have suffered is beyond unacceptable, and you haven't even included us among those affected by the incident? We are shocked.

We demand immediate action from your team,. We expect an email or direct contact from you regarding this matter as soon as possible. We are truly appalled.


andremaytorena
PROOP

2 months ago

I'm yet to receive an email from Railway too


daavoo
PRO

2 months ago

"Impacted customers were notified before releasing this press release.

If you have no email, you are not impacted."

We have been 100% impacted and still haven't received any communication.

To be honest at this point a "we are sorry email" makes no difference to me. The email that I would have wanted is the "we have messed up, you need to take a look immediately" instead of noticing the issue thanks to reports of our own users.


haksonzvakson
HOBBY

2 months ago

Excuse me, but where is my email? My users have had their sessions mixed up - how the hell do you even determine "who was affected"? By checking the thread here and emailing the loud ones?


andremaytorena
PROOP

2 months ago

I kind of need this info soon, need to file report with Amazon


pepijn
PRO

2 months ago

I have 25+ projects with each of domain at least 2 domains being used, stating some of my domains are affected is unsufficient to do any retroactive checks, can I get detailed info or should I presume all of them were affected?


Mind if you send us a direct thread in https://station.railway.com, we have a bridge with our CDN provider, we're doing another pass to get that data for you.


andremaytorena
PROOP

2 months ago

I’ve now opened a support only thread


jhud
PRO

a month ago

We were definitely affected and have evidence of it, and have not got any email or response so far from Railway. We all urgently need the logs from their CDN to do our own forensics ASAP. This is a massive problem.


andremaytorena
PROOP

a month ago

How long can we expect to receive a response, it's now been almost 2 days with nothing.


We got logs from the upstream now, working through the list now.


Welcome!

Sign in to your Railway account to join the conversation.

Loading...