Error 503 error:0A000126:SSL routines::unexpected eof while reading
janezkk
PROOP

16 days ago

This is yet another time this issue has occurred 😭 ,
temporarily disrupting our production environment. Our users are once again encountering this problem in production.

We already provided feedback after the previous incident, but we did not receive any response.
not even an acknowledgment or apology.
https://discord.com/channels/713503345364697088/1471172506260996238

I understand that no one intends for this to happen.
However, this must stop impacting our production environment. Please take immediate action to prevent this from happening again.

Solved

14 Replies

16 days ago

Hello,

It seems like our changelog announcing DDoS protection has provoked someone, and since we are still establishing a baseline with Fastly, full DDoS mitigations weren't applied.

https://status.railway.com/cmlvw0n6u00129w5qnapyib77


janezkk
PROOP

16 days ago

Now our services have fully recovered. However, the damage has already been done. our users experienced the disruption.

This product is built on trust, and incidents like this directly affect that trust. We are concerned about the impact this has had on our users and the confidence they place in our service.

Thank you.


kylekz
PRO

16 days ago

could 429 too many requests errors also be a result of this?


viking-coding
HOBBY

16 days ago

I'm getting the same error; I'm assuming (and hoping) it's related as well


akramh
PRO

16 days ago

it is becoming a daily theme! same here


Hi there, it's a DDoS attack pattern. We were establishing a basline with Fastly hence why we didn't have full mitigations.


16 days ago

Is it possible to disable DDoS protection from Fastly per services?


tibianobr
PRO

16 days ago

For those who use cloudflare, removing the proxy from dns mitigate a little.

But yeah, still a problem roll_eyes emoji at least the 429


Rather, it's the fact that it wasn't fully out, we have gotten it back. The attack is mitigated. Sorry for the impact again.


16 days ago

It is not. Curious why you ask?


16 days ago

I already have it enabled through my clouflare proxy


16 days ago

It might seem unnecessary, and maybe it is, but in practice, it's really not going to hurt anything to have it twice.


janezkk
PROOP

16 days ago

I recall reading a thread suggesting that certain Cloudflare DDoS rules should be turned off. I'm seeking clarification because I am seeing excessive access attempts to .php files. Previously, Cloudflare was blocking this and traffic was lower, but now I see HTTP logs again. The source IPs appear to be from fastly.com. Should I disable Cloudflare, or how should I adjust it?

Screenshot_2569-02-21_at_16.42.25.png
Screenshot_2569-02-21_at_16.42.46.png
Screenshot_2569-02-21_at_16.42.53.png


16 days ago

Fastly isn't going to block bot traffic, they are for DDoS so if this bot is not making excessive requests relative to an attack they will be let through.


Status changed to Solved brody • 16 days ago


Loading...