Error 503 error:0A000126:SSL routines::unexpected eof while reading
janezkk
PROOP

5 months ago

This is yet another time this issue has occurred 😭 ,

temporarily disrupting our production environment. Our users are once again encountering this problem in production.

We already provided feedback after the previous incident, but we did not receive any response.

not even an acknowledgment or apology.

https://discord.com/channels/713503345364697088/1471172506260996238

I understand that no one intends for this to happen.

However, this must stop impacting our production environment. Please take immediate action to prevent this from happening again.

Solved

14 Replies

5 months ago

Hello,

It seems like our changelog announcing DDoS protection has provoked someone, and since we are still establishing a baseline with Fastly, full DDoS mitigations weren't applied.

https://status.railway.com/cmlvw0n6u00129w5qnapyib77


janezkk
PROOP

5 months ago

Now our services have fully recovered. However, the damage has already been done. our users experienced the disruption.

This product is built on trust, and incidents like this directly affect that trust. We are concerned about the impact this has had on our users and the confidence they place in our service.

Thank you.


kylekz
PRO

5 months ago

could 429 too many requests errors also be a result of this?


viking-coding
HOBBY

5 months ago

I'm getting the same error; I'm assuming (and hoping) it's related as well


akramh
PRO

5 months ago

it is becoming a daily theme! same here


Hi there, it's a DDoS attack pattern. We were establishing a basline with Fastly hence why we didn't have full mitigations.


5 months ago

Is it possible to disable DDoS protection from Fastly per services?


tibianobr
PRO

5 months ago

For those who use cloudflare, removing the proxy from dns mitigate a little.

But yeah, still a problem at least the 429


Rather, it's the fact that it wasn't fully out, we have gotten it back. The attack is mitigated. Sorry for the impact again.


5 months ago

It is not. Curious why you ask?


5 months ago

I already have it enabled through my clouflare proxy


5 months ago

It might seem unnecessary, and maybe it is, but in practice, it's really not going to hurt anything to have it twice.


janezkk
PROOP

5 months ago

I recall reading a thread suggesting that certain Cloudflare DDoS rules should be turned off. I'm seeking clarification because I am seeing excessive access attempts to .php files. Previously, Cloudflare was blocking this and traffic was lower, but now I see HTTP logs again. The source IPs appear to be from fastly.com. Should I disable Cloudflare, or how should I adjust it?

Screenshot_2569-02-21_at_16.42.25.png

Screenshot_2569-02-21_at_16.42.46.png

Screenshot_2569-02-21_at_16.42.53.png


5 months ago

Fastly isn't going to block bot traffic, they are for DDoS so if this bot is not making excessive requests relative to an attack they will be let through.


Status changed to Solved brody 5 months ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...