same

It’s resolved here. But lately, there have been too many errors and inconsistencies; unfortunately, I’ll have to make a change.

Same here.

same here

same

Agreed. I am afraid it's not worth the hassle given the secrets leak and this issue.

Even the status page was down for a few minutes and their captcha is broken 😄

Same here.

We're sorry for the 429s and heavy-traffic challenge page affecting your API despite its low request volume.

An attack prevention rule intended to target domain scanners and scrapers was too aggressive and ended up affecting legitimate traffic. We've disabled that rule, and the issue is now resolved.

Please let us know if api.listen.zmievsa.org or the Railway domain is still showing either response.

Should these types of issues be noted on the status page, even if it's just a post-mortem? When status checks for my applications started failing it was the first place I checked, so it doesn't feel right that I have to dig through the forums to discover that I wasn't just hallucinating an outage.

Same here, on our project (ams1 edge, trace ams1.aydy). Between 14:19 and 14:52 UTC, a large share of requests to our service got 429 from railway-hikari. Those responses had no CORS headers, so our web app only showed a generic “Network Error” to users, and they never appeared in the service’s HTTP logs, which made this hard to diagnose. Our traffic reaches Railway through a reverse proxy on Fly.io, so it comes from only a few source IPs: could you make sure that kind of setup isn’t flagged by these rules in the future? And is there a way to see edge 429s in our logs? +1 on posting this kind of incident on the status page, even after the fact.

same

I understand why that was frustrating when your application checks were failing and the status page was the first place you looked.

Our status page reports incidents with significant, widespread user impact. Smaller or isolated issues won't show up there.

We don't publish a status-page entry or post-mortem for every issue. I'm sorry you had to piece together what was happening from the forum reports.

Not having time to update the status page during the 20 minute period is understandable. Refusing to acknowledge the issue with even a "partial outage" is unacceptable. This seems like the exact scenario your status page uses that term for.

The incident has been acknowledged publicly in this thread, including its cause and resolution. We aren't adding a separate retrospective status-page entry for this incident.

On the reverse-proxy question, we can't guarantee that a particular setup will never be affected by an attack prevention rule. For the rejected requests you described, there are no entries in your service's HTTP logs. If it happens again, capture the UTC timestamp, response body and available headers at the caller or reverse proxy so we can investigate.

Do you have a metric that tracks what percentage of users were impacted by this, or is it just an assumption that it wasn't a widespread issue? What is the metric an issue must meet to be acknowledged on the status page? Or is it just vibes? Sorry if I sound confrontational, but this is important to whether or not I can consider this a viable platform, especially with recent incidents in mind.

Update: seeing the same status failures again.

The error again man, this is getting complicated. I’m running into the error once more!!!

Same here. All projects have the same issue.

same here.... here we go again