an hour ago
We got a bunch of 429s this morning and the site was unreachable
0 Threads mention this feature
3 Replies
an hour ago
We're sorry for the 429 errors that made your site unreachable this morning.
An attack prevention rule intended to target domain scanners and scrapers was too aggressive and ended up affecting legitimate traffic. We've disabled that rule, and the issue is now resolved.
Please let us know if you're still seeing 429s when accessing the site.
6 minutes ago
Hi Brody, we're still seeing this, and it came back after your reply.
Service domain: production.docs.page (also docspage-production.up.railway.app)
Project: 96a19c00-f285-4caa-9a38-82e51d418d9d
Service: 5d5162c8-7168-44b6-8dae-86aea49fc2d7
Environment: production
First window: 14:16–14:55 UTC
Started again: about 15:47 UTC, and still happening
Our app makes server-side requests to its own public domain, going through Cloudflare, at about 2 requests a second. Your edge is answering those with 429, text/plain, and the body rate limited. The requests never reach the app (there's no x-railway-request-id), so it looks like the same attack-prevention rule, or a similar one, is still on. Normal browser requests to the same URLs succeed.
Under Attack Mode is off for this service. Can you check the edge rules again and allowlist this domain? Our whole site is down while this is happening. Thanks!
a minute ago
We are experiencing the same on our side, intermittent 429s this morning across multiple services under the same conditions.
Server-side requests made through the services public domain, appears to be a new issue. We are going to route those through the services internal domain when possible, but not an issue we would expect to see.
So perhaps the 429s are related to all our services (originating from the same IPs) hitting the edge network. Given it's internal communication over a public domain?