Getting IP banned (?) when load testing using Locust (hosted in railway)
jonas-admin
PROOP

13 days ago

I am trying to setup a 1-master, 5-slave Locust deployment.

It worked well until I hit 10k+ RPS. After that, all the requests sent by the Locust cluster seems to get a response of 403.

The testing setup is:

  • Go service in Project A

  • Locust cluster in Project B

  • They are bridged using cloudflared (a cloudflare tunnel running in projects A and B)

  • Locust cluster hits Go service using the domain exposed by the cloudflare tunnel

Is there something that can be done to accomplish this? Or am I stuck with managed load testing tools?

Personally, I would've preferred to use Artillery but its distributed option only supports AWS and Azure.

I decided to give Locust a shot since it dons a master-slave setup. I thought could be executed well in Railway, until the rate limiting hit.

Solved

0 Replies

brody
EMPLOYEE

13 days ago

We don't return a 403 under any circumstances.


brody
EMPLOYEE

13 days ago

If you were being rate limited, we would simply drop the packet.


jonas-admin
PROOP

13 days ago

Interesting. It might've been on Cloudflare's side then.


brody
EMPLOYEE

13 days ago

Yeah, undoubtedly.


jonas-admin
PROOP

13 days ago

In any case, a large scale Locust setup wouldn't work in Railway because of the rate limiting. I also noticed that setting a static IP address in a project seems to set the same address for all services under it so this will be rate limited, correct? Unless the services have a dynamic IP, which is unlikely. Or is there something for this niche already in the works for the platform?


brody
EMPLOYEE

13 days ago

We wouldn't rate limit our own IPs.



brody
EMPLOYEE

13 days ago

To test the raw RPS of your app, don't do any tunneling or anything with Cloudflare; do the testing directly over the private network.


jonas-admin
PROOP

13 days ago

Alright, thanks!


Status changed to Solved brody 12 days ago


Loading...