Getting IP banned (?) when load testing using Locust (hosted in railway)
jonas-admin
PROOP

3 months ago

I am trying to setup a 1-master, 5-slave Locust deployment.

It worked well until I hit 10k+ RPS. After that, all the requests sent by the Locust cluster seems to get a response of 403.

The testing setup is:

  • Go service in Project A

  • Locust cluster in Project B

  • They are bridged using cloudflared (a cloudflare tunnel running in projects A and B)

  • Locust cluster hits Go service using the domain exposed by the cloudflare tunnel

Is there something that can be done to accomplish this? Or am I stuck with managed load testing tools?

Personally, I would've preferred to use Artillery but its distributed option only supports AWS and Azure.

I decided to give Locust a shot since it dons a master-slave setup. I thought could be executed well in Railway, until the rate limiting hit.

Solved

9 Replies

3 months ago

We don't return a 403 under any circumstances.


3 months ago

If you were being rate limited, we would simply drop the packet.


jonas-admin
PROOP

3 months ago

Interesting. It might've been on Cloudflare's side then.


3 months ago

Yeah, undoubtedly.


jonas-admin
PROOP

3 months ago

In any case, a large scale Locust setup wouldn't work in Railway because of the rate limiting. I also noticed that setting a static IP address in a project seems to set the same address for all services under it so this will be rate limited, correct? Unless the services have a dynamic IP, which is unlikely. Or is there something for this niche already in the works for the platform?


3 months ago

We wouldn't rate limit our own IPs.



3 months ago

To test the raw RPS of your app, don't do any tunneling or anything with Cloudflare; do the testing directly over the private network.


jonas-admin
PROOP

3 months ago

Alright, thanks!


Status changed to Solved brody 3 months ago


Loading...