Getting IP banned (?) when load testing using Locust (hosted in railway)
jonas-admin
PROOP

10 months ago

I am trying to setup a 1-master, 5-slave Locust deployment.

It worked well until I hit 10k+ RPS. After that, all the requests sent by the Locust cluster seems to get a response of 403.

The testing setup is:

  • Go service in Project A
  • Locust cluster in Project B
  • They are bridged using cloudflared (a cloudflare tunnel running in projects A and B)
  • Locust cluster hits Go service using the domain exposed by the cloudflare tunnel

Is there something that can be done to accomplish this? Or am I stuck with managed load testing tools?

Personally, I would've preferred to use Artillery but its distributed option only supports AWS and Azure.

I decided to give Locust a shot since it dons a master-slave setup. I thought could be executed well in Railway, until the rate limiting hit.

Solved

9 Replies

10 months ago

We don't return a 403 under any circumstances.


10 months ago

If you were being rate limited, we would simply drop the packet.


jonas-admin
PROOP

10 months ago

Interesting. It might've been on Cloudflare's side then.


10 months ago

Yeah, undoubtedly.


jonas-admin
PROOP

10 months ago

In any case, a large scale Locust setup wouldn't work in Railway because of the rate limiting. I also noticed that setting a static IP address in a project seems to set the same address for all services under it so this will be rate limited, correct? Unless the services have a dynamic IP, which is unlikely. Or is there something for this niche already in the works for the platform?


10 months ago

We wouldn't rate limit our own IPs.



10 months ago

To test the raw RPS of your app, don't do any tunneling or anything with Cloudflare; do the testing directly over the private network.


jonas-admin
PROOP

10 months ago

Alright, thanks!


Status changed to Solved brody • 10 months ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...