fd1a1c68-5bf0-4d48-afaa-fa810942c096

By "machine" do you mean "service"?

yes, sorry

That's for public connections.

What kind of service is service A?

It's running a NodeJS backend managed by PM2 in cluster mode

I think it's using like 30 clusters

Aka 30 instances of the application?

Yup

What was your vCPU during load testing?

very low, it hit like 5 cpu cores max

it should use 1 cpu per cluster as the max

Then it seems like you are CPU starved.

we have set this as the resource limit tho

frontend seems to scale fine up to 25 cores with basically the same setup

frontend is running a basic nextjs app in a similar cluster mode as backend

Are you sure pm2 is distributing the incoming requests to all 30 instances?

yes, we have the same setup in our current production server and it's working perfectly

Then your backend should be able to vertically scale to about 30 vCPU, so something in code is a bottleneck.

Hopping in as well.
Our main problem is networking between the two services: ingress-proxy <-> backend.

During load testing ingress-proxy loses connection to backend.

Example:
/ (ingress-proxy) # curl backend.railway.internal:5000
curl: (7) Failed to connect to backend.railway.internal port 5000 after 280 ms: Could not connect to server

The backend instance is perfectly fine and running at that point when running commands in it:
/ (backend) # curl localhost:5000
{"message":"Organization not found","success":false}

Its as if the connection between the two boxes is cut off after a certain limit.

Could you give me a sense of what kind of load you are putting on the backend during this? RPS?

This is sent to ingress-proxy service, which forwards most of the stuff to the backend service

Should be around 2k rps during the peak of the load test

There's no set RPS limit via the internal network fwiw.

Hmm okay, we'll run some additional tests to gather more info

The endpoints that are being hit (not by curl), are they doing db operations?

And admittedly, "could not connect to server" isn't too informative, could you see about getting a far more verbose error?

Yeah, gonna run tons of tests. I think maybe the backend VPS has some bad network configs. Stuff like available sockets being exhausted etc

since the frontend service is working fine under the load it's 99% some shit on our end, we'll play around and provide an update soon

btw, any reason to not use Railway's replica system over pm2?

There is always the possibility it's on our end, but in all my years of doing support for Railway, I've never encountered someone running into any sort of RPS limits on the private network, so please let me know what you find.