please provide more information

i upgraded

instance to pro

requests failing

due to high load

how many connections is "many connections"

100,000

or so

not clear

concurrently?

yes

what can i do

i see this in memory

can we jump on call @Brody

its urgent to fix

thank you

i can upgrade to any plan, already on pro

i don't work for Railway

oh

is there anyone on railway that c help

but I can tell you with absolute certainty that you can not do 100k concurrent requests

ok

railway is down

nevertheless

railway isn't down, they just aren't allowing you to do 100k concurrent requests

what can i do for them to allow

quick question, are you on the new proxy

dont know

please check

i didnt setup anything

how

in the service settings

where

that's your project settings, look in the service settings

ok enabled

is this 100k of legitimate traffic?

yes

its real users

100k is estimate

did something go viral?

just link to app was shared in group with around 80k users

so yes

i thought railway will handle it

in the mean time you can try spinning up more replicas

how

service settings

how to upgrade my instance

im confused

i press this

it wants me to enter name

Nikita?

you are upgraded, you are just selected on the hobby workspace, you need to move your project to the pro workspace

how can i move

while keeping same

url for deploy

you would have been asked to move projects over upon upgrading, if you closed that you can move a project across workspaces from your project settings

but pro users do not get higher request rates

so this won't help?

not for your particular issue since you are not CPU memory bound

you need replicas for this

i have 4

is that enough?

when where they added

2 min ago

if I understood mig correctly each replica should be permitted to allow for 1k RPS with 5k burst

5k burst?

yes

i don't know what the window for that is though

let me know if more replicas make a difference

still failing requests

its unclear why

its on railway though 100%

as I am in close contact with db

and all writes go through there

how are requests failing, please be more specific

its infinite loading

so requests just stalling

for many users

project id please

how to get

from your url

https://railway.app/invite/pPjJ1mRNZqX

I'm sorry but I didn't ask for an invite

https://railway.app/project/0bcb537c-51f6-4c2b-ae34-0db185dda977

@Mig connections hanging with increased traffic

pls resolve

thank you ❤️

I can add 20 replicas

not 4

i guess

but not sure what cost of many replicas is

and if thats the issue

connections just stalling

i need all requests to go through

there is a limit on the total amount of RPS from the host itself, that would be 50k with 500k burst

there's no base cost, you pay for the resources used by them

so what can i do

to make requests go through

im messing around with caching on client now

you can try adding more replicas, but ultimately we will want to hear back from mig

i don't get though why connections stall

graphs look fine

yet connections stall for many

connecting to railway

as I've previously mentioned, you are likely hitting your RPS limit

how to up rps

more replicas

ok 8 replicas

but you could also be hitting the RPS of the host, so we will need to hear back from mig for a definite answer here

requests are stalled again

😐

fk

i hoped railway would be ok

to host my stuff on

if it was personal project i would be fine bu this is client project

its not even free plan

its crazy that it stalls even on 8 replicas

now its ok again

i have asked another team member since mig is offline

i added client side caching

at least for users that loaded before they will see things

thanks

though its monday so they are likely in meetings

Hi!

It looks like the drophunt URL is returning quickly here and you were able to bump replicas

@nikivi lmk if you still need something here

yes pls

i expected railway to support many requests

and many users just had stalled requests

so client probably won't work with me again but oh well

my issue with not understanding railway i guess

but i want to make sure this does not happen again

I was not on free plan and was in close contact with the db I used so all writes to it went through, all issues were on railway

and even now some requests still stall

with 8 replicas

for some reason I thought railway would not just drop requests on non free plan

I guess I should start launches with maybe 20 replicas just in case but that seems like too much

I wish I could just scale the load and so that railway won't be dropping a single request

would be wonderful

I assume railway infra should allow for this

that was 14 separate messages, please try to keep such things contained in as little amount of messages as possible

yea sorry

how can I make railway project where I don't have to think about replicas
and where railway won't be dropping or stalling requests 100%

I am coming to railway from cloudflare workers where thats the default

from ux perspective I assumed that would already be the case, and that things like proxy would be on by default

what you switched on was the beta proxy, its not like your app didnt come with a proxy already

so issue is that railway just drops requests if too many requests land to the replica instance, it does not queue them or anything

I will always turn on new proxy (it seems to be better, not sure)
and ideally I don't want to provision replicas as I don't know load beforehand

yes that is how limits work

can I make railway work with these requirements

I did not set any limits

the platform has RPS limits

ok will try move to self hosted solution with no rps limits

i do like railway ui though

😐

If you're doing self hosted, you're gonna have to manage this

Plus a littany of other issues. Can you show me where the requests are being queued? You seem to have more than enough headroom

You're running at 700mb

somehow requests were still stalling though

i did not get observability into why

at least I did not see in railway how

it's also possible that your app just simply wasn't responding fast enough

Yup. You've got a lot of headroom so, from what I can tell, it's something at the application level

Maybe a downstream service, some errant code, etc

can i cut replicas to 2 at least

what is the approximate level of replicas i have to hold for how many concurrent requests

i wish i didnt have to think about this 😦

we both no longer think this is an issue with railway, after some thinking it's unlikely you are hitting any RPS limit considering 8 replicas did not solve this issue

so with 1 replica

if i have 10,000 concurrent requests

hypothetically

what will railway do

sorry for the confusion, but all signs are pointing to an issue with your application or something your application interacts with

trying this 🙏

lets see

now traffic is lower

• i have caching

so should be safer

please try to limit the amount of messages you send back to back as I am bombarded with notifications

@nikivi is this open source

yes https://github.com/nikitavoloboev/drophunt

p.s. can I clear logs in dashboard

this is not the whole thing

also - what is ronin

https://ronin.co

my data store

its whole thing

yeah so that is your bottleneck

ronin are not bottleneck as they checked their writes coming through

i was on call with them

i would suggest instrumenting your app with something like Opentelemetry and see which call is slow

yea will do

i cant run this locally

yea its bit annoying, will resolve in a bit, ronin uses private packages

well - i can't really help much until then

yea understandable

thanks for offer though ❤️

what's your reason for using ronin btw?

it seems to just be a database?

yea they provide a nice sdk to query https://github.com/ronin-co/client

and dashboard

im about 99% sure your issues lie in either the SDK they provide or the service itself

i would suggest using a normal database along with something like drizzle-orm

railway supports postgres and mysql