App is down

2 years ago

please help

213 Replies

2 years ago

please provide more information


2 years ago

i upgraded


2 years ago

instance to pro


2 years ago

requests failing


2 years ago

due to high load


2 years ago

how many connections is "many connections"


2 years ago

100,000


2 years ago

or so


2 years ago

not clear


2 years ago

concurrently?


2 years ago

yes


2 years ago

what can i do


2 years ago

1252279905505251300


2 years ago

i see this in memory


2 years ago

1252279938833453000


2 years ago

can we jump on call @Brody


2 years ago

its urgent to fix


2 years ago

thank you


2 years ago

i can upgrade to any plan, already on pro


2 years ago

i don't work for Railway


2 years ago

oh


2 years ago

is there anyone on railway that c help


2 years ago

but I can tell you with absolute certainty that you can not do 100k concurrent requests


2 years ago

ok


2 years ago

railway is down


2 years ago

nevertheless


2 years ago

railway isn't down, they just aren't allowing you to do 100k concurrent requests


2 years ago

what can i do for them to allow


2 years ago

quick question, are you on the new proxy


2 years ago

dont know


2 years ago

please check


2 years ago

i didnt setup anything


2 years ago

how


2 years ago

in the service settings


2 years ago

where


2 years ago

1252280848703688700


2 years ago

that's your project settings, look in the service settings


2 years ago

1252280969835184000


2 years ago

ok enabled


2 years ago

is this 100k of legitimate traffic?


2 years ago

yes


2 years ago

its real users


2 years ago

100k is estimate


2 years ago

did something go viral?


2 years ago

just link to app was shared in group with around 80k users


2 years ago

so yes


2 years ago

i thought railway will handle it


2 years ago

in the mean time you can try spinning up more replicas


2 years ago

how


2 years ago

service settings


2 years ago

1252282076745695200


2 years ago

how to upgrade my instance


2 years ago

im confused


2 years ago

i press this


2 years ago

1252282105615090000


2 years ago

1252282145313915000


2 years ago

it wants me to enter name


2 years ago

Nikita?


2 years ago

you are upgraded, you are just selected on the hobby workspace, you need to move your project to the pro workspace


2 years ago

how can i move


2 years ago

while keeping same


2 years ago

url for deploy


2 years ago

you would have been asked to move projects over upon upgrading, if you closed that you can move a project across workspaces from your project settings


2 years ago

but pro users do not get higher request rates


2 years ago

so this won't help?


2 years ago

not for your particular issue since you are not CPU memory bound


2 years ago

you need replicas for this


2 years ago

i have 4


2 years ago

is that enough?


2 years ago

when where they added


2 years ago

2 min ago


2 years ago

if I understood mig correctly each replica should be permitted to allow for 1k RPS with 5k burst


2 years ago

5k burst?


2 years ago

yes


2 years ago

i don't know what the window for that is though


2 years ago

let me know if more replicas make a difference


2 years ago

still failing requests


2 years ago

its unclear why


2 years ago

its on railway though 100%


2 years ago

as I am in close contact with db


2 years ago

and all writes go through there


2 years ago

how are requests failing, please be more specific


2 years ago

its infinite loading


2 years ago

so requests just stalling


2 years ago

for many users


2 years ago

project id please


2 years ago

how to get


2 years ago

1252285017372098600


2 years ago

from your url



2 years ago

I'm sorry but I didn't ask for an invite



2 years ago

@Mig connections hanging with increased traffic


2 years ago

pls resolve


2 years ago

thank you ❤️


2 years ago

I can add 20 replicas


2 years ago

not 4


2 years ago

i guess


2 years ago

but not sure what cost of many replicas is


2 years ago

and if thats the issue


2 years ago

connections just stalling


2 years ago

i need all requests to go through


2 years ago

there is a limit on the total amount of RPS from the host itself, that would be 50k with 500k burst


2 years ago

there's no base cost, you pay for the resources used by them


2 years ago

so what can i do


2 years ago

to make requests go through


2 years ago

im messing around with caching on client now


2 years ago

you can try adding more replicas, but ultimately we will want to hear back from mig


2 years ago

1252286384425664500


2 years ago

i don't get though why connections stall


2 years ago

graphs look fine


2 years ago

yet connections stall for many


2 years ago

connecting to railway


2 years ago

as I've previously mentioned, you are likely hitting your RPS limit


2 years ago

how to up rps


2 years ago

more replicas


2 years ago

ok 8 replicas


2 years ago

but you could also be hitting the RPS of the host, so we will need to hear back from mig for a definite answer here


2 years ago

requests are stalled again


2 years ago

😐


2 years ago

fk


2 years ago

i hoped railway would be ok


2 years ago

to host my stuff on


2 years ago

if it was personal project i would be fine bu this is client project


2 years ago

its not even free plan


2 years ago

its crazy that it stalls even on 8 replicas


2 years ago

now its ok again


2 years ago

i have asked another team member since mig is offline


2 years ago

i added client side caching


2 years ago

at least for users that loaded before they will see things


2 years ago

thanks


2 years ago

though its monday so they are likely in meetings


2 years ago

Hi!


2 years ago

It looks like the drophunt URL is returning quickly here and you were able to bump replicas


2 years ago

@nikivi lmk if you still need something here


2 years ago

yes pls


2 years ago

i expected railway to support many requests


2 years ago

and many users just had stalled requests


2 years ago

so client probably won't work with me again but oh well


2 years ago

my issue with not understanding railway i guess


2 years ago

but i want to make sure this does not happen again


2 years ago

I was not on free plan and was in close contact with the db I used so all writes to it went through, all issues were on railway


2 years ago

and even now some requests still stall


2 years ago

with 8 replicas


2 years ago

for some reason I thought railway would not just drop requests on non free plan


2 years ago

I guess I should start launches with maybe 20 replicas just in case but that seems like too much


2 years ago

I wish I could just scale the load and so that railway won't be dropping a single request


2 years ago

would be wonderful


2 years ago

I assume railway infra should allow for this


2 years ago

that was 14 separate messages, please try to keep such things contained in as little amount of messages as possible


2 years ago

yea sorry


2 years ago

how can I make railway project where I don't have to think about replicas
and where railway won't be dropping or stalling requests 100%


2 years ago

I am coming to railway from cloudflare workers where thats the default


2 years ago

from ux perspective I assumed that would already be the case, and that things like proxy would be on by default


2 years ago

what you switched on was the beta proxy, its not like your app didnt come with a proxy already


2 years ago

so issue is that railway just drops requests if too many requests land to the replica instance, it does not queue them or anything


2 years ago

I will always turn on new proxy (it seems to be better, not sure)
and ideally I don't want to provision replicas as I don't know load beforehand


2 years ago

yes that is how limits work


2 years ago

can I make railway work with these requirements


2 years ago

I did not set any limits


2 years ago

the platform has RPS limits


2 years ago

ok will try move to self hosted solution with no rps limits


2 years ago

i do like railway ui though


2 years ago

😐


2 years ago

If you're doing self hosted, you're gonna have to manage this


2 years ago

Plus a littany of other issues. Can you show me where the requests are being queued? You seem to have more than enough headroom


2 years ago

You're running at 700mb


2 years ago

somehow requests were still stalling though


2 years ago

i did not get observability into why


2 years ago

at least I did not see in railway how


2 years ago

it's also possible that your app just simply wasn't responding fast enough


2 years ago

Yup. You've got a lot of headroom so, from what I can tell, it's something at the application level

Maybe a downstream service, some errant code, etc


2 years ago

can i cut replicas to 2 at least


2 years ago

what is the approximate level of replicas i have to hold for how many concurrent requests


2 years ago

i wish i didnt have to think about this 😦


2 years ago

we both no longer think this is an issue with railway, after some thinking it's unlikely you are hitting any RPS limit considering 8 replicas did not solve this issue


2 years ago

so with 1 replica


2 years ago

if i have 10,000 concurrent requests


2 years ago

hypothetically


2 years ago

what will railway do


2 years ago

sorry for the confusion, but all signs are pointing to an issue with your application or something your application interacts with


2 years ago

trying this 🙏

1252331616634671000


2 years ago

lets see


2 years ago

now traffic is lower


2 years ago

  • i have caching


2 years ago

so should be safer


2 years ago

please try to limit the amount of messages you send back to back as I am bombarded with notifications


2 years ago

@nikivi is this open source



2 years ago

p.s. can I clear logs in dashboard

1252337047646241000


2 years ago

this is not the whole thing


2 years ago

also - what is ronin



2 years ago

my data store


2 years ago

its whole thing


2 years ago

yeah so that is your bottleneck


2 years ago

ronin are not bottleneck as they checked their writes coming through


2 years ago

i was on call with them


2 years ago

i would suggest instrumenting your app with something like Opentelemetry and see which call is slow


2 years ago

1252337532751057000


2 years ago

yea will do


2 years ago

i cant run this locally


2 years ago

yea its bit annoying, will resolve in a bit, ronin uses private packages


2 years ago

well - i can't really help much until then


2 years ago

yea understandable


2 years ago

thanks for offer though ❤️


2 years ago

what's your reason for using ronin btw?


2 years ago

it seems to just be a database?


2 years ago

yea they provide a nice sdk to query https://github.com/ronin-co/client


2 years ago

and dashboard


2 years ago

im about 99% sure your issues lie in either the SDK they provide or the service itself


2 years ago

i would suggest using a normal database along with something like drizzle-orm


2 years ago

railway supports postgres and mysql


Loading...