App is down
nikiv
HOBBYOP

a year ago

please help

213 Replies

brody
EMPLOYEE

a year ago

please provide more information


nikiv
HOBBYOP

a year ago

i upgraded


nikiv
HOBBYOP

a year ago

instance to pro


nikiv
HOBBYOP

a year ago

requests failing


nikiv
HOBBYOP

a year ago

due to high load


brody
EMPLOYEE

a year ago

how many connections is "many connections"


nikiv
HOBBYOP

a year ago

100,000


nikiv
HOBBYOP

a year ago

or so


nikiv
HOBBYOP

a year ago

not clear


brody
EMPLOYEE

a year ago

concurrently?


nikiv
HOBBYOP

a year ago

yes


nikiv
HOBBYOP

a year ago

what can i do


nikiv
HOBBYOP

a year ago

1252279905505251300


nikiv
HOBBYOP

a year ago

i see this in memory


nikiv
HOBBYOP

a year ago

1252279938833453000


nikiv
HOBBYOP

a year ago

can we jump on call @Brody


nikiv
HOBBYOP

a year ago

its urgent to fix


nikiv
HOBBYOP

a year ago

thank you


nikiv
HOBBYOP

a year ago

i can upgrade to any plan, already on pro


brody
EMPLOYEE

a year ago

i don't work for Railway


nikiv
HOBBYOP

a year ago

oh


nikiv
HOBBYOP

a year ago

is there anyone on railway that c help


brody
EMPLOYEE

a year ago

but I can tell you with absolute certainty that you can not do 100k concurrent requests


nikiv
HOBBYOP

a year ago

ok


nikiv
HOBBYOP

a year ago

railway is down


nikiv
HOBBYOP

a year ago

nevertheless


brody
EMPLOYEE

a year ago

railway isn't down, they just aren't allowing you to do 100k concurrent requests


nikiv
HOBBYOP

a year ago

what can i do for them to allow


brody
EMPLOYEE

a year ago

quick question, are you on the new proxy


nikiv
HOBBYOP

a year ago

dont know


brody
EMPLOYEE

a year ago

please check


nikiv
HOBBYOP

a year ago

i didnt setup anything


nikiv
HOBBYOP

a year ago

how


brody
EMPLOYEE

a year ago

in the service settings


nikiv
HOBBYOP

a year ago

where


nikiv
HOBBYOP

a year ago

1252280848703688700


brody
EMPLOYEE

a year ago

that's your project settings, look in the service settings


nikiv
HOBBYOP

a year ago

1252280969835184000


nikiv
HOBBYOP

a year ago

ok enabled


brody
EMPLOYEE

a year ago

is this 100k of legitimate traffic?


nikiv
HOBBYOP

a year ago

yes


nikiv
HOBBYOP

a year ago

its real users


nikiv
HOBBYOP

a year ago

100k is estimate


brody
EMPLOYEE

a year ago

did something go viral?


nikiv
HOBBYOP

a year ago

just link to app was shared in group with around 80k users


nikiv
HOBBYOP

a year ago

so yes


nikiv
HOBBYOP

a year ago

i thought railway will handle it


brody
EMPLOYEE

a year ago

in the mean time you can try spinning up more replicas


nikiv
HOBBYOP

a year ago

how


brody
EMPLOYEE

a year ago

service settings


nikiv
HOBBYOP

a year ago

1252282076745695200


nikiv
HOBBYOP

a year ago

how to upgrade my instance


nikiv
HOBBYOP

a year ago

im confused


nikiv
HOBBYOP

a year ago

i press this


nikiv
HOBBYOP

a year ago

1252282105615090000


nikiv
HOBBYOP

a year ago

1252282145313915000


nikiv
HOBBYOP

a year ago

it wants me to enter name


nikiv
HOBBYOP

a year ago

Nikita?


brody
EMPLOYEE

a year ago

you are upgraded, you are just selected on the hobby workspace, you need to move your project to the pro workspace


nikiv
HOBBYOP

a year ago

how can i move


nikiv
HOBBYOP

a year ago

while keeping same


nikiv
HOBBYOP

a year ago

url for deploy


brody
EMPLOYEE

a year ago

you would have been asked to move projects over upon upgrading, if you closed that you can move a project across workspaces from your project settings


brody
EMPLOYEE

a year ago

but pro users do not get higher request rates


nikiv
HOBBYOP

a year ago

so this won't help?


brody
EMPLOYEE

a year ago

not for your particular issue since you are not CPU memory bound


brody
EMPLOYEE

a year ago

you need replicas for this


nikiv
HOBBYOP

a year ago

i have 4


nikiv
HOBBYOP

a year ago

is that enough?


brody
EMPLOYEE

a year ago

when where they added


nikiv
HOBBYOP

a year ago

2 min ago


brody
EMPLOYEE

a year ago

if I understood mig correctly each replica should be permitted to allow for 1k RPS with 5k burst


nikiv
HOBBYOP

a year ago

5k burst?


brody
EMPLOYEE

a year ago

yes


brody
EMPLOYEE

a year ago

i don't know what the window for that is though


brody
EMPLOYEE

a year ago

let me know if more replicas make a difference


nikiv
HOBBYOP

a year ago

still failing requests


nikiv
HOBBYOP

a year ago

its unclear why


nikiv
HOBBYOP

a year ago

its on railway though 100%


nikiv
HOBBYOP

a year ago

as I am in close contact with db


nikiv
HOBBYOP

a year ago

and all writes go through there


brody
EMPLOYEE

a year ago

how are requests failing, please be more specific


nikiv
HOBBYOP

a year ago

its infinite loading


nikiv
HOBBYOP

a year ago

so requests just stalling


nikiv
HOBBYOP

a year ago

for many users


brody
EMPLOYEE

a year ago

project id please


nikiv
HOBBYOP

a year ago

how to get


nikiv
HOBBYOP

a year ago

1252285017372098600


brody
EMPLOYEE

a year ago

from your url



brody
EMPLOYEE

a year ago

I'm sorry but I didn't ask for an invite



brody
EMPLOYEE

a year ago

@Mig connections hanging with increased traffic


nikiv
HOBBYOP

a year ago

pls resolve


nikiv
HOBBYOP

a year ago

thank you ❤️


nikiv
HOBBYOP

a year ago

I can add 20 replicas


nikiv
HOBBYOP

a year ago

not 4


nikiv
HOBBYOP

a year ago

i guess


nikiv
HOBBYOP

a year ago

but not sure what cost of many replicas is


nikiv
HOBBYOP

a year ago

and if thats the issue


nikiv
HOBBYOP

a year ago

connections just stalling


nikiv
HOBBYOP

a year ago

i need all requests to go through


brody
EMPLOYEE

a year ago

there is a limit on the total amount of RPS from the host itself, that would be 50k with 500k burst


brody
EMPLOYEE

a year ago

there's no base cost, you pay for the resources used by them


nikiv
HOBBYOP

a year ago

so what can i do


nikiv
HOBBYOP

a year ago

to make requests go through


nikiv
HOBBYOP

a year ago

im messing around with caching on client now


brody
EMPLOYEE

a year ago

you can try adding more replicas, but ultimately we will want to hear back from mig


nikiv
HOBBYOP

a year ago

1252286384425664500


nikiv
HOBBYOP

a year ago

i don't get though why connections stall


nikiv
HOBBYOP

a year ago

graphs look fine


nikiv
HOBBYOP

a year ago

yet connections stall for many


nikiv
HOBBYOP

a year ago

connecting to railway


brody
EMPLOYEE

a year ago

as I've previously mentioned, you are likely hitting your RPS limit


nikiv
HOBBYOP

a year ago

how to up rps


brody
EMPLOYEE

a year ago

more replicas


nikiv
HOBBYOP

a year ago

ok 8 replicas


brody
EMPLOYEE

a year ago

but you could also be hitting the RPS of the host, so we will need to hear back from mig for a definite answer here


nikiv
HOBBYOP

a year ago

requests are stalled again


nikiv
HOBBYOP

a year ago

😐


nikiv
HOBBYOP

a year ago

fk


nikiv
HOBBYOP

a year ago

i hoped railway would be ok


nikiv
HOBBYOP

a year ago

to host my stuff on


nikiv
HOBBYOP

a year ago

if it was personal project i would be fine bu this is client project


nikiv
HOBBYOP

a year ago

its not even free plan


nikiv
HOBBYOP

a year ago

its crazy that it stalls even on 8 replicas


nikiv
HOBBYOP

a year ago

now its ok again


brody
EMPLOYEE

a year ago

i have asked another team member since mig is offline


nikiv
HOBBYOP

a year ago

i added client side caching


nikiv
HOBBYOP

a year ago

at least for users that loaded before they will see things


nikiv
HOBBYOP

a year ago

thanks


brody
EMPLOYEE

a year ago

though its monday so they are likely in meetings


jake
EMPLOYEE

a year ago

Hi!


jake
EMPLOYEE

a year ago

It looks like the drophunt URL is returning quickly here and you were able to bump replicas


jake
EMPLOYEE

a year ago

@nikivi lmk if you still need something here


nikiv
HOBBYOP

a year ago

yes pls


nikiv
HOBBYOP

a year ago

i expected railway to support many requests


nikiv
HOBBYOP

a year ago

and many users just had stalled requests


nikiv
HOBBYOP

a year ago

so client probably won't work with me again but oh well


nikiv
HOBBYOP

a year ago

my issue with not understanding railway i guess


nikiv
HOBBYOP

a year ago

but i want to make sure this does not happen again


nikiv
HOBBYOP

a year ago

I was not on free plan and was in close contact with the db I used so all writes to it went through, all issues were on railway


nikiv
HOBBYOP

a year ago

and even now some requests still stall


nikiv
HOBBYOP

a year ago

with 8 replicas


nikiv
HOBBYOP

a year ago

for some reason I thought railway would not just drop requests on non free plan


nikiv
HOBBYOP

a year ago

I guess I should start launches with maybe 20 replicas just in case but that seems like too much


nikiv
HOBBYOP

a year ago

I wish I could just scale the load and so that railway won't be dropping a single request


nikiv
HOBBYOP

a year ago

would be wonderful


nikiv
HOBBYOP

a year ago

I assume railway infra should allow for this


brody
EMPLOYEE

a year ago

that was 14 separate messages, please try to keep such things contained in as little amount of messages as possible


nikiv
HOBBYOP

a year ago

yea sorry


nikiv
HOBBYOP

a year ago

how can I make railway project where I don't have to think about replicas
and where railway won't be dropping or stalling requests 100%


nikiv
HOBBYOP

a year ago

I am coming to railway from cloudflare workers where thats the default


nikiv
HOBBYOP

a year ago

from ux perspective I assumed that would already be the case, and that things like proxy would be on by default


brody
EMPLOYEE

a year ago

what you switched on was the beta proxy, its not like your app didnt come with a proxy already


nikiv
HOBBYOP

a year ago

so issue is that railway just drops requests if too many requests land to the replica instance, it does not queue them or anything


nikiv
HOBBYOP

a year ago

I will always turn on new proxy (it seems to be better, not sure)
and ideally I don't want to provision replicas as I don't know load beforehand


brody
EMPLOYEE

a year ago

yes that is how limits work


nikiv
HOBBYOP

a year ago

can I make railway work with these requirements


nikiv
HOBBYOP

a year ago

I did not set any limits


brody
EMPLOYEE

a year ago

the platform has RPS limits


nikiv
HOBBYOP

a year ago

ok will try move to self hosted solution with no rps limits


nikiv
HOBBYOP

a year ago

i do like railway ui though


nikiv
HOBBYOP

a year ago

😐


jake
EMPLOYEE

a year ago

If you're doing self hosted, you're gonna have to manage this


jake
EMPLOYEE

a year ago

Plus a littany of other issues. Can you show me where the requests are being queued? You seem to have more than enough headroom


jake
EMPLOYEE

a year ago

You're running at 700mb


nikiv
HOBBYOP

a year ago

somehow requests were still stalling though


nikiv
HOBBYOP

a year ago

i did not get observability into why


nikiv
HOBBYOP

a year ago

at least I did not see in railway how


brody
EMPLOYEE

a year ago

it's also possible that your app just simply wasn't responding fast enough


jake
EMPLOYEE

a year ago

Yup. You've got a lot of headroom so, from what I can tell, it's something at the application level

Maybe a downstream service, some errant code, etc


nikiv
HOBBYOP

a year ago

can i cut replicas to 2 at least


nikiv
HOBBYOP

a year ago

what is the approximate level of replicas i have to hold for how many concurrent requests


nikiv
HOBBYOP

a year ago

i wish i didnt have to think about this 😦


brody
EMPLOYEE

a year ago

we both no longer think this is an issue with railway, after some thinking it's unlikely you are hitting any RPS limit considering 8 replicas did not solve this issue


nikiv
HOBBYOP

a year ago

so with 1 replica


nikiv
HOBBYOP

a year ago

if i have 10,000 concurrent requests


nikiv
HOBBYOP

a year ago

hypothetically


nikiv
HOBBYOP

a year ago

what will railway do


brody
EMPLOYEE

a year ago

sorry for the confusion, but all signs are pointing to an issue with your application or something your application interacts with


nikiv
HOBBYOP

a year ago

trying this 🙏

1252331616634671000


nikiv
HOBBYOP

a year ago

lets see


nikiv
HOBBYOP

a year ago

now traffic is lower


nikiv
HOBBYOP

a year ago

  • i have caching


nikiv
HOBBYOP

a year ago

so should be safer


brody
EMPLOYEE

a year ago

please try to limit the amount of messages you send back to back as I am bombarded with notifications


neb
TRIAL

a year ago

@nikivi is this open source



nikiv
HOBBYOP

a year ago

p.s. can I clear logs in dashboard

1252337047646241000


neb
TRIAL

a year ago

this is not the whole thing


neb
TRIAL

a year ago

also - what is ronin



nikiv
HOBBYOP

a year ago

my data store


nikiv
HOBBYOP

a year ago

its whole thing


neb
TRIAL

a year ago

yeah so that is your bottleneck


nikiv
HOBBYOP

a year ago

ronin are not bottleneck as they checked their writes coming through


nikiv
HOBBYOP

a year ago

i was on call with them


neb
TRIAL

a year ago

i would suggest instrumenting your app with something like Opentelemetry and see which call is slow


neb
TRIAL

a year ago

1252337532751057000


nikiv
HOBBYOP

a year ago

yea will do


neb
TRIAL

a year ago

i cant run this locally


nikiv
HOBBYOP

a year ago

yea its bit annoying, will resolve in a bit, ronin uses private packages


neb
TRIAL

a year ago

well - i can't really help much until then


nikiv
HOBBYOP

a year ago

yea understandable


nikiv
HOBBYOP

a year ago

thanks for offer though ❤️


neb
TRIAL

a year ago

what's your reason for using ronin btw?


neb
TRIAL

a year ago

it seems to just be a database?


nikiv
HOBBYOP

a year ago

yea they provide a nice sdk to query https://github.com/ronin-co/client


nikiv
HOBBYOP

a year ago

and dashboard


neb
TRIAL

a year ago

im about 99% sure your issues lie in either the SDK they provide or the service itself


neb
TRIAL

a year ago

i would suggest using a normal database along with something like drizzle-orm


neb
TRIAL

a year ago

railway supports postgres and mysql


Loading...