Slow Performance
mylodiscord
FREEOP

6 months ago

my server was running fine yesterday, today it was a shit show. Had over 200+ people using our bot and it was shambles, yesterday was fine im on pro plan, is there any other service other than Pro to have a much faster server?

Solved

230 Replies

6 months ago

All plans use the same server spec, could you please share more details about your use case as well as usage details?


6 months ago

I’m assuming you mean a discord bot


mylodiscord
FREEOP

6 months ago

discord bot + backend/ws

1415119951106408700


6 months ago

Just the single service? The bot and backend are in the same service?


mylodiscord
FREEOP

6 months ago

yeah


mylodiscord
FREEOP

6 months ago

my structure

1415120129787957200


mylodiscord
FREEOP

6 months ago

src is discord bot, socket-server.js handles all socket related stuff.


6 months ago

There’s your issue. When your bot receives a ton of requests at once, your websocket server gets tied up servicing them all. If you architect your project such that your bot is in a separate service from your backend, you’ll be able to scale up your backend using replicas to handle more requests at once


6 months ago

This is not a Railway issue, but a project architecture issue. If you change the architecture of your bot you’ll be able to dramatically improve performance


mylodiscord
FREEOP

6 months ago

so what should I change?


6 months ago

modify your project so that your bot and backend are in different folders

Create a new service and deploy your backend to one, your bot to the other

Make sure that’s all working, then depending on the way your backend works, you may be able to use replicas. Regardless, they’re on different machines now, so they’re not sharing resources and will have more capacity


mylodiscord
FREEOP

6 months ago

so should i make another workspace for my bot, and deploy the bot on railway and have the socket server seperate? what if i need to use certain stuff from the socket-server.js file


6 months ago

No, not another workspace. Another service


6 months ago

What do you need to use from the server file?


mylodiscord
FREEOP

6 months ago

I use this from the socket-server

1415122564157800400


6 months ago

Sounds like you made two separate parts of your app pretty tightly coupled, that’s not great and will heavily restrict the optimizations you can do


mylodiscord
FREEOP

6 months ago

Wdym another service?


6 months ago

this box on your bot’s project is a service. It’s an individual container hosting your bot’s code. The goal here is to separate your bot’s container from your server’s container so they don’t share resources and you can scale up your server count


mylodiscord
FREEOP

6 months ago

how do I make another one


6 months ago

Should be a new service button at the top right of the project page, or right click anywhere in your project and select new service


mylodiscord
FREEOP

6 months ago

this?

1415124145175400400


6 months ago

yes


mylodiscord
FREEOP

6 months ago

what would I have to change in my code? will I have to re-code everything?


6 months ago

You will need to recode quite a bit, it seems like. But if you’re expecting to handle a lot of users, this is what you’ll need to do


mylodiscord
FREEOP

6 months ago

😭


mylodiscord
FREEOP

6 months ago

1.5k+ lines in socket-server.js


6 months ago

All that needs to be done is decouple your bot from your backend, that shouldn’t be too complex


mylodiscord
FREEOP

6 months ago

which one would i add?


6 months ago

Recoding your app is the first step here, you’ll add a new service for your server once you’ve done that


6 months ago

But if your code is on Github, then Github. However you uploaded your first service


mylodiscord
FREEOP

6 months ago

ok, what would i need to re-code? socket or discord bot


6 months ago

I don’t know enough about your project to tell you that


6 months ago

Any place where they’re directly referencing code files from each other, they shouldn’t


mylodiscord
FREEOP

6 months ago

i'll tell you more info soon, ppl are interacting with our bot, 100+ ppl


mylodiscord
FREEOP

6 months ago

would you be able to help me soon? @Adam


6 months ago

I’ve given you all the information I can, at this point all that’s left is for you to start coding


mylodiscord
FREEOP

6 months ago

So keep everything in one workspace?


mylodiscord
FREEOP

6 months ago

i'm a bit confused


6 months ago

Keep everything in one workspace, one project, but not one service


6 months ago

^


mylodiscord
FREEOP

6 months ago

so all i need to do is create 2 folders - 1 is discord bot and 2 is websocket?


6 months ago

Pretty much yes, but you need to make sure those two folders don’t have any crossover.


6 months ago

Once the two are decoupled, you host each one as an individual service on Railway


mylodiscord
FREEOP

6 months ago

ah… i see, we were using the bot in a small server (around 8,000-9,000 member server), and we just went back to our old discord (with 60,000+ members), and it's really slow in that cord, probably because of the member count.. just tested the bot in the 9k member cord and its really fast


6 months ago

Are you implementing discord's sharding feature? I think they suggest it around the ~10k mark


mylodiscord
FREEOP

6 months ago

i thought sharding is only needed if the bot is in 2,000+ servers or something?


6 months ago

Oh i might be mistaking then. Havent had a need to use it hahaha


6 months ago

let me go read up on the docs real quick


6 months ago

And I assume this is a discord.js bot right?


mylodiscord
FREEOP

6 months ago

yeah, discord.js bot + uses socket.io


mylodiscord
FREEOP

6 months ago

has a bot, and a backend


6 months ago

Do you have any leveling logic on messageCreate event?


6 months ago

Or something that would get triggered on a commonly occuring event. Curious if theres some hanging with cache, db calls, or operations.


mylodiscord
FREEOP

6 months ago

could you elaborate on "leveling logic"


6 months ago

Was driving at db related functionality on messageCreate events.
Some servers have an "xp" system where you gain certain xp points per message sent.


mylodiscord
FREEOP

6 months ago

ohh, we have the following :

messageReactionAdd
messageReactionRemove
messageCreate

1415418120406958000


mylodiscord
FREEOP

6 months ago

we have a channel if we need some people , we ask for people in there and they type and it'll return with a button to add them


6 months ago

Oh my…


6 months ago

There's your problem right there!


6 months ago

if i understand correctly, each and every one of those events is being fired every time a user sends a message.


mylodiscord
FREEOP

6 months ago

tho, Adam said it's because i have socket-server and a discord bot in the same project and I should make 2 separate services

1415418448548204500


6 months ago

<:Hmmmmm:891565000068587551> that doesnt sound like it would be a large factor although I could see the concern.


mylodiscord
FREEOP

6 months ago

We need those commands tho


6 months ago

Heres my suggestion for steps of debugging and verification.


6 months ago

Split this into 2 major services (easiest). Websocket (backend) discord bot (the bot).
Have them communicate over private network. In general thats better practice long term.
It'll also help verify thats not the issue.


6 months ago

I'm also curious, what database are you using?


mylodiscord
FREEOP

6 months ago

MongoDB


mylodiscord
FREEOP

6 months ago

tried this last night, there's so much that needs to be changed.. 😭


6 months ago

Yeahhh in terms of best project structure that's honestly your best bet.
It'll help you service things much better in the future


mylodiscord
FREEOP

6 months ago

thing is, i don't think i can do it all in one day, and we use this bot for our discord daily, at around 6pm est


6 months ago

Yeah i can understand that.
So for a bit of preface, were you using this bot in this 60k member server before?


6 months ago

Can I get a bit of info on the timeline?


mylodiscord
FREEOP

6 months ago

We provide "scrim-practice" for the Fortnite community to over 60,000 members, and we use that bot to host "scrim-practice", like practice games


mylodiscord
FREEOP

6 months ago

No, we were using it in a 3,000 member server & a 9,000 member server


6 months ago

Have you always been using Railway to host it?


mylodiscord
FREEOP

6 months ago

Yes


6 months ago

Okay yeah that clears things up


6 months ago

in your server, can you search for the total number of messages from a given day?


6 months ago

1415420339063623700


mylodiscord
FREEOP

6 months ago

i can try, one sec


6 months ago

curious what this number is


mylodiscord
FREEOP

6 months ago

😭

1415420439701749800


mylodiscord
FREEOP

6 months ago

our cord is really active


6 months ago

And by any chance, on the 9k one what are you seeing?


mylodiscord
FREEOP

6 months ago

1415420543515230500


6 months ago

Mhm


6 months ago

So


mylodiscord
FREEOP

6 months ago

we don't use the server anymore, its locked down


6 months ago

Well you can search on a past day


6 months ago

if you have a rough time of when it was most active


6 months ago

What you're running into is scaling issues.
60k is a very large number. From a 9k member server -> 60k member server you're looking at an increase of 6 times.
From 3k -> 60k thats ~20x larger.

If i understand that screenshot with folder structure correctly, it looks like you're running ~17 different events every single time somebody sends a message.
If each of those events trigger a DB call or are blocking you're having race conditions and a whole host of problems that come with databases and scaling


mylodiscord
FREEOP

6 months ago

so what should i do?


6 months ago

Typing out what I would do now <:picassoface:1398147502598918144>


mylodiscord
FREEOP

6 months ago


mylodiscord
FREEOP

6 months ago

here's a vid of what i mean:


mylodiscord
FREEOP

6 months ago

ah,.


mylodiscord
FREEOP

6 months ago

wait one sec


mylodiscord
FREEOP

6 months ago


mylodiscord
FREEOP

6 months ago

have a look


mylodiscord
FREEOP

6 months ago

see how long it takes


6 months ago

If i were in your situation here's what I would do. The server didnt grow from 0-60 overnight. I assume you guys were using something before so I highly suggest switching back to it and doing maintenance on this bot for a few days.
Split it into 2 services, for our conversations sake: BOT and WS.

Each of those services will run on Railway in a separate service. (Same project)
Now you have per service 32g of ram and 32vCPU available. You shouldnt ever run into a resource issue on them.

That's step 1.

Your bot should never actively query the database it should only make a request to WS which is the only service who talks to your database.
Each of your events is making a call to your websocket so look into pairing down.

I suggest figuring out which parts of the logic actively need to talk to your database and what doesnt. I'd implement a layer of caching with Redis for repetitive calls (username, user id, avatar, team, etc etc.)

Split the services up + cache data will fix a good bit of the latency


6 months ago

I'd also do some live debugging to figure out what part of that process is taking the longest. Is it the db call? Response from websocket? Once you know what's taking longest you can debug that


mylodiscord
FREEOP

6 months ago

unsure, it was instant in the other servers with low member count


6 months ago

there's 9k members joining instantly? Sheesh


mylodiscord
FREEOP

6 months ago

i can show you via dm


6 months ago

Sure my dms are open. As for solving this I prefer keeping them in threads so if other users have similar issues they can reference this


6 months ago

Yeah @mylo I think I'd look into doing what I suggested with figuring out which process is taking longest


6 months ago

Simple as adding console logs at certain points with durations until next points


mylodiscord
FREEOP

6 months ago

i'll do it later as we have to use the bot in 1 hour


6 months ago

Alright. I'm more then happy to help with suggestions/direction however I can 😁


mylodiscord
FREEOP

6 months ago

❤️


6 months ago


6 months ago

@mylo Railway supports running a mongo DB on project!

1415428678879547600


mylodiscord
FREEOP

6 months ago

how would this work?


6 months ago

I'm not personally familiar with mongo as much as I'd like, the process would be backing up the cloud mongo db you pay for and restoring to the railway hosted one.


6 months ago

That'll remove all network latency on each request and could help by a decent margin


mylodiscord
FREEOP

6 months ago

how would i connect it?

1415429302081687600


6 months ago

In your service its pointing to a connection url/uri so you would just swap the one its looking for out to the INTERNAL_URL variable found in the variables tab of that service


mylodiscord
FREEOP

6 months ago

ohh, do i just need to grab the mongo url and update it in the websocket-server


6 months ago

Yep! But I suggest the internal one


mylodiscord
FREEOP

6 months ago

whichj one ?

1415429728021647400


6 months ago

You'd be using this one

1415429871655456800


6 months ago

this is still sensitive i wouldnt share it


mylodiscord
FREEOP

6 months ago

can i keep the name like this and just grab the connection string and replace it?

1415430030355599600


6 months ago

wdym


6 months ago

Is that the env var on your websocket service?


mylodiscord
FREEOP

6 months ago

yeah


6 months ago

Oh perfect, type this


6 months ago

in the value side


6 months ago

${{MongoDB.INTERNAL_URL}}


6 months ago

It's a variable reference so it'll go look at the value of that env var on the mongo service and use that


mylodiscord
FREEOP

6 months ago

ok, one sec


mylodiscord
FREEOP

6 months ago

all my collections are gone, i cant use it


mylodiscord
FREEOP

6 months ago

this one has 2k i need all of it

1415431335618805800


6 months ago

Yeah its a blank mongodb. As I mentioned, you would need to save a copy/backup from the production mongo one and back it up onto that railway project


mylodiscord
FREEOP

6 months ago

Holy…


mylodiscord
FREEOP

6 months ago

Big difference


6 months ago

unsure the best way to do that


mylodiscord
FREEOP

6 months ago

It's instant LOL


6 months ago

Did you backup things?


mylodiscord
FREEOP

6 months ago

Not yet


mylodiscord
FREEOP

6 months ago

what should i export

1415431707112374300


mylodiscord
FREEOP

6 months ago

JSON/CSV


mylodiscord
FREEOP

6 months ago

JSON right?


6 months ago

I usually love JSON. Totally up to you though


6 months ago

Make sure that any changes you do are non destructive haha


mylodiscord
FREEOP

6 months ago

and how do I import it into railway?


mylodiscord
FREEOP

6 months ago

theres no import

1415431969273282600


6 months ago

You can connect to that database using the railway cli https://docs.railway.com/guides/cli and import data from there


6 months ago

Once you are connected with a shell you can just use mongo's version of pgdump from a file


mylodiscord
FREEOP

6 months ago

i have to use CLI to import data?


6 months ago

Yes. Theres no GUI for it


6 months ago

cli isnt hard at all haha


6 months ago

looks a bit scary but its not difficult to get a hang of


6 months ago

ss from an old support thread i helped with

1415432490537193500


6 months ago

once installed its simply
railway login
railway link
railway connect


6 months ago

and boom


mylodiscord
FREEOP

6 months ago

but how would I import data?


6 months ago

Actually I just looked online. You are dumping the entire mongo db into a single json file right?


6 months ago

the one you're downloading


mylodiscord
FREEOP

6 months ago

im exporting this one first

1415433276176334800


mylodiscord
FREEOP

6 months ago

there's 2,000+ documents in there



6 months ago

looking through this right now


6 months ago

iirc you can pass the public connection string when running this command.


6 months ago

(that way you dont have to be railway cli sshd in)


mylodiscord
FREEOP

6 months ago

if i could get all my data into the railway db that would be perfect


6 months ago

If you're able to export into a single json/csv file it looks easiest


6 months ago

the entire database that is


mylodiscord
FREEOP

6 months ago

could you help me soon with importing data? i cant do it rn as we have to use the bot in 30 min


6 months ago

I might be free we will see. Ping me when you're done and if im free happy to help


mylodiscord
FREEOP

6 months ago

Okay , thank you so much


mylodiscord
FREEOP

6 months ago

what timezone are you?


6 months ago

I'm EST based


6 months ago

Usually up until 10-10:30 EST


mylodiscord
FREEOP

6 months ago

perfect, we'll be done before that


mylodiscord
FREEOP

6 months ago

btw ^


mylodiscord
FREEOP

6 months ago

should i buy the dedicated plan aswell or nah?


6 months ago

If you're switching over to Railway as your mongo db host you dont need to 😁


mylodiscord
FREEOP

6 months ago

Kk


mylodiscord
FREEOP

6 months ago

btw how do i do railway link in cmd? looked at docs and cant get it working


6 months ago

did you first run railway login ?


mylodiscord
FREEOP

6 months ago

1415437446753161200


mylodiscord
FREEOP

6 months ago

1415437464599925000


6 months ago

Oh you have to restart your cmd instance haha


mylodiscord
FREEOP

6 months ago

Wdym?


mylodiscord
FREEOP

6 months ago

1415437875415093200


6 months ago

OH you installed a "project specific" version of railway


6 months ago

you have to run npm i -g railway


6 months ago

-g means global


6 months ago

then when you restart the CMD your PATH changes take effect


mylodiscord
FREEOP

6 months ago

erm

1415438412541857800


6 months ago

I guess its @railway/cli oops


mylodiscord
FREEOP

6 months ago

npm i -g @Railway/cli?


6 months ago

Sorry, I have homebrew so for me its just brew install railway-cli or something like that


mylodiscord
FREEOP

6 months ago

whoopps


6 months ago

npm i -g @railway/cli yes


mylodiscord
FREEOP

6 months ago

1415438740049887500


mylodiscord
FREEOP

6 months ago

how can i uninstall railway? i tried npm uninstall railway didn't work


6 months ago

I'm quite confused now <:Hmmmmm:891565000068587551>


6 months ago

npm uninstall -g railway


6 months ago

i think


mylodiscord
FREEOP

6 months ago

ok weorked


6 months ago

time to google 😁


6 months ago

oh great


mylodiscord
FREEOP

6 months ago

ok good

1415438920459489500


mylodiscord
FREEOP

6 months ago

works


6 months ago

Wonderful haha


mylodiscord
FREEOP

6 months ago

so what can i do with this?


6 months ago

you can connect to the shell of that mongo db instance.
Or, if you point the mongodb command to a connection string (the public url) it'll work too


mylodiscord
FREEOP

6 months ago

could you guide me with importing data into the railway db? just in case you're offline later


mylodiscord
FREEOP

6 months ago

so i can come back here


mylodiscord
FREEOP

6 months ago

Question @Adam would I be able to put my other bots (Excel and Excel Verification) into railway, split into different services? I use pebble host for those 2 bots atm and pay around $20 total

1415451089242816500


6 months ago

If you have the source code for those bots, absolutely you can


6 months ago

Did you and Noah get splitting your backend + bot working?


mylodiscord
FREEOP

6 months ago

Haven't done that yet, but we found the issue. It was the database, I'm gonna add MongoDB onto railway soon


mylodiscord
FREEOP

6 months ago

Nvm got it all working, thanks @Error for the help 🙂


6 months ago

Awesome! Sorry i just got home otherwise i’d have loved to help with the migration. Is it running any faster?


mylodiscord
FREEOP

6 months ago

Much faster


6 months ago

Yay! Good to know for future reference too.

Still suggest splitting them up into different services haha


mylodiscord
FREEOP

6 months ago

that'll be down the track 😅


mylodiscord
FREEOP

6 months ago

@Error Question, I can create multiple mongodb services for different services right?

1415465087820370000


6 months ago

Yep! Each is a contained mongo database


mylodiscord
FREEOP

6 months ago

Perfect


mylodiscord
FREEOP

6 months ago

@Adam It's been applying 23 changes for 5 mins now

1415496011131912200


mylodiscord
FREEOP

6 months ago

what should i do>?


6 months ago

Give it a reload and let me know if its still applying those changes 😁


mylodiscord
FREEOP

6 months ago

It's fixed now


6 months ago

Oh good to hear!


6 months ago

!s


Status changed to Solved noahd 6 months ago


Loading...