All plans use the same server spec, could you please share more details about your use case as well as usage details?

I’m assuming you mean a discord bot

discord bot + backend/ws

Just the single service? The bot and backend are in the same service?

yeah

my structure

src is discord bot, socket-server.js handles all socket related stuff.

There’s your issue. When your bot receives a ton of requests at once, your websocket server gets tied up servicing them all. If you architect your project such that your bot is in a separate service from your backend, you’ll be able to scale up your backend using replicas to handle more requests at once

This is not a Railway issue, but a project architecture issue. If you change the architecture of your bot you’ll be able to dramatically improve performance

so what should I change?

modify your project so that your bot and backend are in different folders

Create a new service and deploy your backend to one, your bot to the other

Make sure that’s all working, then depending on the way your backend works, you may be able to use replicas. Regardless, they’re on different machines now, so they’re not sharing resources and will have more capacity

so should i make another workspace for my bot, and deploy the bot on railway and have the socket server seperate? what if i need to use certain stuff from the socket-server.js file

No, not another workspace. Another service

What do you need to use from the server file?

I use this from the socket-server

Sounds like you made two separate parts of your app pretty tightly coupled, that’s not great and will heavily restrict the optimizations you can do

Wdym another service?

this box on your bot’s project is a service. It’s an individual container hosting your bot’s code. The goal here is to separate your bot’s container from your server’s container so they don’t share resources and you can scale up your server count

how do I make another one

Should be a new service button at the top right of the project page, or right click anywhere in your project and select new service

this?

yes

what would I have to change in my code? will I have to re-code everything?

You will need to recode quite a bit, it seems like. But if you’re expecting to handle a lot of users, this is what you’ll need to do

😭

1.5k+ lines in socket-server.js

All that needs to be done is decouple your bot from your backend, that shouldn’t be too complex

which one would i add?

Recoding your app is the first step here, you’ll add a new service for your server once you’ve done that

But if your code is on Github, then Github. However you uploaded your first service

ok, what would i need to re-code? socket or discord bot

I don’t know enough about your project to tell you that

Any place where they’re directly referencing code files from each other, they shouldn’t

i'll tell you more info soon, ppl are interacting with our bot, 100+ ppl

would you be able to help me soon? @Adam

I’ve given you all the information I can, at this point all that’s left is for you to start coding

So keep everything in one workspace?

i'm a bit confused

Keep everything in one workspace, one project, but not one service

^

so all i need to do is create 2 folders - 1 is discord bot and 2 is websocket?

Pretty much yes, but you need to make sure those two folders don’t have any crossover.

Once the two are decoupled, you host each one as an individual service on Railway

ah… i see, we were using the bot in a small server (around 8,000-9,000 member server), and we just went back to our old discord (with 60,000+ members), and it's really slow in that cord, probably because of the member count.. just tested the bot in the 9k member cord and its really fast

Are you implementing discord's sharding feature? I think they suggest it around the ~10k mark

i thought sharding is only needed if the bot is in 2,000+ servers or something?

Oh i might be mistaking then. Havent had a need to use it hahaha

let me go read up on the docs real quick

And I assume this is a discord.js bot right?

yeah, discord.js bot + uses socket.io

has a bot, and a backend

Do you have any leveling logic on messageCreate event?

Or something that would get triggered on a commonly occuring event. Curious if theres some hanging with cache, db calls, or operations.

could you elaborate on "leveling logic"

Was driving at db related functionality on messageCreate events.
Some servers have an "xp" system where you gain certain xp points per message sent.

ohh, we have the following :

messageReactionAdd
messageReactionRemove
messageCreate

we have a channel if we need some people , we ask for people in there and they type and it'll return with a button to add them

Oh my…

There's your problem right there!

if i understand correctly, each and every one of those events is being fired every time a user sends a message.

tho, Adam said it's because i have socket-server and a discord bot in the same project and I should make 2 separate services

<:Hmmmmm:891565000068587551> that doesnt sound like it would be a large factor although I could see the concern.

We need those commands tho

Heres my suggestion for steps of debugging and verification.

Split this into 2 major services (easiest). Websocket (backend) discord bot (the bot).
Have them communicate over private network. In general thats better practice long term.
It'll also help verify thats not the issue.

I'm also curious, what database are you using?

MongoDB

tried this last night, there's so much that needs to be changed.. 😭

Yeahhh in terms of best project structure that's honestly your best bet.
It'll help you service things much better in the future

thing is, i don't think i can do it all in one day, and we use this bot for our discord daily, at around 6pm est

Yeah i can understand that.
So for a bit of preface, were you using this bot in this 60k member server before?

Can I get a bit of info on the timeline?

We provide "scrim-practice" for the Fortnite community to over 60,000 members, and we use that bot to host "scrim-practice", like practice games

No, we were using it in a 3,000 member server & a 9,000 member server

Have you always been using Railway to host it?

Yes

Okay yeah that clears things up

in your server, can you search for the total number of messages from a given day?

i can try, one sec

curious what this number is

😭

our cord is really active

And by any chance, on the 9k one what are you seeing?

Mhm

So

we don't use the server anymore, its locked down

Well you can search on a past day

if you have a rough time of when it was most active

What you're running into is scaling issues.
60k is a very large number. From a 9k member server -> 60k member server you're looking at an increase of 6 times.
From 3k -> 60k thats ~20x larger.

If i understand that screenshot with folder structure correctly, it looks like you're running ~17 different events every single time somebody sends a message.
If each of those events trigger a DB call or are blocking you're having race conditions and a whole host of problems that come with databases and scaling

so what should i do?

Typing out what I would do now <:picassoface:1398147502598918144>

1415422112302366811

here's a vid of what i mean:

ah,.

wait one sec

https://streamable.com/jbcyi2

have a look

see how long it takes

If i were in your situation here's what I would do. The server didnt grow from 0-60 overnight. I assume you guys were using something before so I highly suggest switching back to it and doing maintenance on this bot for a few days.
Split it into 2 services, for our conversations sake: BOT and WS.

Each of those services will run on Railway in a separate service. (Same project)
Now you have per service 32g of ram and 32vCPU available. You shouldnt ever run into a resource issue on them.

That's step 1.

Your bot should never actively query the database it should only make a request to WS which is the only service who talks to your database.
Each of your events is making a call to your websocket so look into pairing down.

I suggest figuring out which parts of the logic actively need to talk to your database and what doesnt. I'd implement a layer of caching with Redis for repetitive calls (username, user id, avatar, team, etc etc.)

Split the services up + cache data will fix a good bit of the latency

I'd also do some live debugging to figure out what part of that process is taking the longest. Is it the db call? Response from websocket? Once you know what's taking longest you can debug that

unsure, it was instant in the other servers with low member count

there's 9k members joining instantly? Sheesh

i can show you via dm

Sure my dms are open. As for solving this I prefer keeping them in threads so if other users have similar issues they can reference this

Yeah @mylo I think I'd look into doing what I suggested with figuring out which process is taking longest

Simple as adding console logs at certain points with durations until next points

i'll do it later as we have to use the bot in 1 hour

Alright. I'm more then happy to help with suggestions/direction however I can 😁

❤️

@mylo Railway supports running a mongo DB on project!

how would this work?

I'm not personally familiar with mongo as much as I'd like, the process would be backing up the cloud mongo db you pay for and restoring to the railway hosted one.

That'll remove all network latency on each request and could help by a decent margin

how would i connect it?

In your service its pointing to a connection url/uri so you would just swap the one its looking for out to the INTERNAL_URL variable found in the variables tab of that service

ohh, do i just need to grab the mongo url and update it in the websocket-server

Yep! But I suggest the internal one

whichj one ?

You'd be using this one

this is still sensitive i wouldnt share it

can i keep the name like this and just grab the connection string and replace it?

wdym

Is that the env var on your websocket service?

yeah

Oh perfect, type this

in the value side

${{MongoDB.INTERNAL_URL}}

It's a variable reference so it'll go look at the value of that env var on the mongo service and use that

ok, one sec

all my collections are gone, i cant use it

this one has 2k i need all of it

Yeah its a blank mongodb. As I mentioned, you would need to save a copy/backup from the production mongo one and back it up onto that railway project

Holy…

Big difference

unsure the best way to do that

It's instant LOL

Did you backup things?

Not yet

what should i export

JSON/CSV

JSON right?

I usually love JSON. Totally up to you though

Make sure that any changes you do are non destructive haha

and how do I import it into railway?

theres no import

You can connect to that database using the railway cli https://docs.railway.com/guides/cli and import data from there

Once you are connected with a shell you can just use mongo's version of pgdump from a file

i have to use CLI to import data?

Yes. Theres no GUI for it

cli isnt hard at all haha

looks a bit scary but its not difficult to get a hang of

ss from an old support thread i helped with

once installed its simply
railway login
railway link
railway connect

and boom

but how would I import data?

Actually I just looked online. You are dumping the entire mongo db into a single json file right?

the one you're downloading

im exporting this one first

there's 2,000+ documents in there

https://www.mongodb.com/docs/database-tools/mongoimport/mongoimport-examples/

looking through this right now

iirc you can pass the public connection string when running this command.

(that way you dont have to be railway cli sshd in)

if i could get all my data into the railway db that would be perfect

If you're able to export into a single json/csv file it looks easiest

the entire database that is

could you help me soon with importing data? i cant do it rn as we have to use the bot in 30 min

I might be free we will see. Ping me when you're done and if im free happy to help

Okay , thank you so much

what timezone are you?

I'm EST based

Usually up until 10-10:30 EST

perfect, we'll be done before that

btw ^

should i buy the dedicated plan aswell or nah?

If you're switching over to Railway as your mongo db host you dont need to 😁

Kk

btw how do i do railway link in cmd? looked at docs and cant get it working

did you first run railway login ?

Oh you have to restart your cmd instance haha

Wdym?

OH you installed a "project specific" version of railway

you have to run npm i -g railway

-g means global

then when you restart the CMD your PATH changes take effect

erm

I guess its @railway/cli oops

npm i -g @Railway/cli?

Sorry, I have homebrew so for me its just brew install railway-cli or something like that

whoopps

npm i -g @railway/cli yes

how can i uninstall railway? i tried npm uninstall railway didn't work

I'm quite confused now <:Hmmmmm:891565000068587551>

npm uninstall -g railway

i think

ok weorked

time to google 😁

oh great

ok good

works

Wonderful haha

so what can i do with this?

you can connect to the shell of that mongo db instance.
Or, if you point the mongodb command to a connection string (the public url) it'll work too

could you guide me with importing data into the railway db? just in case you're offline later

so i can come back here

Question @Adam would I be able to put my other bots (Excel and Excel Verification) into railway, split into different services? I use pebble host for those 2 bots atm and pay around $20 total

If you have the source code for those bots, absolutely you can

Did you and Noah get splitting your backend + bot working?

Haven't done that yet, but we found the issue. It was the database, I'm gonna add MongoDB onto railway soon

Nvm got it all working, thanks @Error for the help 🙂

Awesome! Sorry i just got home otherwise i’d have loved to help with the migration. Is it running any faster?

Much faster

Yay! Good to know for future reference too.

Still suggest splitting them up into different services haha

that'll be down the track 😅

@Error Question, I can create multiple mongodb services for different services right?

Yep! Each is a contained mongo database