a year ago
Hi,
I am using Postgres database in Railway and I am getting this error. Can someone please help me figure out why this is happening? All of the solutions I see online tells me to change shm_size
of the database. But I don't think there is a way to change shm_size
in Railway.
This is only happening for some of the complex or large queries that returns a lot of rows. How to fix this? What can I do to solve this?
Project ID: ef4b4933-5e06-4f9f-93d1-1153cb313e98
Some of the errors I am seeing are these:
ERROR: could not resize shared memory segment "/PostgreSQL.1432368484" to 12615680 bytes: No space left on device
FATAL: terminating background worker "parallel worker" due to administrator command
ERROR: could not resize shared memory segment "/PostgreSQL.1662690888" to 12615680 bytes: No space left on device
background worker "parallel worker" (PID 35856) exited with exit code 1
My usage dashboard says Your plan includes up to 8 GB of RAM, 8 vCPU, and 100 GB of Shared Disk. Manage your Plan.
and I am no where near those limits.
0 Replies
a year ago
the default shm size on railway is 64mb and there is no way to increase this
@Brody Yes, I know. Hence this question. Will wait for someone from team to suggest a solution to this.
a year ago
ive talked to the team about this myself, they don’t seem too interested in this issue, you may need to host the database elsewhere
Wow! Didn't expect that. Ok, then I guess I need to move my database off of Railway!
a year ago
yeah for your workload you would need to, as this is definitely a blocker they don’t seem too interested in solving unfortunately
a year ago
no problem, sorry for the letdown here
Are these any team members who checks the conversations in discord? or is it all community managed?
a year ago
This is a community forum, you would need to be on Pro in order to get support from the team
a year ago
but having Pro wouldn't help here as other Pro users have been left in the dark on this issue as well
Not really sure if this answers is satisfying, but I will ask the team directly as well on Slack. Should be as simple as passing a variable to the docker run command.
a year ago
Yes in theory it's that simple, but they very likely don't have a mechanism in place that would allow them to tack on an extra flag to a specific users container run command
a year ago
the team has directly told me and another pro user they can't offer custom shm sizes, so unless you are actually enterprise then you may get the same answer I did: you would need to tune postgres to not use as much shared memory
a year ago
hey all! Brody is correct, you can't pass a flag to your start command, and we hadn't been able to prioritize a fix for this until recently, but I'm happy to report that we do have an environment variable now that enables control over shm_size
. the variable name is RAILWAY_SHM_SIZE_BYTES
which takes a value of bytes (as the name implies). let us know if you have any problems
a year ago
I shall test that with my database!
edit, works perfectly!
a year ago
@Bhanu Teja P, flagging Melissa's note above which should unblock you by allowing you to control shm_size
using the RAILWAY_SHM_SIZE_BYTES
variable.
@christian where do I add this env variable? In my repo which connects to the database? Or should I add this in the env variables of the database?
a year ago
please do not tag the team, you add it to your service variables
Hmm… ok. Can you please share where that is? Service variables of the database?
a year ago
correct
a year ago
You can follow these steps:
Click on your database on the canvas
In the service settings pane, click Variables from the top menu
Click
New Variable
Enter VARIABLE_NAME =
shm_size
// VALUE = your custom shm sizeClick
Add
Thanks Christian. Is there a way to directly reach out to your team when needing customer support instead of posting it on a community managed forum. Previously, it used to be so simple to contact the team via discord. Now it looks like my ticket automatically got marked as resolved without the team even seeing it
a year ago
^
Yeah, but you said don't take pro as they would say the same thing. If I would have listened to that, I would have migrated to another service
a year ago
I can only provide the information I am aware of, I did not know they were going to add this variable
a year ago
the statement is true regardless
a year ago
please don't set it anywhere near that high, as a hobby user that would likely be far above the limit for shm sizes on the hobby plan when they implement the limits
a year ago
set it to 128MB, or 134217728 in bytes
a year ago
the error above states you only went 12mb out of memory
I have no idea. I tried 128 MB, but it still throws the same error. I set it to 512 MB, and no error
a year ago
we may set the default to 160mb so try that
a year ago
please provide the errors
Same error
Raw query failed. Code: `53100`. Message: `db error: ERROR: could not resize shared memory segment "/PostgreSQL.1645299550" to 12615680 bytes: No space left on device`
a year ago
yeah same error, but the number would change
My guess is there are multiple of these queries running in parallel, and together combined is more than 160 MB
a year ago
this is starting to sound not like a hobby usecase if I'm being honest
a year ago
that's good to hear
a year ago
no problem
a year ago
I'll preemptively say this, when / if you use your 5gb volume up on the database, you will need to upgrade to pro
a year ago
click on the volume attached to postgres
a year ago
not even close to where you should be looking haha
a year ago
yep, looks like you are soon going to hit the limit soon, once that happens your database becomes inaccessible
a year ago
pro users get 50gb
a year ago
once on pro don't feel bad if you need to set your shm size higher
a year ago
have you moved the project over to the pro workspace?
a year ago
please confirm that the project is now located in the pro workspace
a year ago
show me what's in the drop-down for the workspace selector on the left of this screenshot
a year ago
interesting, can you quickly check your transactions for the card you used to upgrade, where you charged $20?
a year ago
you would need to grow it, it's not done automatically as growing the volume means downtime, so it's left to the users discretion, you can grow it when you want to in it's settings menu
a year ago
no problem