ERROR: could not resize shared memory segment "/PostgreSQL.1432368484" to 12615680 bytes:

pbteja1998
HOBBY

a year ago

Hi,

I am using Postgres database in Railway and I am getting this error. Can someone please help me figure out why this is happening? All of the solutions I see online tells me to change shm_size of the database. But I don't think there is a way to change shm_size in Railway.

This is only happening for some of the complex or large queries that returns a lot of rows. How to fix this? What can I do to solve this?

Project ID: ef4b4933-5e06-4f9f-93d1-1153cb313e98

Some of the errors I am seeing are these:

ERROR: could not resize shared memory segment "/PostgreSQL.1432368484" to 12615680 bytes: No space left on device

FATAL: terminating background worker "parallel worker" due to administrator command

ERROR: could not resize shared memory segment "/PostgreSQL.1662690888" to 12615680 bytes: No space left on device

background worker "parallel worker" (PID 35856) exited with exit code 1

My usage dashboard says Your plan includes up to 8 GB of RAM, 8 vCPU, and 100 GB of Shared Disk. Manage your Plan.

and I am no where near those limits.

0 Replies

a year ago

the default shm size on railway is 64mb and there is no way to increase this


pbteja1998
HOBBY

a year ago

@Brody Yes, I know. Hence this question. Will wait for someone from team to suggest a solution to this.


a year ago

ive talked to the team about this myself, they don’t seem too interested in this issue, you may need to host the database elsewhere


pbteja1998
HOBBY

a year ago

Wow! Didn't expect that. Ok, then I guess I need to move my database off of Railway!


a year ago

yeah for your workload you would need to, as this is definitely a blocker they don’t seem too interested in solving unfortunately


pbteja1998
HOBBY

a year ago

Got it. Thanks Brody!


a year ago

no problem, sorry for the letdown here


pbteja1998
HOBBY

a year ago

Are these any team members who checks the conversations in discord? or is it all community managed?


a year ago

This is a community forum, you would need to be on Pro in order to get support from the team


a year ago

but having Pro wouldn't help here as other Pro users have been left in the dark on this issue as well


pepijn
PRO

a year ago

Not really sure if this answers is satisfying, but I will ask the team directly as well on Slack. Should be as simple as passing a variable to the docker run command.


a year ago

Yes in theory it's that simple, but they very likely don't have a mechanism in place that would allow them to tack on an extra flag to a specific users container run command


a year ago

the team has directly told me and another pro user they can't offer custom shm sizes, so unless you are actually enterprise then you may get the same answer I did: you would need to tune postgres to not use as much shared memory


a year ago

hey all! Brody is correct, you can't pass a flag to your start command, and we hadn't been able to prioritize a fix for this until recently, but I'm happy to report that we do have an environment variable now that enables control over shm_size. the variable name is RAILWAY_SHM_SIZE_BYTES which takes a value of bytes (as the name implies). let us know if you have any problems


a year ago

I shall test that with my database!

edit, works perfectly!


@Bhanu Teja P, flagging Melissa's note above which should unblock you by allowing you to control shm_size using the RAILWAY_SHM_SIZE_BYTES variable.


pbteja1998
HOBBY

a year ago

@christian where do I add this env variable? In my repo which connects to the database? Or should I add this in the env variables of the database?


pbteja1998
HOBBY

a year ago

@Brody where did you add this? And how much did you set it to?


a year ago

please do not tag the team, you add it to your service variables


pbteja1998
HOBBY

a year ago

Hmm… ok. Can you please share where that is? Service variables of the database?


a year ago

correct


You can follow these steps:

  1. Click on your database on the canvas

  2. In the service settings pane, click Variables from the top menu

  3. Click New Variable

  4. Enter VARIABLE_NAME = shm_size // VALUE = your custom shm size

  5. Click Add


pbteja1998
HOBBY

a year ago

Thanks Christian. Is there a way to directly reach out to your team when needing customer support instead of posting it on a community managed forum. Previously, it used to be so simple to contact the team via discord. Now it looks like my ticket automatically got marked as resolved without the team even seeing it


a year ago

^


pbteja1998
HOBBY

a year ago

Yeah, but you said don't take pro as they would say the same thing. If I would have listened to that, I would have migrated to another service


a year ago

I can only provide the information I am aware of, I did not know they were going to add this variable


a year ago

the statement is true regardless


pbteja1998
HOBBY

a year ago

Hmm… ok, thanks.


pbteja1998
HOBBY

a year ago

I set the value to 512000000 (assuming this means 512 MB)


pbteja1998
HOBBY

a year ago

Hoping this will solve the problem


a year ago

please don't set it anywhere near that high, as a hobby user that would likely be far above the limit for shm sizes on the hobby plan when they implement the limits


a year ago

set it to 128MB, or 134217728 in bytes


pbteja1998
HOBBY

a year ago

I tried 128 MB, but it's still not enough for my usecase


a year ago

the error above states you only went 12mb out of memory


pbteja1998
HOBBY

a year ago

I have no idea. I tried 128 MB, but it still throws the same error. I set it to 512 MB, and no error


pbteja1998
HOBBY

a year ago

let me set it back to 128 MB again and retry it


a year ago

we may set the default to 160mb so try that


pbteja1998
HOBBY

a year ago

160 MB didn't work


pbteja1998
HOBBY

a year ago

Trying 256 MB next


a year ago

please provide the errors


pbteja1998
HOBBY

a year ago

Same error

Raw query failed. Code: `53100`. Message: `db error: ERROR: could not resize shared memory segment "/PostgreSQL.1645299550" to 12615680 bytes: No space left on device`

a year ago

yeah same error, but the number would change


pbteja1998
HOBBY

a year ago

It didn't change


pbteja1998
HOBBY

a year ago

My guess is there are multiple of these queries running in parallel, and together combined is more than 160 MB


pbteja1998
HOBBY

a year ago

otherwise, it would not have worked for 512 MB too


a year ago

this is starting to sound not like a hobby usecase if I'm being honest


pbteja1998
HOBBY

a year ago

what's a hobby usecase?


pbteja1998
HOBBY

a year ago

Setting it to 256 MB worked. I am going to leave it at this.


a year ago

that's good to hear


pbteja1998
HOBBY

a year ago

Thanks for all the help @Brody


a year ago

no problem


pbteja1998
HOBBY

a year ago

Now, I don't have to write a migration script anymore 😅


a year ago

I'll preemptively say this, when / if you use your 5gb volume up on the database, you will need to upgrade to pro


pbteja1998
HOBBY

a year ago

Where do I check this?


pbteja1998
HOBBY

a year ago

I currently see this

1227381365599899600


a year ago

click on the volume attached to postgres


a year ago

not even close to where you should be looking haha


pbteja1998
HOBBY

a year ago

This?

1227381626506318000


a year ago

yep, looks like you are soon going to hit the limit soon, once that happens your database becomes inaccessible


pbteja1998
HOBBY

a year ago

In Pro, do I get a higher size? How much higher?


a year ago

pro users get 50gb


pbteja1998
HOBBY

a year ago

Nice! Will upgrade to Pro


a year ago

once on pro don't feel bad if you need to set your shm size higher


pbteja1998
HOBBY

a year ago

As soon as I upgraded to Pro, I see this

1227382730074099700


a year ago

have you moved the project over to the pro workspace?


pbteja1998
HOBBY

a year ago

yeah


pbteja1998
HOBBY

a year ago

what do I do?


a year ago

please confirm that the project is now located in the pro workspace


pbteja1998
HOBBY

a year ago

It is

1227383520658325500


pbteja1998
HOBBY

a year ago

1227383606586904600


a year ago

show me what's in the drop-down for the workspace selector on the left of this screenshot


pbteja1998
HOBBY

a year ago

1227383835457622000


a year ago

interesting, can you quickly check your transactions for the card you used to upgrade, where you charged $20?


pbteja1998
HOBBY

a year ago

Checking


pbteja1998
HOBBY

a year ago

There was an unpaid invoice. Paid it. Now, it's back to normal.

1227384850466603000


pbteja1998
HOBBY

a year ago

Any idea why this would still show 5 GB?

1227384988828434400


a year ago

you would need to grow it, it's not done automatically as growing the volume means downtime, so it's left to the users discretion, you can grow it when you want to in it's settings menu


pbteja1998
HOBBY

a year ago

Understood. Again, thanks for all the help!


a year ago

no problem


ERROR: could not resize shared memory segment "/PostgreSQL.1432368484" to 12615680 bytes: - Railway Help Station