ERROR: could not resize shared memory segment "/PostgreSQL.1432368484" to 12615680 bytes:

pbteja1998
HOBBY

2 years ago

Hi,

I am using Postgres database in Railway and I am getting this error. Can someone please help me figure out why this is happening? All of the solutions I see online tells me to change shm_size of the database. But I don't think there is a way to change shm_size in Railway.

This is only happening for some of the complex or large queries that returns a lot of rows. How to fix this? What can I do to solve this?

Project ID: ef4b4933-5e06-4f9f-93d1-1153cb313e98

Some of the errors I am seeing are these:

ERROR: could not resize shared memory segment "/PostgreSQL.1432368484" to 12615680 bytes: No space left on device

FATAL: terminating background worker "parallel worker" due to administrator command

ERROR: could not resize shared memory segment "/PostgreSQL.1662690888" to 12615680 bytes: No space left on device

background worker "parallel worker" (PID 35856) exited with exit code 1

My usage dashboard says Your plan includes up to 8 GB of RAM, 8 vCPU, and 100 GB of Shared Disk. Manage your Plan.

and I am no where near those limits.

0 Replies

2 years ago

the default shm size on railway is 64mb and there is no way to increase this


pbteja1998
HOBBY

2 years ago

@Brody Yes, I know. Hence this question. Will wait for someone from team to suggest a solution to this.


2 years ago

ive talked to the team about this myself, they don’t seem too interested in this issue, you may need to host the database elsewhere


pbteja1998
HOBBY

2 years ago

Wow! Didn't expect that. Ok, then I guess I need to move my database off of Railway!


2 years ago

yeah for your workload you would need to, as this is definitely a blocker they don’t seem too interested in solving unfortunately


pbteja1998
HOBBY

2 years ago

Got it. Thanks Brody!


2 years ago

no problem, sorry for the letdown here


pbteja1998
HOBBY

2 years ago

Are these any team members who checks the conversations in discord? or is it all community managed?


2 years ago

This is a community forum, you would need to be on Pro in order to get support from the team


2 years ago

but having Pro wouldn't help here as other Pro users have been left in the dark on this issue as well


pepijn
PRO

2 years ago

Not really sure if this answers is satisfying, but I will ask the team directly as well on Slack. Should be as simple as passing a variable to the docker run command.


2 years ago

Yes in theory it's that simple, but they very likely don't have a mechanism in place that would allow them to tack on an extra flag to a specific users container run command


2 years ago

the team has directly told me and another pro user they can't offer custom shm sizes, so unless you are actually enterprise then you may get the same answer I did: you would need to tune postgres to not use as much shared memory


2 years ago

hey all! Brody is correct, you can't pass a flag to your start command, and we hadn't been able to prioritize a fix for this until recently, but I'm happy to report that we do have an environment variable now that enables control over shm_size. the variable name is RAILWAY_SHM_SIZE_BYTES which takes a value of bytes (as the name implies). let us know if you have any problems


2 years ago

I shall test that with my database!

edit, works perfectly!


@Bhanu Teja P, flagging Melissa's note above which should unblock you by allowing you to control shm_size using the RAILWAY_SHM_SIZE_BYTES variable.


pbteja1998
HOBBY

2 years ago

@christian where do I add this env variable? In my repo which connects to the database? Or should I add this in the env variables of the database?


pbteja1998
HOBBY

2 years ago

@Brody where did you add this? And how much did you set it to?


2 years ago

please do not tag the team, you add it to your service variables


pbteja1998
HOBBY

2 years ago

Hmm… ok. Can you please share where that is? Service variables of the database?


2 years ago

correct


You can follow these steps:

  1. Click on your database on the canvas

  2. In the service settings pane, click Variables from the top menu

  3. Click New Variable

  4. Enter VARIABLE_NAME = shm_size // VALUE = your custom shm size

  5. Click Add


pbteja1998
HOBBY

2 years ago

Thanks Christian. Is there a way to directly reach out to your team when needing customer support instead of posting it on a community managed forum. Previously, it used to be so simple to contact the team via discord. Now it looks like my ticket automatically got marked as resolved without the team even seeing it


2 years ago

^


pbteja1998
HOBBY

2 years ago

Yeah, but you said don't take pro as they would say the same thing. If I would have listened to that, I would have migrated to another service


2 years ago

I can only provide the information I am aware of, I did not know they were going to add this variable


2 years ago

the statement is true regardless


pbteja1998
HOBBY

2 years ago

Hmm… ok, thanks.


pbteja1998
HOBBY

2 years ago

I set the value to 512000000 (assuming this means 512 MB)


pbteja1998
HOBBY

2 years ago

Hoping this will solve the problem


2 years ago

please don't set it anywhere near that high, as a hobby user that would likely be far above the limit for shm sizes on the hobby plan when they implement the limits


2 years ago

set it to 128MB, or 134217728 in bytes


pbteja1998
HOBBY

2 years ago

I tried 128 MB, but it's still not enough for my usecase


2 years ago

the error above states you only went 12mb out of memory


pbteja1998
HOBBY

2 years ago

I have no idea. I tried 128 MB, but it still throws the same error. I set it to 512 MB, and no error


pbteja1998
HOBBY

2 years ago

let me set it back to 128 MB again and retry it


2 years ago

we may set the default to 160mb so try that


pbteja1998
HOBBY

2 years ago

160 MB didn't work


pbteja1998
HOBBY

2 years ago

Trying 256 MB next


2 years ago

please provide the errors


pbteja1998
HOBBY

2 years ago

Same error

Raw query failed. Code: `53100`. Message: `db error: ERROR: could not resize shared memory segment "/PostgreSQL.1645299550" to 12615680 bytes: No space left on device`

2 years ago

yeah same error, but the number would change


pbteja1998
HOBBY

2 years ago

It didn't change


pbteja1998
HOBBY

2 years ago

My guess is there are multiple of these queries running in parallel, and together combined is more than 160 MB


pbteja1998
HOBBY

2 years ago

otherwise, it would not have worked for 512 MB too


2 years ago

this is starting to sound not like a hobby usecase if I'm being honest


pbteja1998
HOBBY

2 years ago

what's a hobby usecase?


pbteja1998
HOBBY

2 years ago

Setting it to 256 MB worked. I am going to leave it at this.


2 years ago

that's good to hear


pbteja1998
HOBBY

2 years ago

Thanks for all the help @Brody


2 years ago

no problem


pbteja1998
HOBBY

2 years ago

Now, I don't have to write a migration script anymore 😅


2 years ago

I'll preemptively say this, when / if you use your 5gb volume up on the database, you will need to upgrade to pro


pbteja1998
HOBBY

2 years ago

Where do I check this?


pbteja1998
HOBBY

2 years ago

I currently see this

1227381365599899600


2 years ago

click on the volume attached to postgres


2 years ago

not even close to where you should be looking haha


pbteja1998
HOBBY

2 years ago

This?

1227381626506318000


2 years ago

yep, looks like you are soon going to hit the limit soon, once that happens your database becomes inaccessible


pbteja1998
HOBBY

2 years ago

In Pro, do I get a higher size? How much higher?


2 years ago

pro users get 50gb


pbteja1998
HOBBY

2 years ago

Nice! Will upgrade to Pro


2 years ago

once on pro don't feel bad if you need to set your shm size higher


pbteja1998
HOBBY

2 years ago

As soon as I upgraded to Pro, I see this

1227382730074099700


2 years ago

have you moved the project over to the pro workspace?


pbteja1998
HOBBY

2 years ago

yeah


pbteja1998
HOBBY

2 years ago

what do I do?


2 years ago

please confirm that the project is now located in the pro workspace


pbteja1998
HOBBY

2 years ago

It is

1227383520658325500


pbteja1998
HOBBY

2 years ago

1227383606586904600


2 years ago

show me what's in the drop-down for the workspace selector on the left of this screenshot


pbteja1998
HOBBY

2 years ago

1227383835457622000


2 years ago

interesting, can you quickly check your transactions for the card you used to upgrade, where you charged $20?


pbteja1998
HOBBY

2 years ago

Checking


pbteja1998
HOBBY

2 years ago

There was an unpaid invoice. Paid it. Now, it's back to normal.

1227384850466603000


pbteja1998
HOBBY

2 years ago

Any idea why this would still show 5 GB?

1227384988828434400


2 years ago

you would need to grow it, it's not done automatically as growing the volume means downtime, so it's left to the users discretion, you can grow it when you want to in it's settings menu


pbteja1998
HOBBY

2 years ago

Understood. Again, thanks for all the help!


2 years ago

no problem


ERROR: could not resize shared memory segment "/PostgreSQL.1432368484" to 12615680 bytes: - Railway Help Station