a08f7d7d-760f-41c8-8002-b417a6adf583

@Brody it's urgent please help!

<#727685388893945877> #5

I apologise sir - please redirect me to a person if possible, i am sorry to tag you - but you've always been of great help didn't know what to do

ConnectorError Postgress [Production] Application

i've made the needed change to prevent the error going forward, its in a staged change, feel free to apply when you want

Thanks a ton can you help me with what exactly the issue is ?

not enough shm space

Okay so this won't happen again right ?

after applying my change, no

Thanks a ton, i really appreciate it -
Applied it

Apologies again on tagging you -we have a campaign running and have ~4k-5k concurrent users on platform

Hence was a bit anxious

perhaps you would be interested in enterprise, that way you could wake us up in the middle of the night for platform issues, and we wouldn't send you the readme! 😆

100% for future campaign would consider it - Thanks for the support just bear with me on this one.

And can we keep this open for another day the campaign ends in ~36hrs

yep, wont close, but without enterprise, i can't promise we will be able to answer anywhere as fast as ive done today

Pro SLO is 12 hours iirc

What's timezone ?
I won't be able to this time it's be tough to get approvals.

But thanks a ton I appreciate i just hope this doesn't happen again and we should be good!

timezone doesnt come into play here, you'd get an answer within 12 hours max

understood thanks

Hey team i am getting the above error again

Can you please check

i feel there's a pattern after every certain number of read/writes this happens

what postgres service?

This is logged on Nextjs using with Prisma - is that what you meant ?

nope, in the project you linked, there are two postgres services

Postgres

is the prod env

id please

one moment checking - where can i find it

in the url

postgresql://postgres:*@junction.proxy.rlwy.net:34872/railway

no what i meant haha, but that works too

happened again

please provide the latest error

ConnectorError(ConnectorError { userfacingerror: None, kind: QueryError(PostgresError { code: "55000", message: "lost connection to parallel worker", severity: "ERROR", detail: None, column: None, hint: None }), transient: false })

thats not the same error?

ConnectorError(ConnectorError { user_facing_error: None, kind: QueryError(PostgresError { code: "55000", message: "parallel worker failed to initialize", severity: "ERROR", detail: None, column: None, hint: Some("More details may be available in the server log.") }), transient: false })

OH yeah this looks a different one

where is the next site hosted?

Vercel

then you are opening and closing database connections for every request since its serverless

What can i do here ?

use pgbouncer or run your app on railway within the same project

Is there no other solution ?
As i have continous users on the site

it would be quite difficult to move it

fun fact, we moved the railway.com site from vercel to self hosted on railway live

My problem is time - this is a short term project which closes in <24hrs now

you are bumping up against postgres max connection limits, i can increase them?

Are you sure ?
because it's currently set to 1k

i know, your current connections is 976

989 now

let me restart the db once - that'll reset the connections as well right ?

yeah it would drop a lot of connections, im not sure you want that?

things are already failing - and railway would be back in couple seconds i feel ?

well you already did it

okay closed processes it's 250 now

didn't restart but close processes

climbing fast

Okay let's increase the connection limit if that' the only solve sir

well it would be the fastest

Right yes - thank you so much

is it restarting ?

this could also drop connections, are you sure you want me to set it?

sure let's do it

i can see things failing now

unable to reach db

the database needs to be restarted after making this change, am i good to do that

Yes

okay now max 4k conns

okay thankyou

i'll monitor now and see if it happens again

i'd like to take this time to again mention that i was only able to answer since i was at my laptop on discord, we cannot promise anyone will answer this fast when youre only on pro

ConnectorError(ConnectorError { userfacingerror: None, kind: QueryError(PostgresError { code: "53100", message: "could not resize shared memory segment \"/PostgreSQL.1707297426\" to 196736 bytes: No space left on device", severity: "ERROR", detail: None, column: None, hint: None }), transient: false })

Still getting this

I understand sir - i really appreciate your help

Getting the shared memory issue again

i can increase the shm size to 1gb, this will fully redeploy the database

my data ?

no it doesnt touch your data

okay let's do it - how much time does it take ?

a coupl mins is fine

10 seconds tops

done pleasee

okay, deploying now

yes

done

okay thankyou - i do understand about your enterprise plan solutions and i will 100% consider it for future big launches.

i have to ask, what would you have done if i had gone to sleep? it is 1:30am here after all lol

I am so sorry - the only thing i could have done is keep closing connecitons
and restarting the db hoping atleast some people keep getting though

i know that's the worst thing to do

enterprise is a year commitment btw

how much is it ?

1k /month

paid monthy

clear yeah

will def go for it

next campaign

you'd have to sign a contract to pay 1k per month for a year

i'll reach out to railway once i am ready

sounds good to me

though next time, maybe run the site on railway too, if railway can run railway.com, it can run your site

It's happening again

the parallel worker failed error

hmmm

it's under the limit right ?

would you like me to increase maxworkerprocesses?

Whatever would help

it will restart postgres again

sure

done

okay, looks like worker count is regularly going above the previous default of 8

how much have we set it to now?

256

Okay amazing

Thankyou so much for your help - won't make you stay up for longer - really appreciate all the help

for what its worth, this is technically something you could have done, not saying i had any problem doing it for you, just letting you know that its not like i went in a tweaked secret things in railway, everything ive been doing is something a non admin can do too (non admin as in you still have to own the database in railway)

understood - basically if i knew what to i could have did it too ?

yeah exactly

ahh - understood
My lil minds could not understand what to do but will surely learn

but it still is happening

i don't understand ….

you must really be hammering this database

i think it's the number of users ?
We have had more than a million users in 2 days

damn and youve only paid us $20 <:kekw:788259314607325204>

it's a campaign xD
we're just spending gaining nothing

and we were not ready for this lol

if it was not going to end tomorrow i would have upgraded to enterprise

okay, plan b, i deployed pgbouncer for you, all you need to do is copy its DATABASE_PUBLIC_URL and then update that in vercel

doing it now

i think it's not able to connect to db for some reason

why do you say that, im connected to pgbouncer

but i am sure this is something on my end not yours

who do you have your domain with? cloudflare?

yes

and its simply pointed to the cname vercel gave you?

that's right

and theres still issues?

yeah - i think it's prisma config i am on it currentlyt

what errors are you getting?

no errors just it isn't responding i think

i think we should attempt to run your app on railway and make the switch, cloudflare should make it seemless

btw moving to pgbouncer does it afftect my data ?

do i need to have access to cloudfalre or domain ?

as i don't have it

no it doesnt

yes you would

i don't have it, that would be a problem getting it currently

then thats all plans i have exhausted

so if i could get access - i just deploy it on railway that's it right ?
i'll try to reach the team

then do you want to spend your current time at least trying to run it on railway? do you use any vercel specific features?

edge functions etc nothing special

thats special

oh xD

i think you need to be running on vercel's infra for that dont you?

that was one of the reasons edge functions and ssr + the urgency

Can u help me with how can i check if my traffic is going through pgbouncer ?

have you replaced the old database url in vercel with the one from pgbouncer?

yes i did

the connections gettingin timed out

according to the network metrics, looks like most data is still going to just the database

i might have to revert for now - but i'll check what's wrong

it's some config on my end - i'll handle it - if the team is ready will try moving to railway

you'd likely have to move off the edge functions first

but, i have to sign off for the night as its now 2:30am for me

sure
thanks for your help

Good night sir

thank you

!s