postgres terminating abnormally

gibsfinancedevPRO

a year ago

for some reason when both the ui and i try to connect to my postgres instance, we seem to be getting failures. the error message i am seeing is

psql: error: connection to server at "monorail.proxy.rlwy.net" (000.000.000.000), port 00000 failed: server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.

wondering how to approach this / debug it or maybe there is something i am doing wrong. my start command is

docker-entrypoint.sh -c 'shared_buffers=1GB' -c 'temp_buffers=32MB' -c 'max_connections=500' -c 'work_mem=256MB' -c 'maintenance_work_mem=256MB' -c 'max_wal_size=1GB' -c 'listen_addresses=*'

my docker container is https://hub.docker.com/r/gibsfinance/assets-postgres

the only other difference is that i am using

/var/lib/postgresql/data/pgdata13

for my PGDATA value. i did this change in an effort to get around this weird error that i was seeing where the db was complaining that the database was created by postgres v16, when i built and everything is using v13

0 Replies

gibsfinancedevPRO

a year ago

42bfb463-d11d-431d-b46e-4bceb96bd943


gibsfinancedevPRO

a year ago

what is more is i don't even see any connection break in the logs. they are just chilling, waiting to accept connections


a year ago

can you check to see if the database is using the legacy or V2 runtime?


a year ago

please also provide the deploy logs for the database, and a screenshot of the tcp proxy


gibsfinancedevPRO

a year ago

seems to happen for both


gibsfinancedevPRO

a year ago


gibsfinancedevPRO

a year ago

monorail.proxy.rlwy.net:15788


a year ago

can I get that screenshot please?


gibsfinancedevPRO

a year ago

1249741933597819000


gibsfinancedevPRO

a year ago

oh, wait, is the proxy forwarding to a different port? / do i need to fix the pg port on 5432?


gibsfinancedevPRO

a year ago

that was it


gibsfinancedevPRO

a year ago

i guess that's what i get for working too late at night


a year ago

beat me to it


a year ago

though whenever you can, definitely connect to the database via the private network so you can avoid the egress fees


gibsfinancedevPRO

a year ago

ah, interesting. ok. so the "private" network is available from outside of railway?


a year ago

nope, wouldn't be very private then


a year ago

that's why I said whenever you can, because you aren't going to be able to use the private network to access the database from your local computer, but your app that runs in the same environment will be able to


gibsfinancedevPRO

a year ago

ok. cool. thank you


a year ago

let me know if you run into any more difficulties or have any more questions


gibsfinancedevPRO

a year ago

actually, i am having some difficulty for some reason with accessing the db from other services.


gibsfinancedevPRO

a year ago

when i read, it says that it is unable to find the row i am looking for, which is correct, the row does not exist


gibsfinancedevPRO

a year ago

the migrations seem to have worked, but during data collection/seeding i hit this issues of being unable to aquire connection


gibsfinancedevPRO

a year ago

i am also having difficulty for some reason downloading submodules that i access during seeding


a year ago

can you share the specfic error?


gibsfinancedevPRO

a year ago

yeah, so there are 2


gibsfinancedevPRO

a year ago

'    at Client_PG.acquireConnection (/usr/src/app/node_modules/knex/lib/client.js:311:13)\n' +

'    at Runner.ensureConnection (/usr/src/app/node_modules/knex/lib/execution/runner.js:305:46)\n' +

'    at Runner.run (/usr/src/app/node_modules/knex/lib/execution/runner.js:30:30)\n' +

'    at Target.then (/usr/src/app/node_modules/knex/lib/builder-interface-augmenter.js:24:43)\n' +

'    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)'

a year ago

thats only 5 log lines, please use this -


gibsfinancedevPRO

a year ago


gibsfinancedevPRO

a year ago

the other error is the no such file or directory, scandir line which i also find odd because it should have downloaded during the docker build


a year ago

Dockerfile or nixpacks?


gibsfinancedevPRO

a year ago

should be dockerfile { "$schema": "https://railway.app/railway.schema.json", "build": { "builder": "DOCKERFILE", "dockerfilePath": "Dockerfile.worker" }, "deploy": { "runtime": "LEGACY", "numReplicas": 1, "cronSchedule": "0 0 * * * *", "startCommand": "npm run collect", "sleepApplication": false, "restartPolicyType": "ON_FAILURE", "restartPolicyMaxRetries": 10 } }


a year ago

i dont even think 0 0 * * * * is a valid cron expression


a year ago

would you even want to be running a worker on a cron schedule? a worker is usually a long lived task


gibsfinancedevPRO

a year ago

oh, it's just what i called it. it does one task for a couple minutes then exits


a year ago

oh okay cool


gibsfinancedevPRO

a year ago

1249769004323836000


a year ago

looks like whatever frontend library railway is using can parse it, i would use the more standard 0 * * * * though


gibsfinancedevPRO

a year ago

i think i may have figured out an issue with the git submodules


a year ago

side question, are you seeing these characters in your logs on railway too? c-200-erc-20⠼ phux


gibsfinancedevPRO

a year ago

so i deleted locally and tried rebuilding and that seemed to fail


gibsfinancedevPRO

a year ago

this is…


gibsfinancedevPRO

a year ago

i'll get you the package


a year ago

right but do they show up in the logs window on railway?


gibsfinancedevPRO

a year ago

@topcli/spinner


gibsfinancedevPRO

a year ago

oh, no they show up as the character. but the log line is updated so a lot of them show


a year ago

can i see a screenshot of your logs from railway?


gibsfinancedevPRO

a year ago

1249770081651527700


a year ago

for the connection issue, try changing LEGACY to V2


a year ago

thanks


gibsfinancedevPRO

a year ago

ok, so i think the reason i am getting the submodules error is because it does not seem to have the submodules before the dockerfile is run


a year ago

i unfortunately wouldnt know anything about submodules


gibsfinancedevPRO

a year ago

is there a command that i can run before the docker file is run?


gibsfinancedevPRO

a year ago

if not i can just create a container


gibsfinancedevPRO

a year ago

i suppose this would be closer to editing the command for git pull



a year ago

whats stopping you from making the necessary modifications to your Dockerfile so that it works properly?


gibsfinancedevPRO

a year ago

it's just a wierd pathway. one that i did not expect. i have to initialize a git repository, then add the origin from github, then pull all the submodules


gibsfinancedevPRO

a year ago

is that the official way to do it?


gibsfinancedevPRO

a year ago

this works for me locally because i have all the submodules already


gibsfinancedevPRO

a year ago

but it seems like railway does not have them when it starts the docker build


a year ago

right, the .git folder is not pulled


gibsfinancedevPRO

a year ago

cool cool. working through this pathway


gibsfinancedevPRO

a year ago

am i charged for building large images? 2.45GB is what my local is showing


a year ago

railway does not currently charge for the builder or the registry storage


gibsfinancedevPRO

a year ago

ok, another interesting wrinkle


gibsfinancedevPRO

a year ago

nope. nevermind. application logic


gibsfinancedevPRO

a year ago

ok, here's a question. i am reading files from the submodules i mentioned above

1249797043245809774


a year ago

you gotta turn off that spinner thing lol


gibsfinancedevPRO

a year ago

i am seeing these fetch failures where it should be going to that path under usr


gibsfinancedevPRO

a year ago

fetch failure 'pls369' -> '/usr/src/app/submodules/pulsechain-assets/blockchain/pulsechain/assets/0xB64ef51C888972c908CFacf59B47C1AfBC0Ab8aC/logo.png'


gibsfinancedevPRO

a year ago

i am wondering if there are permissions that i need to be concerned about


gibsfinancedevPRO

a year ago

like fs permissions


gibsfinancedevPRO

a year ago

i'll work on it


a year ago

"fetch failure" is no too helpfull, do you have a more verbose error?


gibsfinancedevPRO

a year ago

working on that too


gibsfinancedevPRO

a year ago

basically, it failed to read from the file system


a year ago

does the file exist at that location?


gibsfinancedevPRO

a year ago

hm. thinking it's a different weird error. i'll investigate


gibsfinancedevPRO

a year ago

is this anything to be concerned about? when i remove the active ones the ui eventually shows the top one as active

1249816513561760000


a year ago

yeah I've been seeing that a lot


10 months ago

hey @gibsfinance the team just pushed a fix for this, can you let me know if you are able to remove those extra services?


gibsfinancedevPRO

10 months ago

hey, looks good on my end. thanks!


10 months ago

awsome!


postgres terminating abnormally - Railway Help Station