postgres terminating abnormally
gibsfinancedev
PROOP

2 years ago

for some reason when both the ui and i try to connect to my postgres instance, we seem to be getting failures. the error message i am seeing is

psql: error: connection to server at "monorail.proxy.rlwy.net" (000.000.000.000), port 00000 failed: server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.

wondering how to approach this / debug it or maybe there is something i am doing wrong. my start command is

docker-entrypoint.sh -c 'shared_buffers=1GB' -c 'temp_buffers=32MB' -c 'max_connections=500' -c 'work_mem=256MB' -c 'maintenance_work_mem=256MB' -c 'max_wal_size=1GB' -c 'listen_addresses=*'

my docker container is https://hub.docker.com/r/gibsfinance/assets-postgres

the only other difference is that i am using

/var/lib/postgresql/data/pgdata13

for my PGDATA value. i did this change in an effort to get around this weird error that i was seeing where the db was complaining that the database was created by postgres v16, when i built and everything is using v13

83 Replies

gibsfinancedev
PROOP

2 years ago

42bfb463-d11d-431d-b46e-4bceb96bd943


gibsfinancedev
PROOP

2 years ago

what is more is i don't even see any connection break in the logs. they are just chilling, waiting to accept connections


brody
EMPLOYEE

2 years ago

can you check to see if the database is using the legacy or V2 runtime?


brody
EMPLOYEE

2 years ago

please also provide the deploy logs for the database, and a screenshot of the tcp proxy


gibsfinancedev
PROOP

2 years ago

seems to happen for both


gibsfinancedev
PROOP

2 years ago


gibsfinancedev
PROOP

2 years ago

monorail.proxy.rlwy.net:15788


brody
EMPLOYEE

2 years ago

can I get that screenshot please?


gibsfinancedev
PROOP

2 years ago

1249741933597819000


gibsfinancedev
PROOP

2 years ago

oh, wait, is the proxy forwarding to a different port? / do i need to fix the pg port on 5432?


gibsfinancedev
PROOP

2 years ago

that was it


gibsfinancedev
PROOP

2 years ago

i guess that's what i get for working too late at night


brody
EMPLOYEE

2 years ago

beat me to it


brody
EMPLOYEE

2 years ago

though whenever you can, definitely connect to the database via the private network so you can avoid the egress fees


gibsfinancedev
PROOP

2 years ago

ah, interesting. ok. so the "private" network is available from outside of railway?


brody
EMPLOYEE

2 years ago

nope, wouldn't be very private then


brody
EMPLOYEE

2 years ago

that's why I said whenever you can, because you aren't going to be able to use the private network to access the database from your local computer, but your app that runs in the same environment will be able to


gibsfinancedev
PROOP

2 years ago

ok. cool. thank you


brody
EMPLOYEE

2 years ago

let me know if you run into any more difficulties or have any more questions


gibsfinancedev
PROOP

2 years ago

actually, i am having some difficulty for some reason with accessing the db from other services.


gibsfinancedev
PROOP

2 years ago

when i read, it says that it is unable to find the row i am looking for, which is correct, the row does not exist


gibsfinancedev
PROOP

2 years ago

the migrations seem to have worked, but during data collection/seeding i hit this issues of being unable to aquire connection


gibsfinancedev
PROOP

2 years ago

i am also having difficulty for some reason downloading submodules that i access during seeding


brody
EMPLOYEE

2 years ago

can you share the specfic error?


gibsfinancedev
PROOP

2 years ago

yeah, so there are 2


gibsfinancedev
PROOP

2 years ago

'    at Client_PG.acquireConnection (/usr/src/app/node_modules/knex/lib/client.js:311:13)\n' +

'    at Runner.ensureConnection (/usr/src/app/node_modules/knex/lib/execution/runner.js:305:46)\n' +

'    at Runner.run (/usr/src/app/node_modules/knex/lib/execution/runner.js:30:30)\n' +

'    at Target.then (/usr/src/app/node_modules/knex/lib/builder-interface-augmenter.js:24:43)\n' +

'    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)'

brody
EMPLOYEE

2 years ago

thats only 5 log lines, please use this -


gibsfinancedev
PROOP

2 years ago


gibsfinancedev
PROOP

2 years ago

the other error is the no such file or directory, scandir line which i also find odd because it should have downloaded during the docker build


brody
EMPLOYEE

2 years ago

Dockerfile or nixpacks?


gibsfinancedev
PROOP

2 years ago

should be dockerfile { "$schema": "https://railway.app/railway.schema.json", "build": { "builder": "DOCKERFILE", "dockerfilePath": "Dockerfile.worker" }, "deploy": { "runtime": "LEGACY", "numReplicas": 1, "cronSchedule": "0 0 * * * *", "startCommand": "npm run collect", "sleepApplication": false, "restartPolicyType": "ON_FAILURE", "restartPolicyMaxRetries": 10 } }


brody
EMPLOYEE

2 years ago

i dont even think 0 0 * * * * is a valid cron expression


brody
EMPLOYEE

2 years ago

would you even want to be running a worker on a cron schedule? a worker is usually a long lived task


gibsfinancedev
PROOP

2 years ago

oh, it's just what i called it. it does one task for a couple minutes then exits


brody
EMPLOYEE

2 years ago

oh okay cool


gibsfinancedev
PROOP

2 years ago

1249769004323836000


brody
EMPLOYEE

2 years ago

looks like whatever frontend library railway is using can parse it, i would use the more standard 0 * * * * though


gibsfinancedev
PROOP

2 years ago

i think i may have figured out an issue with the git submodules


brody
EMPLOYEE

2 years ago

side question, are you seeing these characters in your logs on railway too? c-200-erc-20⠼ phux


gibsfinancedev
PROOP

2 years ago

so i deleted locally and tried rebuilding and that seemed to fail


gibsfinancedev
PROOP

2 years ago

this is…


gibsfinancedev
PROOP

2 years ago

i'll get you the package


brody
EMPLOYEE

2 years ago

right but do they show up in the logs window on railway?


gibsfinancedev
PROOP

2 years ago

@topcli/spinner


gibsfinancedev
PROOP

2 years ago

oh, no they show up as the character. but the log line is updated so a lot of them show


brody
EMPLOYEE

2 years ago

can i see a screenshot of your logs from railway?


gibsfinancedev
PROOP

2 years ago

1249770081651527700


brody
EMPLOYEE

2 years ago

for the connection issue, try changing LEGACY to V2


brody
EMPLOYEE

2 years ago

thanks


gibsfinancedev
PROOP

2 years ago

ok, so i think the reason i am getting the submodules error is because it does not seem to have the submodules before the dockerfile is run


brody
EMPLOYEE

2 years ago

i unfortunately wouldnt know anything about submodules


gibsfinancedev
PROOP

2 years ago

is there a command that i can run before the docker file is run?


gibsfinancedev
PROOP

2 years ago

if not i can just create a container


gibsfinancedev
PROOP

2 years ago

i suppose this would be closer to editing the command for git pull



brody
EMPLOYEE

2 years ago

whats stopping you from making the necessary modifications to your Dockerfile so that it works properly?


gibsfinancedev
PROOP

2 years ago

it's just a wierd pathway. one that i did not expect. i have to initialize a git repository, then add the origin from github, then pull all the submodules


gibsfinancedev
PROOP

2 years ago

is that the official way to do it?


gibsfinancedev
PROOP

2 years ago

this works for me locally because i have all the submodules already


gibsfinancedev
PROOP

2 years ago

but it seems like railway does not have them when it starts the docker build


brody
EMPLOYEE

2 years ago

right, the .git folder is not pulled


gibsfinancedev
PROOP

2 years ago

cool cool. working through this pathway


gibsfinancedev
PROOP

2 years ago

am i charged for building large images? 2.45GB is what my local is showing


brody
EMPLOYEE

2 years ago

railway does not currently charge for the builder or the registry storage


gibsfinancedev
PROOP

2 years ago

ok, another interesting wrinkle


gibsfinancedev
PROOP

2 years ago

nope. nevermind. application logic


gibsfinancedev
PROOP

2 years ago

ok, here's a question. i am reading files from the submodules i mentioned above

1249797043245809774


brody
EMPLOYEE

2 years ago

you gotta turn off that spinner thing lol


gibsfinancedev
PROOP

2 years ago

i am seeing these fetch failures where it should be going to that path under usr


gibsfinancedev
PROOP

2 years ago

fetch failure 'pls369' -> '/usr/src/app/submodules/pulsechain-assets/blockchain/pulsechain/assets/0xB64ef51C888972c908CFacf59B47C1AfBC0Ab8aC/logo.png'


gibsfinancedev
PROOP

2 years ago

i am wondering if there are permissions that i need to be concerned about


gibsfinancedev
PROOP

2 years ago

like fs permissions


gibsfinancedev
PROOP

2 years ago

i'll work on it


brody
EMPLOYEE

2 years ago

"fetch failure" is no too helpfull, do you have a more verbose error?


gibsfinancedev
PROOP

2 years ago

working on that too


gibsfinancedev
PROOP

2 years ago

basically, it failed to read from the file system


brody
EMPLOYEE

2 years ago

does the file exist at that location?


gibsfinancedev
PROOP

2 years ago

hm. thinking it's a different weird error. i'll investigate


gibsfinancedev
PROOP

2 years ago

is this anything to be concerned about? when i remove the active ones the ui eventually shows the top one as active

1249816513561760000


brody
EMPLOYEE

2 years ago

yeah I've been seeing that a lot


brody
EMPLOYEE

2 years ago

hey @gibsfinance the team just pushed a fix for this, can you let me know if you are able to remove those extra services?


gibsfinancedev
PROOP

2 years ago

hey, looks good on my end. thanks!


brody
EMPLOYEE

2 years ago

awsome!


Loading...