to be clear, you are getting this error during build or during runtime?

during the build

i've tried re-deployment a couple of times.. it's possible there's an intermittent connection issue, i do not have adequate monitoring

Should I just try again?

I started another attempt

looked like it was able to connect for the 3rd attempt

try running your migrations during runtime with the private network

i'll work on that, thanks

so the deployment i did 10m ago was successful, I had not changed anything

I just changed the database variable to use the internal one, and i'm getting the same error I was getting earlier:
#12 1.987 Error: P1001: Can't reach database server at postgres.railway.internal:5432

well.. same, except the reference being to the internal one

is there some particular internal-networking thing I need to enable?

change the runtime to v2 in the service settings

i'm not familiar with this setting, is there anything i should know?

https://docs.railway.app/reference/runtime

yep you beat me to it

aight, deploying

same error

is your postgres database in the same project and environment?

yes

and are you now making the migration during runtime?

no, that's a bigger change, i need to work on that

okay then we can try something else for that

is the internal network not available during deployment?

you mean to ask if it is available during build, and the answer to that would be no, but lets try change that

add a service variable RAILWAY_BETA_ENABLE_BUILD_V2=1

more info:

for clarity the v2 runtime and v2 builder are separate systems

sorry, yes, build

it does look like it got past the migration this time, so that appears to have worked

sounds like the BUILD_V2 thing will be a nice change for you guys

and me

the v2 builder and v2 runtime aim fix so many things and bring so many improvements

though.. it seems like it's failing the health check.. retrying 7 times now

what do your deploy logs look like?

deploy log looks like everything is working

service started successfully

what is the reason the health check fails?

Attempt #5 failed with service unavailable. Continuing to retry for 4m44s

Attempt #10 failed with service unavailable. Continuing to retry for 2m28s

etc

what kind of app is this?

no other log message, and nothing showing up in the deploy log

it's a node.js app

rest api

share your build and deploy logs please -

not really comfortable sharing the log in a public forum

is there sensitive information in your logs?

well.. it looks like i'm printing out the database url there.. so yeah

separately, something to fix

with the username and password?

yeah, the full thing

i mean you can edit the logs before you send them

1249550731862610000
1249550732315857009

do you have a PORT service variable set?

no

it defaults to 8080

can you show me your app.listen?

you mean the typescript code?

yes it would be code

why?

im just trying to work through some debug steps

which question could be answered by that?

this is probably what you are most likely interested in:

if you arent interested in debugging this ill have to ask you to revert back to the legacy builder and make your migrations during runtime before starting your app

this is parsing process.env

i'm interested in debugging this, but I'm a bit confused how that could help.. the only thing i can think of is:

• either you think there's a bug in the code

• you are trying to figure out whether the app is really listening to port 8080

• you are wondering if the app is listening to only a particular host (it's listning to 0.0.0.0:8080)

and to those, i think it would be more efficient to ask me

it would perhaps be also easier to test out on the dev environment

since i could just push debug changes etc

ive been doing this for a while and screenshots of code is far more helpful than a written response

i'm sorry, i'm probably being rude, i'm sorry - in my defense it is 3 am here and i mostly just want to go to bed 🙂

thank you

now whats up with the build logs containing just the health check, and the deploy logs containing both the build and deploy logs?

yeah, that was weird to me.. because that's not what they look like in the UI

thats not an edit you did?

it's just how they came out using the bookmarklet

no

and i also had to click the bookmarklet twice.. first time it only downloaded that deploy log, second time it did this weird combo for the other file

i'm assuming it had something to do with the fact that i had the logs open at the time?

possibly i could try to look at the bookmarklet source and debug it

it calls railway's api, it doesnt scrape the web page

weird.. i can try clicking it again

but im not sure whats going on with the failed health check, can you try removing the health check for now

well.. now the bookmarklet says:

do you have a deployment open?

🤦‍♂️

same thing .. buildlogsxxx just has the output of the health check

interesting, ill have to look into this

yeah, if you open the deployment log.. it will give you the actual build log.. concatenated with the output of what should be the deployment log

i'm a bit worried that it will not work if i remove the health check

it might not

i'm fine with the currently running one just staying on for a while

until i can figure out this health check thing

i've got to go to bed - I should just start another thread tomorrow, ok?

this thread is good

ok, thanks a lot for your help ❤️

no problem!

fwiw I fixed the bookmarklet, you just have to delete your old one and drag over the new one -

ah, nice

ok, so i've made the same changes in the dev environment, and it's failing the health check in the same way

but i can push code on to there to debug the issue

i'll start by removing the healthcheck to see if it all still works

--
Is anybody watching this thread?

Anyways, I'm still trying to debug this issue. I haven't made changes related to the active deployment, but it is now all of a sudden not able to connect to the database.

This is the same issue that's been happening on and off lately - and my initial suspicion for what was causing the failed migrations during deployment

Can't reach database server at monorail.proxy.rlwy.net:10861\n\nPlease make sure your database server is running at monorail.proxy.rlwy.net:10861.

This is a way bigger issue than anything related to the deploment process

I opened a new issue for this, since it's not really related to the build process

--
on to the previous issue, it looks like disabling the health check did not adversely affect things - and in the dev environment setting the database env var to the internal one + setting

• runtime: V2

• RAILWAYBETAENABLEBUILDV2=1

with the disabled health check it really does seem like everything is working and the service does get deployed successfully

But i'd really like to have the health check there.. in cases where for some reason I make a mistake and the service doesn't actually work

it was already 3:30am for me when I sent the message for the bookmarklet and I went to bed shortly after that

last night I went around enabling the v2 runtime on some services I had that were still using the legacy runtime, and my umami service started failing the health check with the same reason yours did, did you say your app was a nest app?

side note, I very much appreciate all the testing you've done!!

went to bed before i did more testing last night, but having umami listen on :: instead of 0.0.0.0 allowed the health check to pass and the service to come online without issue

cc @char8 runtime v2 health check oddities

wow, dedication, i like it

did you say your app was a nest app?
not a nest app, but a fastify app - some nest setups use fastify - not sure if that's related, but it would seem strange if what software was running could affect the health check

it bugged me that it didnt work

i feel you

ah, listening on the ipv6 address space instead

that would make sense

i mean it shouldn't make sense, the health check should work for either ipv4 or ipv6

the service is probably not listening to incoming connections on ipv6, let me try this

https://fastify.dev/docs/latest/Reference/Server/#listentextresolver

i'm using 0.0.0.0 which doesn't listen on ipv6 so that would do it

ideally you would be able to listen on either ipv4 or ipv6 and the health check would work regardless

yeah, setting to :: should work, good call

i'm testing that right now

if this turns out to be the solution we are golden

i should have read through this: https://docs.railway.app/reference/private-networking 🙂 could have saved me some hours

no no, you should not need to listen on ipv6 for the health check, thats a bug

but thats the current work around

it literally says:

You will need to bind to a IPv6 port to receive traffic on the private network.

ahh.. i see what you mean

right but the health check is not private networking

riight

so this line that says that private networking is not available during the build phase, that's what will no longer be the case for the new builder?

thats correct

the v2 builder is now a part of the private network

i remember fighing this a bit when i was setting things up initially.. and distinctly remember giving up on the private network for the db connection, probably because of this..

so i might be able to make my db private-network only

thats the goal!

that's super nice

btw i really have been happy with railway in general and it has been really solid for me.. been using it now for two years - i think it's an excellent product and kind of overlooked in the dev community - fly.io etc grabbing all the attention

even though i dont work for railway, i appreciate that

you don't? you're just really helpful?

how do you know about the RAILWAY_BETA_ENABLE_BUILD_V2 flag etc?

and the bookmarklet .. that's just your own thing to made to make it easier to help people?

well i am a mod so i was told many weeks ago about that, but the info has since been made public

and yep the bookmarklet is my own thing, screenshots of code may be helpfull, but screenshots of build logs are rarely helpful

well, who do you work for then?

no one

you're at school?

nope

just doing this for the pleasure of helping people?

yeah!

🙇‍♂️

well, i bought you some trains - let me know if there's ever anything i can help you with

thank you so much!! i appreciate that!!

btw the service was deployed successfully after changing the listening address.. so all problems resolved

thats awsome!

are you now using the private network to run your migrations during build and for regular communication during runtime?

yes

or i'm deploying that to production atm

works on dev

sounds good!

btw the new build thing injects a bunch of extra lines into the logs:

INFO[0007] No files changed in this command, skipping snapshotting.

do you know if there's a way to omit these?

i don't see this when building the docker image locally, so i'm not sure what that is

the v2 builder doesnt use buildx it uses something else entirely, and its a little more verbose, theres no way to omit that info

ok, thanks anyways

and thanks a lot for your help, i'm running on private networking both in the build and in the runtime 👍 both on dev & production

how do i mark this thread as resolved?

thats great, glad the health check issue was a simple fix

thats something a mod can only do, ill go find the best message to mark as the solution

to recap -

enable the v2 builder and the v2 runtime, if your health check fails, have your app listen on the :: host

well the bot is offline so i will mark that as the answer when the bot come back online

i can also summarize:

I was using the public network to run DB migrations during the build process. I started having connectivity issues when doing that. I made my service use the DB's private networking url and enabled the V2 runtime in settings and set the RAILWAY_BETA_ENABLE_BUILD_V2 variable to 1 - this resolved the issue

(there was a bug in the health check, where it was connecting to the service in the ipv6 range, so i had to also enable that for my app - but that was a separate issue)

yes, or your recap, sorry, i started writing before i saw that

yours is better, ill use yours

probably more succinct