cfe555eb-4fc0-417e-b1fe-88b5516418dd

hmm, seems to have something to do with the predeploy command - is it trying to healthcheck the pre-deploy script? that doesn't make much sense if so

ok nope it's actually not

yeah it's literally just seeming like health checks are not working at all?

Health checks failing (service is perfectly healthy)

That is odd, we haven't seen any other reports of this, do you have an MRE we could look into?

I can try to make one

much appreciated

any chance that health checks can be extended to say the actual message that the request failed with?

They already do. The body will be printed out if the application responds, but that isn't even happening here, so there's no message available.

same meme, is there any documentation for what exactly the health check is doing (curl command? some process on the machine doing a fetch?)

just an http request to the path set

where is it done from? what http client?

i'm just trying to figure out what exactly is happening so I can get a better idea of why this could be happening

It would be done from the stacker with go's stdlib http client

side note: railpack really needs to start using precompiled ruby versions instead of building them from source. 2 minutes is crazy

Railpack is OSS 😉

i know, just shouting into the air

i shall do it some day

Can you open an issue on the Railpack GitHub? we have someone staffed to tackle them.

okay i've got a MRE

1408138603430809731

literally just ran rails@7 bootstrap

here's a real world example @ project a8763251-12dc-4d8d-a29d-4849bca82e4b

I made this project just for this so y'all have my full permission to do whatever to it

version without healthchecks is perfectly fine

https://tiny-rails-starter-production.up.railway.app/up

version with healthchecks fails

puma says it's listening so I have no reason to doubt that's the case

but yeah you can either use that project, or use the zip + add RAILS_ENV=production and SECRET_KEY_BASE=literallywhatever and that's enough to repro the issue

can you make the repo public?

yeah one sec

Done

I also added you as a collab in case you wanna try committing or anything

The only reason i said this is because right now it doesn't say what happened at all. I.e. "service unavailable" doesn't tell me if the request got to the server and timed out, or if it couldn't establish a connection, or if the connection was refused, so on

My original hypothesis was that the DNS rebinding protection built into rails was kicking in and not allowing it with no host header, but if the application can't be reached at all, i don't think that is the case

I can reproduce the issue, but the root cause isn't clear to me at this time. You're the first user to report this specific health check problem - we haven't seen other reports of this type of failure beyond some configuration misunderstandings.

I've created a ticket to track this and will add any similar reports we receive. Once we have a few more reports to help identify patterns, we'll be able to investigate more thoroughly.

Update: on puma it doesn't respond, but on Unicorn it 301's

would you be able to check if the raw Go HTTP request is setting any of the following headers (and if so, to which values?)

Location
X-Forwarded-For
Host

These are the only headers that are set -

{
  "Accept-Encoding": [
    "gzip"
  ],
  "Connection": [
    "close"
  ],
  "User-Agent": [
    "RailwayHealthCheck/1.0"
  ]
}

hmm interesting. I think what might be happening is that Rack sees that no intended host is set and is trying to redirect to https://xyz.up.railway.app/up

I managed to come up with a Caddy setup that mitigates the problem, but still feels less than ideal

I had this same issue. I managed to fix it by disabling ssl redirects for the /up endpoint

This should he documented by Railway as a known platform bug