FastAPI service health-check fails in IPV6

mneira10
PRO

2 years ago

I'm running a fastAPI service in railway. I setup a /healthcheck endpoint. It worked well when I exposed it publicly and bound it to 0.0.0.0:8000 (I've also specified the PORT=8000 env var).

However, I want an nginx reverse proxy to be the only publicly exposed service since that backend service will work alongside a separate nextjs frontend service (that won't be publicly exposed either).

In trying to communicate the service privately though, I changed the host to :: (same 8000 port) as per the private networking docs: https://docs.railway.app/guides/private-networking#listen-on-ipv6 and railway can't hit the /healthcheck endpoint anymore and so the build logs say the service never came online. Any thoughts?

0 Replies

mneira10
PRO

2 years ago

Project ID: 2cc7bfdb-9bfd-4c76-8d34-28e269d7406f


2 years ago

Uvicorn does not support dual stack binding (IPv6 and IPv4) from the CLI, so while that start command will work to enable access from within the private network, this prevents you from accessing the app from the public domain if needed, I recommend using Hypercorn instead


mneira10
PRO

2 years ago

I don't need to access it from the public domain though


2 years ago

health checks use ipv4


2 years ago

an example hypercorn start command would be hypercorn main:app --bind [::]:$PORT


mneira10
PRO

2 years ago

Whoa


mneira10
PRO

2 years ago

health checks use ipv4

Any chance you could mention that in the docs?


2 years ago

of course you'd want to set a fixed PORT service variable


2 years ago

will bring that up to the applicable person


mneira10
PRO

2 years ago

Thanks!

So if I disable the healthchecks all should work well, correct?


2 years ago

yeah but i wouldnt disable the health check, i would use hypercorn


mneira10
PRO

2 years ago

Noted!

Our production systems are battle tested on uvicorn. I can't justify switching over given that railway only has healthchecks over ipv4 though. It would also be a non trivial change in the codebase


2 years ago

thats fair, but without a health check railway wont know when your app is able to handle traffic


mneira10
PRO

2 years ago

Yup, understood. I'm running an MVP to test out the viability of switching our k8s cluster over to railway. If we do decide to pull the trigger on the migration we would consider switching to accomodate for the healthchecks


2 years ago

sounds good, and with the upcoming runtime im sure adding ipv6 capabilities to the health check would be already done by default or easy enough to implement


mneira10
PRO

2 years ago

Nice! That's good to hear!

Do you have a rough ETA for when those changes would land?


2 years ago

runtime v2 is pre-alpha right now, so i dont have any real eta to give you, and in fact the v2 runtime doesnt even support any health checks right now


mneira10
PRO

2 years ago

Coming back to this, my nginx service isn't able to communicate with the fastapi service


2 years ago

lets see the nginx.conf


mneira10
PRO

2 years ago

I've disabled the healthcheck and have nginx pointed to [http://upcodes-backend.railway.internal:8000](http://upcodes-backend.railway.internal:8000)


mneira10
PRO

2 years ago

server {
    listen 80;

    location /v0 {
        proxy_pass http://upcodes-backend.railway.internal:8000;
    }


    location / {
        proxy_pass http://upcodes-frontend.railway.internal;
    }
}

2 years ago

nginx is not ideal for this, but i assume you dont want to switch to caddy?


mneira10
PRO

2 years ago

Is there are reason why nginx wouldn't work? Happy to switch if that helps/is easier


mneira10
PRO

2 years ago

nginx is a typical reverse proxy for these things


2 years ago

nginx tries to resolve the domains at first start, this is not ideal for two resaons, the private network is not available at first start, the services do not have static ip(v6) addresses


2 years ago

of course nginx can be configured to not do these things, but caddy doesnt do them by default


mneira10
PRO

2 years ago

Interesting. I'll give caddy a shot. Looks like you configured the template!


2 years ago

yep that template covers your use case, though mine calls the backend endpoint /api but thats a simple change


mneira10
PRO

2 years ago

Cool! What whould that change look like? I'm not familiar with caddy


2 years ago

/api/* -> /v0/*


mneira10
PRO

2 years ago

Right but where? Ah nvm I see the Caddyfile in the repo.


2 years ago

yeah you would need to deploy the template and then eject from it


mneira10
PRO

2 years ago

Cool. Yeah, just copy pasted your code into my repo and deployed from there


2 years ago

sounds good


FastAPI service health-check fails in IPV6 - Railway Help Station