Reverse proxy backend is unresponsive

coderjoshdk
TRIAL

a year ago

I have a revers proxy through Caddy, that is connecting my frontend and backend. It seems to be working fine for my frontend. But is failing on the backend. The backend is a django service.

The proxy is defined by

# Proxy file for connecting services through railway.app
{
    debug
    admin off # theres no need for the admin api in railway's environment
    persist_config off # storage isn't persistent anyway
    auto_https off # railway handles https for us, this could in some cases cause issues if left enabled
    log {
        format json # set runtime log format to json mode
    }
    # server options
    servers {
        trusted_proxies static private_ranges # trust railway's proxy
    }
}

:{$PORT} {
    log {
        format json # set access log format to json mode
    }

    encode zstd gzip

    handle_path /ingest* {
        rewrite * {path}
        reverse_proxy https://us.i.posthog.com:443 {
          header_up Host us.i.posthog.com
          header_down -Access-Control-Allow-Origin
        }
    }

    @django path /admin* /api* /accounts* /static/*
    handle @django {
        reverse_proxy {$BACKEND_HOST}
    }


    handle {
    reverse_proxy {$FRONTEND_HOST}
    }
}

I am not getting any responses from django. But I am hitting it. I see some network activity. But no response
I put the service in debug mode, but yet, it still is not responding with anything. I am getting 502 errors.

The django service is being ran with python [manage.py](manage.py) runserver 0.0.0.0:${PORT:-8000}.
Unsure what to do / change to get this running

0 Replies

coderjoshdk
TRIAL

a year ago

40943c2e-d9a8-4f3c-b414-73ea73eccd8f


coderjoshdk
TRIAL

a year ago

There are no real logs from django (normally it would respond with what endpoints are hit in debug mode)

1251227678715940900


coderjoshdk
TRIAL

a year ago

Requests to the backend, end up with an error

1251227929795498000


coderjoshdk
TRIAL

a year ago

And frontend requests make it through no problem

1251228337276452900


coderjoshdk
TRIAL

a year ago

My env for the reverse proxy:

BACKEND_HOST=${{Django.RAILWAY_PRIVATE_DOMAIN}}:${{Django.PORT}}
FRONTEND_HOST=${{Frontend.RAILWAY_PRIVATE_DOMAIN}}:${{Frontend.PORT}}
PORT=3000

1251228668055785500


a year ago

runserver starts a development server, even if you are only testing you still dont want to be using it on Railway, instead use gunicorn like this gunicorn mysite.wsgi -b [::]:$PORT (replacing mysite with the name of the folder that contains wsgi.py)


coderjoshdk
TRIAL

a year ago

I was originally doing gunicorn BBWare.wsgi -b [::]:${PORT:-8000} but was still getting the same issue. But yea, let me try with that but debug still on (I have my deployment select between the two based on if debug is on)


a year ago

little context, the private network is ipv6 only so your app would need to listen on ipv6


coderjoshdk
TRIAL

a year ago

For the run server, yea, I guess it wouldn't be. But the gunicorn should be correct for doing that? Unless there is an extra setting I need to change in the django settings?


a year ago

what you have should have worked, maybe you are overwriting the start command somewhere else


coderjoshdk
TRIAL

a year ago

Also, can confirm, just having the gunicorn config file in existence does make it get used. Even if the cli isn't pointing to it. 😄
Going to see though real quick if the gunicorn + debug produces anything


a year ago

just set the workers to a fixed number


coderjoshdk
TRIAL

a year ago

No yea, for sure. Was just circling back on that earlier topic


coderjoshdk
TRIAL

a year ago

Yep, didn't change anything


a year ago

show me the depoloy logs for your django app please


coderjoshdk
TRIAL

a year ago

Still get this when I go directly to a backend url

1251231756065968000


coderjoshdk
TRIAL

a year ago

1251231801985470500


coderjoshdk
TRIAL

a year ago

You reached the start of the rangeJun 14, 2024 1:45 PM
Starting Container
+ python manage.py migrate --noinput
  Apply all migrations: account, admin, auth, authtoken, breaker, contenttypes, permissions, purchase, sale, sessions, sites, socialaccount, stock
+ 
python manage.py collectstatic --noinput

+ gunicorn BBWare.wsgi -b [::]:3000
[2024-06-14 17:46:54 +0000] [30] [INFO] Starting gunicorn 22.0.0
[2024-06-14 17:46:54 +0000] [30] [INFO] Listening at: http://[::]:3000 (30)
[2024-06-14 17:46:54 +0000] [30] [INFO] Using worker: sync
[2024-06-14 17:46:54 +0000] [31] [INFO] Booting worker with pid: 31

a year ago

can you send the link to the proxy



coderjoshdk
TRIAL

a year ago

Oh lol, I have a typo in the desc :p


a year ago

looks like it does make it to the django backend

1251232524949131500


coderjoshdk
TRIAL

a year ago

You can see the pure frontend working if you go to https://website-reverse-proxy-production.up.railway.app/sign-in


coderjoshdk
TRIAL

a year ago

Works for me. 500s becuase can't hit the auth endpoint becuase backend isn't working.

1251232699989885000


a year ago

can you add --forwarded-allow-ips="*" to your gunicorn start command


coderjoshdk
TRIAL

a year ago

Well the "good" news is that the requests are now responding with 301 and not 502. But not actually sure if that is better.
It is redirecting to nothing? Not sure.

1251233147769852000


coderjoshdk
TRIAL

a year ago

Same result


coderjoshdk
TRIAL

a year ago

1251235604885147600


a year ago

django and or gunicorn is attempting to redirect for some reason, do you have any idea why?


coderjoshdk
TRIAL

a year ago

The only thing I could think of is that I force things to be HTTPS:

SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")
SECURE_SSL_REDIRECT = True
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True

This would normally work no issue. At least it has with Caddy. But maybe becuase these Caddyfiles have https off, it is messing with things? Idk how railway is taking care of https?


a year ago

yeah dont do that


a year ago

railway does https on its own proxy for you, you should run your app in http with no notion of https


coderjoshdk
TRIAL

a year ago

What part should I be getting rid of?
But also, there really should be a way to make this work without removing that stuff


a year ago

well railway sets a X-Forwarded-Proto header, so you need to have your app accept that value


coderjoshdk
TRIAL

a year ago

It feels very wrong to remove those protections on the django level


a year ago

your app on railway can only ever be accessed over https so theres no issues removing them


coderjoshdk
TRIAL

a year ago

I mean sure. But it is more like "extra precaution".


a year ago

i promise you its not needed, there is no way to access a railway hosted app over http


coderjoshdk
TRIAL

a year ago

This is going to make for quite the blog post. Eventually. Long list of backlog things to get to before I touch this.


a year ago

well you have two options

  • railway sets a X-Forwarded-Proto header, have your app accept that value

  • remove anything that tries to redirect to https


coderjoshdk
TRIAL

a year ago


coderjoshdk
TRIAL

a year ago

It is working


coderjoshdk
TRIAL

a year ago

I officially can transfer everything to railway.

….

That is going to be a bit of a pain. But it being possible is the first step. Now I just need to also do all the other silly things like add monitoring and add db backup cron stuff. That will be fun


a year ago

use the railway template for db backups!


coderjoshdk
TRIAL

a year ago

Yea, I will. Thanks


a year ago

no problem!