Reverse proxy backend is unresponsive
coderjoshdk
TRIALOP

a year ago

I have a revers proxy through Caddy, that is connecting my frontend and backend. It seems to be working fine for my frontend. But is failing on the backend. The backend is a django service.

The proxy is defined by

# Proxy file for connecting services through railway.app
{
    debug
    admin off # theres no need for the admin api in railway's environment
    persist_config off # storage isn't persistent anyway
    auto_https off # railway handles https for us, this could in some cases cause issues if left enabled
    log {
        format json # set runtime log format to json mode
    }
    # server options
    servers {
        trusted_proxies static private_ranges # trust railway's proxy
    }
}

:{$PORT} {
    log {
        format json # set access log format to json mode
    }

    encode zstd gzip

    handle_path /ingest* {
        rewrite * {path}
        reverse_proxy https://us.i.posthog.com:443 {
          header_up Host us.i.posthog.com
          header_down -Access-Control-Allow-Origin
        }
    }

    @django path /admin* /api* /accounts* /static/*
    handle @django {
        reverse_proxy {$BACKEND_HOST}
    }


    handle {
    reverse_proxy {$FRONTEND_HOST}
    }
}

I am not getting any responses from django. But I am hitting it. I see some network activity. But no response
I put the service in debug mode, but yet, it still is not responding with anything. I am getting 502 errors.

The django service is being ran with python [manage.py](manage.py) runserver 0.0.0.0:${PORT:-8000}.
Unsure what to do / change to get this running

46 Replies

coderjoshdk
TRIALOP

a year ago

40943c2e-d9a8-4f3c-b414-73ea73eccd8f


coderjoshdk
TRIALOP

a year ago

There are no real logs from django (normally it would respond with what endpoints are hit in debug mode)

1251227678715940900


coderjoshdk
TRIALOP

a year ago

Requests to the backend, end up with an error

1251227929795498000


coderjoshdk
TRIALOP

a year ago

And frontend requests make it through no problem

1251228337276452900


coderjoshdk
TRIALOP

a year ago

My env for the reverse proxy:

BACKEND_HOST=${{Django.RAILWAY_PRIVATE_DOMAIN}}:${{Django.PORT}}
FRONTEND_HOST=${{Frontend.RAILWAY_PRIVATE_DOMAIN}}:${{Frontend.PORT}}
PORT=3000

1251228668055785500


brody
EMPLOYEE

a year ago

runserver starts a development server, even if you are only testing you still dont want to be using it on Railway, instead use gunicorn like this gunicorn mysite.wsgi -b [::]:$PORT (replacing mysite with the name of the folder that contains wsgi.py)


coderjoshdk
TRIALOP

a year ago

I was originally doing gunicorn BBWare.wsgi -b [::]:${PORT:-8000} but was still getting the same issue. But yea, let me try with that but debug still on (I have my deployment select between the two based on if debug is on)


brody
EMPLOYEE

a year ago

little context, the private network is ipv6 only so your app would need to listen on ipv6


coderjoshdk
TRIALOP

a year ago

For the run server, yea, I guess it wouldn't be. But the gunicorn should be correct for doing that? Unless there is an extra setting I need to change in the django settings?


brody
EMPLOYEE

a year ago

what you have should have worked, maybe you are overwriting the start command somewhere else


coderjoshdk
TRIALOP

a year ago

Also, can confirm, just having the gunicorn config file in existence does make it get used. Even if the cli isn't pointing to it. 😄
Going to see though real quick if the gunicorn + debug produces anything


brody
EMPLOYEE

a year ago

just set the workers to a fixed number


coderjoshdk
TRIALOP

a year ago

No yea, for sure. Was just circling back on that earlier topic


coderjoshdk
TRIALOP

a year ago

Yep, didn't change anything


brody
EMPLOYEE

a year ago

show me the depoloy logs for your django app please


coderjoshdk
TRIALOP

a year ago

Still get this when I go directly to a backend url

1251231756065968000


coderjoshdk
TRIALOP

a year ago

1251231801985470500


coderjoshdk
TRIALOP

a year ago

You reached the start of the rangeJun 14, 2024 1:45 PM
Starting Container
+ python manage.py migrate --noinput
  Apply all migrations: account, admin, auth, authtoken, breaker, contenttypes, permissions, purchase, sale, sessions, sites, socialaccount, stock
+ 
python manage.py collectstatic --noinput

+ gunicorn BBWare.wsgi -b [::]:3000
[2024-06-14 17:46:54 +0000] [30] [INFO] Starting gunicorn 22.0.0
[2024-06-14 17:46:54 +0000] [30] [INFO] Listening at: http://[::]:3000 (30)
[2024-06-14 17:46:54 +0000] [30] [INFO] Using worker: sync
[2024-06-14 17:46:54 +0000] [31] [INFO] Booting worker with pid: 31

brody
EMPLOYEE

a year ago

can you send the link to the proxy



coderjoshdk
TRIALOP

a year ago

Oh lol, I have a typo in the desc :p


brody
EMPLOYEE

a year ago

looks like it does make it to the django backend

1251232524949131500


coderjoshdk
TRIALOP

a year ago

You can see the pure frontend working if you go to https://website-reverse-proxy-production.up.railway.app/sign-in


coderjoshdk
TRIALOP

a year ago

Works for me. 500s becuase can't hit the auth endpoint becuase backend isn't working.

1251232699989885000


brody
EMPLOYEE

a year ago

can you add --forwarded-allow-ips="*" to your gunicorn start command


coderjoshdk
TRIALOP

a year ago

Well the "good" news is that the requests are now responding with 301 and not 502. But not actually sure if that is better.
It is redirecting to nothing? Not sure.

1251233147769852000


coderjoshdk
TRIALOP

a year ago

Same result


coderjoshdk
TRIALOP

a year ago

1251235604885147600


brody
EMPLOYEE

a year ago

django and or gunicorn is attempting to redirect for some reason, do you have any idea why?


coderjoshdk
TRIALOP

a year ago

The only thing I could think of is that I force things to be HTTPS:

SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")
SECURE_SSL_REDIRECT = True
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True

This would normally work no issue. At least it has with Caddy. But maybe becuase these Caddyfiles have https off, it is messing with things? Idk how railway is taking care of https?


brody
EMPLOYEE

a year ago

yeah dont do that


brody
EMPLOYEE

a year ago

railway does https on its own proxy for you, you should run your app in http with no notion of https


coderjoshdk
TRIALOP

a year ago

What part should I be getting rid of?
But also, there really should be a way to make this work without removing that stuff


brody
EMPLOYEE

a year ago

well railway sets a X-Forwarded-Proto header, so you need to have your app accept that value


coderjoshdk
TRIALOP

a year ago

It feels very wrong to remove those protections on the django level


brody
EMPLOYEE

a year ago

your app on railway can only ever be accessed over https so theres no issues removing them


coderjoshdk
TRIALOP

a year ago

I mean sure. But it is more like "extra precaution".


brody
EMPLOYEE

a year ago

i promise you its not needed, there is no way to access a railway hosted app over http


coderjoshdk
TRIALOP

a year ago

This is going to make for quite the blog post. Eventually. Long list of backlog things to get to before I touch this.


brody
EMPLOYEE

a year ago

well you have two options

  • railway sets a X-Forwarded-Proto header, have your app accept that value

  • remove anything that tries to redirect to https


coderjoshdk
TRIALOP

a year ago


coderjoshdk
TRIALOP

a year ago

It is working


coderjoshdk
TRIALOP

a year ago

I officially can transfer everything to railway.

….

That is going to be a bit of a pain. But it being possible is the first step. Now I just need to also do all the other silly things like add monitoring and add db backup cron stuff. That will be fun


brody
EMPLOYEE

a year ago

use the railway template for db backups!


coderjoshdk
TRIALOP

a year ago

Yea, I will. Thanks


brody
EMPLOYEE

a year ago

no problem!


Loading...