Reverse proxy backend is unresponsive
coderjoshdk
TRIALOP

2 years ago

I have a revers proxy through Caddy, that is connecting my frontend and backend. It seems to be working fine for my frontend. But is failing on the backend. The backend is a django service.

The proxy is defined by

# Proxy file for connecting services through railway.app
{
    debug
    admin off # theres no need for the admin api in railway's environment
    persist_config off # storage isn't persistent anyway
    auto_https off # railway handles https for us, this could in some cases cause issues if left enabled
    log {
        format json # set runtime log format to json mode
    }
    # server options
    servers {
        trusted_proxies static private_ranges # trust railway's proxy
    }
}

:{$PORT} {
    log {
        format json # set access log format to json mode
    }

    encode zstd gzip

    handle_path /ingest* {
        rewrite * {path}
        reverse_proxy https://us.i.posthog.com:443 {
          header_up Host us.i.posthog.com
          header_down -Access-Control-Allow-Origin
        }
    }

    @django path /admin* /api* /accounts* /static/*
    handle @django {
        reverse_proxy {$BACKEND_HOST}
    }


    handle {
    reverse_proxy {$FRONTEND_HOST}
    }
}

I am not getting any responses from django. But I am hitting it. I see some network activity. But no response
I put the service in debug mode, but yet, it still is not responding with anything. I am getting 502 errors.

The django service is being ran with python [manage.py](manage.py) runserver 0.0.0.0:${PORT:-8000}.
Unsure what to do / change to get this running

46 Replies

coderjoshdk
TRIALOP

2 years ago

40943c2e-d9a8-4f3c-b414-73ea73eccd8f


coderjoshdk
TRIALOP

2 years ago

There are no real logs from django (normally it would respond with what endpoints are hit in debug mode)

1251227678715940900


coderjoshdk
TRIALOP

2 years ago

Requests to the backend, end up with an error

1251227929795498000


coderjoshdk
TRIALOP

2 years ago

And frontend requests make it through no problem

1251228337276452900


coderjoshdk
TRIALOP

2 years ago

My env for the reverse proxy:

BACKEND_HOST=${{Django.RAILWAY_PRIVATE_DOMAIN}}:${{Django.PORT}}
FRONTEND_HOST=${{Frontend.RAILWAY_PRIVATE_DOMAIN}}:${{Frontend.PORT}}
PORT=3000

1251228668055785500


2 years ago

runserver starts a development server, even if you are only testing you still dont want to be using it on Railway, instead use gunicorn like this gunicorn mysite.wsgi -b [::]:$PORT (replacing mysite with the name of the folder that contains wsgi.py)


coderjoshdk
TRIALOP

2 years ago

I was originally doing gunicorn BBWare.wsgi -b [::]:${PORT:-8000} but was still getting the same issue. But yea, let me try with that but debug still on (I have my deployment select between the two based on if debug is on)


2 years ago

little context, the private network is ipv6 only so your app would need to listen on ipv6


coderjoshdk
TRIALOP

2 years ago

For the run server, yea, I guess it wouldn't be. But the gunicorn should be correct for doing that? Unless there is an extra setting I need to change in the django settings?


2 years ago

what you have should have worked, maybe you are overwriting the start command somewhere else


coderjoshdk
TRIALOP

2 years ago

Also, can confirm, just having the gunicorn config file in existence does make it get used. Even if the cli isn't pointing to it. 😄
Going to see though real quick if the gunicorn + debug produces anything


2 years ago

just set the workers to a fixed number


coderjoshdk
TRIALOP

2 years ago

No yea, for sure. Was just circling back on that earlier topic


coderjoshdk
TRIALOP

2 years ago

Yep, didn't change anything


2 years ago

show me the depoloy logs for your django app please


coderjoshdk
TRIALOP

2 years ago

Still get this when I go directly to a backend url

1251231756065968000


coderjoshdk
TRIALOP

2 years ago

1251231801985470500


coderjoshdk
TRIALOP

2 years ago

You reached the start of the rangeJun 14, 2024 1:45 PM
Starting Container
+ python manage.py migrate --noinput
  Apply all migrations: account, admin, auth, authtoken, breaker, contenttypes, permissions, purchase, sale, sessions, sites, socialaccount, stock
+ 
python manage.py collectstatic --noinput

+ gunicorn BBWare.wsgi -b [::]:3000
[2024-06-14 17:46:54 +0000] [30] [INFO] Starting gunicorn 22.0.0
[2024-06-14 17:46:54 +0000] [30] [INFO] Listening at: http://[::]:3000 (30)
[2024-06-14 17:46:54 +0000] [30] [INFO] Using worker: sync
[2024-06-14 17:46:54 +0000] [31] [INFO] Booting worker with pid: 31

2 years ago

can you send the link to the proxy



coderjoshdk
TRIALOP

2 years ago

Oh lol, I have a typo in the desc :p


2 years ago

looks like it does make it to the django backend

1251232524949131500


coderjoshdk
TRIALOP

2 years ago

You can see the pure frontend working if you go to https://website-reverse-proxy-production.up.railway.app/sign-in


coderjoshdk
TRIALOP

2 years ago

Works for me. 500s becuase can't hit the auth endpoint becuase backend isn't working.

1251232699989885000


2 years ago

can you add --forwarded-allow-ips="*" to your gunicorn start command


coderjoshdk
TRIALOP

2 years ago

Well the "good" news is that the requests are now responding with 301 and not 502. But not actually sure if that is better.
It is redirecting to nothing? Not sure.

1251233147769852000


coderjoshdk
TRIALOP

2 years ago

Same result


coderjoshdk
TRIALOP

2 years ago

1251235604885147600


2 years ago

django and or gunicorn is attempting to redirect for some reason, do you have any idea why?


coderjoshdk
TRIALOP

2 years ago

The only thing I could think of is that I force things to be HTTPS:

SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")
SECURE_SSL_REDIRECT = True
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True

This would normally work no issue. At least it has with Caddy. But maybe becuase these Caddyfiles have https off, it is messing with things? Idk how railway is taking care of https?


2 years ago

yeah dont do that


2 years ago

railway does https on its own proxy for you, you should run your app in http with no notion of https


coderjoshdk
TRIALOP

2 years ago

What part should I be getting rid of?
But also, there really should be a way to make this work without removing that stuff


2 years ago

well railway sets a X-Forwarded-Proto header, so you need to have your app accept that value


coderjoshdk
TRIALOP

2 years ago

It feels very wrong to remove those protections on the django level


2 years ago

your app on railway can only ever be accessed over https so theres no issues removing them


coderjoshdk
TRIALOP

2 years ago

I mean sure. But it is more like "extra precaution".


2 years ago

i promise you its not needed, there is no way to access a railway hosted app over http


coderjoshdk
TRIALOP

2 years ago

This is going to make for quite the blog post. Eventually. Long list of backlog things to get to before I touch this.


2 years ago

well you have two options

  • railway sets a X-Forwarded-Proto header, have your app accept that value

  • remove anything that tries to redirect to https


coderjoshdk
TRIALOP

2 years ago


coderjoshdk
TRIALOP

2 years ago

It is working


coderjoshdk
TRIALOP

2 years ago

I officially can transfer everything to railway.

….

That is going to be a bit of a pain. But it being possible is the first step. Now I just need to also do all the other silly things like add monitoring and add db backup cron stuff. That will be fun


2 years ago

use the railway template for db backups!


coderjoshdk
TRIALOP

2 years ago

Yea, I will. Thanks


2 years ago

no problem!


Loading...