Additional:

Another API is called during build that returns an extremely small response (JSON with ~5 entries) and this works fine.

do you have an endpoint i could curl to reproduce this issue?

curl -s -X GET 'https://share-a-podcast-be-prod.up.railway.app/test-new-endpoint/' | jq

is that supported to work?

ha - yes - what are you seeing?

json

yes

can i have an endpoint that reproduces the issue, im not sure what i should be doing with an enpoint that works fine

yes… I mean that's the issue

The endpoint works fine, every time, when called in curl/postman.

But in build it gives the output I shared in the first post. This is what I can't understand.

during build of your FE?

yep

are you able to see my build logs?

how is this url being called?

one sec, getting the code

1283461441025740820

forgive the terrible loggin

did you say you have disabled GZIP on the BE and still got the same garbage response?

yep - I thought the logs look like printed gzip

so I removed GZipMiddleware, same issue

is GZIP currently disabled (removed)?

no i renabled but can disable it again

lets leave it for now

just trying to parse through these logs

good luck

LMK if I can help

i dont see content-encoding = gzip anywhere in these FE build logs, does response.headers.entries() strip that out?

take a look at the backend logs - I'm logging the headers on inbound requests:

right but content-encoding = gzip should be returned as a response header

yeah agree it's not there

but why?

curl -I "https://share-a-podcast-be-prod.up.railway.app/test-new-endpoint/"
HTTP/1.1 200 OK
Allow: GET, HEAD, OPTIONS
Content-Type: application/json
Cross-Origin-Opener-Policy: same-origin
Date: Wed, 11 Sep 2024 16:31:23 GMT
Referrer-Policy: same-origin
Server: railway-edge
Vary: Accept-Encoding, origin, Cookie
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Railway-Request-Id: nmUVBEMHRwKJVioSmvMM1Q_2020806880

misbehaving middleware?

but you said it was the same with the middleware removed?

either way, can you disable / remove the gzip middleware for now?

dude sorry, back at keys

disabling now

for context, curl was showing that it got the gzip response header -

C:\Users\Brody>curl --location -I "https://share-a-podcast-be-prod.up.railway.app/test-new-endpoint/" --header "Accept-Encoding: gzip, deflate, br" --header "Content-Type: application/json"
HTTP/1.1 200 OK
Allow: GET, HEAD, OPTIONS
Content-Encoding: gzip
Content-Length: 826
Content-Type: application/json
Cross-Origin-Opener-Policy: same-origin
Date: Wed, 11 Sep 2024 16:40:05 GMT
Referrer-Policy: same-origin
Server: railway-edge
Vary: Accept-Encoding, origin, Cookie
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Railway-Request-Id: 1bO2oXJ6Sn-rCTujPm4rCw_882434190

this is what led me to think it might be something to do with the proxy

communicating with the end point directly works

the proxy doesnt touch any of these headers afaik

https://help.railway.app/questions/new-proxy-stripping-accept-encoding-head-56fb16a0

aware this is now solved, but wondering if related?

well besides the headers it does touch, you know what i mean lol

btw, I noticed that when I deploy the backend, I have to commit/push to force a rebuild on my frontend, rather than just trigging a deploy from last commit

Hello all, we have pushed a change to our proxy, we no longer strip out that header!

what happens when you do deploy from last commit?

I know, but just wondering if could be a related thing?

I'll show you

at this time, i dont think so

backend is deployed

frontend deploying now with deploy from last commit

i can confirm with postman the BE response is no longer gzipped, no encoding header and resonse body is larger

agreed, see the same

interesting

ok - frontend built correctly with trigger last commit

but same issue

lets use the private domain then

in build?

yeah

oooh

have you just opted me in

fine if you have

yeah, just trying to move fast here, i will ask next time

no don't worry, just wonderd

didn't remember doing it and I've been battling this for 1.5 days… starting to forget what life was like before this bug

I think it's worked

either way - being able to use private urls in build is a win

shall I renable gzip on the BE?

oh, it worked…

i was kinda hoping it wouldnt

yeah, as is we haven't learnt anything

exactly

just that the new builder works for some reason

enabling gzip

hold on

not yet please

oops

aborted the build

oh wait, this is a production site, can you afford to keep this offline as we debug?

yes

all my user will be ok with it

are you sure? we could spin up a dev env in railway

yes don't worry

alright, sounds good, but yeah lets keep gzip off for now, it eliminates variables

anything I can do to help debug?

I would like to remove the hard-coded response, and replace with DB query

ok for me to try?

I'll leave gzip off

yeah thats fine

that's deployed

redeploying FE

well my fridge doesn't cool anymore, I gotta go AFK while I transfer everything to another fridge outside

good luck

I'm going to try renabling gzip while you're fridging

FYI database connection worked

they didn't before?

ah, it was just one of many things I've been trying to debug the issue - hardcoding the json response in my backend

was looking for special characters/fixing the size of the repsonse, etc

gzip works

okay I have some more things I'd like to try when I'm back

where are we on the fridge outage, Brody? P1? Users affected? Is there an incident channel we should join?

there's an incident? I thought we left off in a working state with the new builder?

was a joke about your fridge, ignore

anyway, new builder working - I'm back to coding, thanks for your help

oh I thought you were serious 😅

rarely

I thought you were like, screw your fridge, help me with this gzip stuff

haha

would never priotise work above food

I'll be back at the computer as soon as I finish eating and moving the food to the other fridge

alright, i am back