2dabc326-678a-4f5e-8e12-6a40c656ef91

how are you starting your server?

funny you should ask, you provided the settings for me… NEW_RELIC_CONFIG_FILE=newrelic.ini NEW_RELIC_ENVIRONMENT=$ENVIRONMENT newrelic-admin run-program uvicorn app.main:app --host 0.0.0.0 --port $PORT --workers $UVICORN_WORKERS --loop uvloop --http httptools --limit-concurrency $UVICORN_LIMIT_CONCURRENCY --backlog $UVICORN_BACKLOG

UVICORNWORKERS=4 UVICORNLIMITCONCURRENCY=256 UVICORNBACKLOG=512

should we be setting --timeout-keep-alive (default 5 seconds)

the 502 im seeing happened in 57 milliseconds, so im not sure if it has to do with keep alives

something at the application layer is closing the connection, unfortunately i dont see anything sus here

yeah, today I only saw 4 502's…

i think its time to add more tracing into your app to help you find out whats going wrong

Yeah, I've added trace Ids and such, but nothing to trace on the app sinc the request never makes it.

There were more 502's earlier this week..

We instrumented from the caller (NextJS) as far as we can. It's a blackbox between Cloudflare and Railway now (above the app layer). The app never gets the receives the request. (We log every reuqest)

i assume the logging will only actually log if the request completes, but if something is going wrong at your application level to cause this, there may be no logs

Well we log all exceptions and have sentry/newrelic instrumentation as well

Do you have access or load balancer logs?

We are logging everything that reaches our instance.

our http logs are the same as we give you

Gotcha .

did you know you can expand the http logs to see more information

Are you talking about in railway logs or cloudflare ?

our http logs

Yup. We are ingesting them to newrelic. Let me check the railway console, maybe newrelic is dropping some logs.

how are you ingesting our http logs to new relic?

Sorry for the delay. We are using NewRelic to grab the output from stdout.

Oh, and I didn't realize you can open the http logs.. there's not much additional info:

you are looking at your logs, not our logs, try clicking the http logs tab

oh, i didn't realize there were separate HTTP logs…

nice.

Is there a way to filter by status?

yeah put a @ symbol into the search and you will get an option to filter by http status

Neat. Ok, I can see the 502 in the http logs. Yeah, there's a strange… responseDetails: "failed to forward request to upstream: connection closed unexpectedly"

And a missing timestamp for a log in the Deploy logs for that time.

yeah that means your logging and error handling did not capture whatever error caused that request to close prematurely

Yup. Thanks for the http logs. I didn’t realize those existed in the other tab. And looks like I can’t log drain those, only my app logs.

Is there any way to export the http logs other than manual exports?

no there arent, sorry

No worries. I read the feature request pages. Are there any policies against scraping the logs?

intresting. did you fix it?

you would want to use the API to grab them

It started happening when we increased workers to 16. We reduced to 4 and are not seeing any 502s yet. Possibly some connection keep alive issue occurring at the starlette connection handling before it’s handed off to our app.

i maybe off my base but isn't 16 workers alot tho. python is notoriously ram hungry and a worker can go upto 500mb if not optimized. so you could be saturating the 8gb ram railway has per service

that could explain y connection closes unexpectedly if it's saturating the ram

I thought each service has 32 GB of ram and scales appropriately? We were running about 300Mb per worker… here's our memory usage before and after…

Traffic was low so maybe workers were idling out? Not quite sure. I think there's something with keep alive settings between Railway HTTP load balancer and FastAPI Uvicorn, but not sure what the settings are for the HTTP load balancer. Are there recommend network settings?

the 502 I saw happened in 50 ms so timeouts do not come into play with such short requests

The other timeout was 0 ms, so I don't think it's timeout (like the request took too long), but possible connection re-use and timeout while managing connections in a pool/load balancer.

I totally agree with @Brody it’s something with our FastApi service. I’ll continue to monitor the workers and 502s. Thanks for all the help!

https://stackoverflow.com/questions/79485999/gunicorn-uvicorn-extremely-rare-502-errors

do you have max_requests?

No, we use only max concurrency and backlog.

FYI. Following up with FastAPI folks

Quick update here… switched to Gunicorn with Uvicorn workers and had the same issue, switched to Granian and occurrences are much less now (once a day). I think at this point it's something to do with NewRelic monitoring. Closing out this thread. Thanks all!