API server keeps crashing
yaniv297
PROOP

23 days ago

So I've been using railway with a supabase DB for my app for months, i was the only user, it worked perfectly.
In the last few days I've started letting a few more users in - and ever since it simply crashes all the time. Even a load of 3-4 users is enough to crash it. The frontend works fine but every API call simply doesn't respond. After a while I get 502 errors.

I've upgraded both Railway and Supabase to Pro tier (even though my usage numbers were still waaay within the Hobby range) and still no improvement. Plus I've done a lot to fix and optimize my code.

Now it happens again, my Railway api server just randomly (and after heavier load) decides to crash. Here's the logs:
INFO: Shutting down

INFO: Waiting for application shutdown.

INFO: Application shutdown complete.

INFO: Finished server process [1]

I've had a few other crashes: users hanging in session, connection pool issues, i've fixed all those. Also I was hitting the max memory on Supabase, so I upgraded it. Now I'm pretty sure I'm good on all fronts but STILL it just crashes regularly with no explanation or problem...?

Why is it happening? Redeploying fixes it every time but I can't be around the server to redeploy every 15 minutes.

Getting quite desperate. Would appreciate any help.

Solved$10 Bounty

8 Replies

Railway
BOT

23 days ago

Hey there! We've found the following might help you get unblocked faster:

If you find the answer from one of these, please let us know by solving the thread!


noahd
EMPLOYEE

23 days ago

I'm looking at your service metrics and it appears like everything is within bounds. I will say though we do support hosting supabase on your railway project! Theres a template here:
https://railway.com/deploy/supabase

Are you able to try using that and seeing if that fixes it?


Status changed to Awaiting User Response Railway 23 days ago


noahd

I'm looking at your service metrics and it appears like everything is within bounds. I will say though we do support hosting supabase on your railway project! Theres a template here:https://railway.com/deploy/supabaseAre you able to try using that and seeing if that fixes it?

yaniv297
PROOP

23 days ago

It just happened again frowning_face emoji 12 minutes ago same thing, just a random shutdown.

I printed the memory and it never went above 72%

I'll try the supabase thing tomorrow but I don't think it'll fix that, there's no way to investigate why those shutdowns are happening?


Status changed to Awaiting Railway Response Railway 23 days ago


noahd
EMPLOYEE

23 days ago

I don't see any erroneous logs on my end which would indicate a platform issue. This does feel a bit like an issue with how your app is exiting on a form of error. I'll open this to the community to help!


Status changed to Awaiting User Response Railway 23 days ago


noahd

I don't see any erroneous logs on my end which would indicate a platform issue. This does feel a bit like an issue with how your app is exiting on a form of error. I'll open this to the community to help!

yaniv297
PROOP

23 days ago

I really don't think my app is crashing... It's been working a long time when I was the only user and in local env. This really seems to be a load issue. It crashes all the time and on pretty minimal use. It doesn't make any sense. All signs point to a SIGTERM from the platform or from the DB. I also had this error at times:

psycopg2.OperationalError: SSL connection has been closed unexpectedly


yaniv297
PROOP

23 days ago

And now it's down again... It can't continue like this, it just keeps happening. Again no error message. Working perfect and than boom shut down.


Anonymous
PRO

23 days ago

Bro the issue isn’t Railway the platform is killing your API because your Postgres connections are dropping and your API freezes. Do these two things and it will stop crashing

  1. Add Postgres connection pooling - If you’re using Python, stop creating psycopg2 connections on every request.

use pool

from psycopg_pool import ConnectionPool

pool = ConnectionPool(conninfo=DATABASE_URL, min_size=1, max_size=20)

with pool.connection() as conn:

This prevents Supabase from randomly closing SSL connections.

  1. Run your API with multiple workers. If you’re starting your server with uvicorn app:app than that’s only 1 worker, so one blocked DB call kills the whole app. Instead Start it like this instead - gunicorn app:app -k uvicorn.workers.UvicornWorker --workers 20 --timeout 60

    Multiple workers = no more random shutdowns when a DB connection drops.


yaniv297

And now it's down again... It can't continue like this, it just keeps happening. Again no error message. Working perfect and than boom shut down.

passos
MODERATOR

22 days ago

Hey, let's go through this step by step as there could be many reasons why that's happening. Also, there's no need to use Supabase on Railway, so don't worry about that.

First of all, please share more logs about your application when it crashes. Do you see any pattern regarding when these crashes occur (any endpoints being called, traffic spikes, etc.)? Are you able to connect to your database after the crashes happen? Additionally, enabling your Supabase Postgres Logs is an effective way to identify who is terminating the connection: https://supabase.com/docs/guides/telemetry/logs#logging-postgres-queries

I would also recommend using sentry.io in your application as it's good to track these kinds of issues.


Status changed to Awaiting User Response Railway 22 days ago


Railway
BOT

15 days ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway 15 days ago


Loading...