Recent 500s and timeouts in the past 24 hours
saarimz
PROOP

6 months ago

It seems like in the last 24 hours, my Railway deployments have gotten nerfed and I'm noticing a lot more timeouts and 500s happen in my staging and production databases. Nothing has changed on my end, but it looks like there are much bigger node memory leaks and I'm not sure why this is all happening.

Solved

4 Replies

Railway
BOT

6 months ago

Our team is working on getting back to you as soon as possible. In the meantime, we've found the following might help you get unlocked faster:

If you find the answer from one of these, please let us know by solving the thread! Otherwise, we’ll be in touch.


saarimz
PROOP

6 months ago

I really think that there is something that happened on Railways end:

  • tons of more 500s

  • frequent timeouts

  • sudden increase in cpu + memory

i am using a nest js application with prisma ORM connected to a railway postgres db over private networking


echohack
EMPLOYEE

6 months ago

Hey saarimz,

I'm taking a deeper look at your 500 errors. At first glance, I am noticing an increase in CPU as well as network egress. I'm going to have to take a deeper look to see root cause, but I'm not seeing anything from our end in terms of the underlying compute that your Services are running on. I'm guessing this is related to increased traffic on your application.


Status changed to Awaiting User Response Railway 7 months ago


echohack
EMPLOYEE

6 months ago

Hey Saarimz,

I'm certain what you're seeing is connection pool exhaustion, which makes sense with increased traffic to your application. I'm not seeing any PGBouncer or pooler in front of your postgres server that would mitigate this. Once you increase traffic to your database, you're going to run out of connections. You mentioned Prisma -- prisma usually also has some configuration that you may want to check around connection pooling.

Have you considered using PGBouncer in your architecture? I would highly recommend it so you can both reuse connections and keep the connections warm for faster transactions. https://www.pgbouncer.org/

PGBouncer can be run in several modes. I'd consider using session mode for your kind of application for maximum connection warmth. https://www.pgbouncer.org/config.html


Railway
BOT

5 months ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway 6 months ago


Loading...