Frequent connection resets between app service and Postgres — possibly related PgBouncer/private networking incident
douglasdario-a11y
HOBBYOP

a month ago

Project ID: 2613e0d7-17f8-4766-b94d-10cde415ad7e

My backend service (lapanaderia-pos) has been experiencing frequent, chronic connection drops to its Postgres database (same Railway project) since 5:18 AM Costa Rica time (11:18 UTC) on July 18, 2026. The deploy logs show repeated ECONNRESET and "Connection terminated unexpectedly" errors on the Postgres client connection — not a one-time blip, but recurring many times per hour throughout the entire day (confirmed via a continuous 4+ hour log sample from 17:13 UTC to 21:12 UTC on the same day).

This caused the service to crash outright a few times early on (unhandled error events crashing the Node process), which I've since fixed on my end by adding proper error handlers and automatic query retries — so the service no longer goes down because of it. However, the underlying connection resets themselves are still happening constantly, and at least one database write was delayed by a retry as a direct result (a sale record that took an extra ~1 second to save due to a reset-and-retry cycle).

I noticed another thread reporting an intermittent PgBouncer connectivity issue on the same date ("Intermittent production database connectivity through PgBouncer in us-east4 beginning July 18 at 04:19 UTC"). Unlike that report, my project connects directly to Postgres via postgres.railway.internal:5432 (no PgBouncer in this stack) and experienced the identical Connection terminated unexpectedly symptom — suggesting this isn't confined to PgBouncer-fronted setups, but a broader intermittent issue with Railway's private networking on that date.

I checked status.railway.com and found no incident listed for July 18 covering this. Could you help confirm whether there's something on the infrastructure/networking side (between app services and Postgres) that could be causing these frequent resets? Happy to share deploy log excerpts if useful.

$10 Bounty

6 Replies

Railway
BOT

a month ago

This thread has been opened as a bounty so the community can help solve it.

Status changed to Open Railway about 1 month ago


kailuowang
PRO

a month ago

same here


douglasdario-a11y
HOBBYOP

a month ago

Update: Also noticed this line in my logs, which stands out from the rest:

Error cargando productos connect ETIMEDOUT 66.33.22.224:18203

That IP:port matches my DATABASE_PUBLIC_URL (the reseau.proxy.rlwy.net:18203 proxy), not the internal connection (postgres.railway.internal:5432) my service is actually configured to use. Seems odd that address would show up at all — might be a useful clue if there's some DNS resolution or internal failover behavior happening during these connection drops.


kailuowang
PRO

a month ago

My data ingestion (through public TcP proxy) kept getting connection dropped. On service log I am seeing dozens of " unexpected EOF on client connection with an open transaction". We didn't have any of those since at least June 11


kailuowang
PRO

a month ago

I am seeing frequent web socket connection reset as well.


douglasdario-a11y
HOBBYOP

a month ago

Update (July 19): This is still ongoing on a second consecutive day — same pattern of frequent ECONNRESET/"Connection terminated unexpectedly" throughout the entire business day (12:08–18:09 UTC today), same anomalous public-proxy IP showing up again (66.33.22.224:18203). No crashes on my end (my fixes are holding), but the underlying network instability clearly hasn't resolved itself — this looks like a sustained degradation, not a one-day blip.


douglasdario-a11y
HOBBYOP

a month ago

Update: Looks like this resolved on its own around July 19, ~21:45 UTC (15:45 CST) — no ECONNRESET/"Connection terminated" errors since then, running clean for 1.5+ days now (through July 21 morning). Marking this as recovered on my end, but still curious about the root cause if you find anything.


Welcome!

Sign in to your Railway account to join the conversation.

Loading...