a month ago
Hello Railway Support,
We experienced prolonged intermittent connectivity between our backend and PgBouncer. Postgres remained running without resource, crash, or connection-limit problems. We recovered by connecting directly to Postgres.
Resources:
- Project: itm-backend (0696700c-6952-4780-8a3d-01600989ab48)
- Production environment: a0eb9749-521d-4644-83c7-9062f63eb931
- Region: us-east4-eqdc4a
- Backend service/deployment: 75b3cb48-626f-4cda-bd54-dbbc895870ac / 8f1d589b-281e-40c3-b3ac-88023dd7dfe2
- PgBouncer service/deployment: 97489b2a-fa18-4c1a-8c40-cb67a9d56ea7 / 07e93f1f-3698-4fc7-b6d9-cb29b457bf03
- Postgres service/deployment: d270acf2-5adf-4f29-97ce-0926832d2eee / bb09beac-b50a-4437-8ab2-f4efdf3aff47
Timeline (July 18, 2026 UTC):
Logs were healthy through 04:18. Simultaneous heartbeat and new-connection failures across both backend replicas began around 04:19:17 (12:19 AM EDT). At 04:19, 21/43 GraphQL requests failed; at 04:20, 37/41 failed. One request exceeded 40 seconds, and later backlogged requests exceeded 200 seconds.
The issue continued intermittently for approximately eleven hours and became severe again around 14:42. Both replicas restarted around 14:43, but failures continued. At 14:57:47, we changed DATABASE_URL from PgBouncer to direct Postgres, triggering deployment 48e45331-ba79-440f-aedb-27a12f535e4e. It succeeded shortly after 15:05 and service recovered. No service deployment occurred near the original 04:19 onset.
Observed behavior:
The replicas used pgbouncer.railway.internal:6432, with pools capped at eight connections each and a five-second timeout. Both repeatedly logged:
- "timeout exceeded when trying to connect"
- "Connection terminated due to connection timeout"
- "Connection terminated unexpectedly"
During the 14:00 hour, we counted approximately 2,709 pool-acquisition timeouts and 180 new-connection timeouts, while Railway reported all deployments as SUCCESS.
PgBouncer was normal immediately before onset. Throughput fell to approximately four transactions/second at 04:19, then zero transactions and queries at 04:20–04:22. Around 04:23 it began logging client_login_timeout. Some connections appeared as (nodb)/(nouser), others as railway/postgres; source IPv6 addresses matched the backend replicas.
Postgres showed no crash, restart, OOM, connection-slot error, or saturation. Checkpoints continued, CPU and memory stayed low, and max_connections was 100 with three reserved. Redis remained healthy.
Tests from inside a backend container before the change:
- PgBouncer: 12/12 successful at that moment, 29–188 ms
- Direct Postgres: 12/12 successful, 11–18 ms
This indicates an intermittent PgBouncer-path failure rather than a permanent outage.
After switching to postgres.railway.internal:5432:
- 20/20 DB, 20/20 Redis, and 20/20 aggregate health checks passed
- No new connection or heartbeat timeout logs
- Postgres had 18 sessions, one active, with no capacity/fatal errors
The deployment also replaced the app instances, so it does not conclusively prove the PgBouncer process was responsible. However, earlier replica restarts did not resolve the incident, while bypassing PgBouncer did.
Please investigate:
- PgBouncer service/host degradation beginning 2026-07-18 04:19 UTC.
- Private IPv6 networking among the backend replicas, PgBouncer, and Postgres.
- Packet loss, stalled connections, host migration, or connection-tracking issues.
- Why accepted connections ended in client_login_timeout.
- Any unreported/project-specific event or infrastructure change in us-east4-eqdc4a, including migration, rescheduling, or throttling without a status change.
- Whether internal telemetry shows incomplete handshakes/resets, and whether PgBouncer should be redeployed or recreated before restoring traffic.
We are temporarily using direct Postgres and request a root-cause analysis and guidance on restoring pooling.
3 Replies
a month ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • about 1 month ago
a month ago
Try a manual redeploy of the PgBouncer service (via an empty commit or the Railway CLI).
Additionally, while your 5-second pool-acquisition timeout is an excellent fail-fast mechanism, you may want to temporarily relax it to 10–15 seconds.
a month ago
Hi Railway team — I completed a deeper investigation and have additional evidence that this was not a PostgreSQL outage or ordinary pool
saturation.
The incident began at 2026-07-18 04:19:17 UTC (12:19:17 AM EDT) in us-east4-eqdc4a. The dashboard graph appears bucketed at midnight, but the
raw backend failures began at 04:19.
What the logs show
Both backend replicas simultaneously stopped completing new connections through PgBouncer. Importantly, physical connection failures happened
before local pool exhaustion:
- 04:19: 29 physical connection timeouts; 4 pool-acquisition timeouts
- 04:20: 38 physical connection timeouts; 9 pool-acquisition timeouts
- 04:21: 5 physical connection timeouts; 19 pool-acquisition timeouts
This indicates that pool exhaustion was a downstream effect of failed physical connections.
PgBouncer 1.24.1 remained alive and continued emitting stats, but traffic collapsed:
- 04:18: 16 transactions/sec, 20 queries/sec
- 04:20–04:22: effectively zero transactions and queries
- CPU and memory remained low
PgBouncer logged nine plain client_login_timeout errors, including both:
(nodb)/(nouser) ... client_login_timeout
railway/postgres ... client_login_timeout
There were no:
client_login_timeout (server down)
server_connect_timeout
server login failed
PgBouncer crash/OOM/fatal errors
This distinction is significant. In PgBouncer 1.24.1, client_login_timeout (server down) is used when a client is waiting for PgBouncer’s first
PostgreSQL server connection/welcome packet. A plain client_login_timeout means the connection is still in the client-login path.
Our PgBouncer uses static local SCRAM authentication, not auth_query, so PostgreSQL is not involved in this login phase. Some sockets never
delivered a parseable startup packet, while others delivered the database/user fields but stalled before SCRAM authentication completed.
PostgreSQL remained healthy
PostgreSQL showed:
- No restart or crash recovery
- No OOM
- No too many connections or reserved-slot exhaustion
- No abnormal CPU pressure
- Normal checkpoints
- No corresponding fatal authentication or server-connect errors
Both backend replicas also had healthy CPU, memory, and event-loop metrics. Redis remained responsive. This makes an application-process stall,
traffic overload, or PostgreSQL saturation very unlikely.
Redeploy behavior
I attempted to redeploy PgBouncer earlier during the incident, but it did not restore service at that time—the backend continued experiencing
connection failures. I cannot confirm whether that earlier attempt successfully replaced/rescheduled the container or merely restarted it on the
same underlying host/network state.
Railway’s deployment history shows a successful PgBouncer deployment at:
2026-07-18T15:05:08Z
deployment: 75d03bac-f0a4-47b4-9fe7-e4b52800d46b
It used the exact same image digest as the old deployment.
At nearly the same time, I changed the backend DATABASE_URL to connect directly to Postgres and redeployed the backend. Therefore, final
recovery is confounded between:
- Bypassing PgBouncer
- Replacing both backend replicas
- Successfully replacing/rescheduling PgBouncer
A later protocol-aware check succeeded through the new PgBouncer from both backend replicas, so the configuration/image is not persistently
broken.
Likely failure boundary
The evidence localizes the failure to:
backend replicas → PgBouncer client ingress/login path
The most likely cause is a transient Railway private-network/WireGuard/IPv6 or PgBouncer-host networking impairment. The main alternative is
transient PgBouncer/container socket state. The available customer logs cannot distinguish those without host-level telemetry.
We also found multiple other July 18 reports involving ECONNRESET, private IPv6 timeouts, public database proxy failures, and normal database
metrics in US East. No corresponding public incident currently appears on the Railway status page.
Please investigate
Could you inspect the following around 04:19–04:43 UTC and the recurrence around 14:42 UTC?
- Private IPv6/WireGuard routing and peer health
- Packet drops, retransmits, asymmetric routing, or conntrack exhaustion
- PgBouncer host socket/accept-queue state
- Host placement for the two backend replicas and PgBouncer
- Whether the earlier PgBouncer redeploy stayed on the same host
- Whether the successful 15:05 deployment moved PgBouncer to a different host
- Correlation with other July 18 US East connectivity reports
- Whether there was a subset-host incident that did not meet the public status threshold
The service is currently healthy using direct Postgres, but I would appreciate confirmation of the underlying infrastructure failure before
routing production traffic through PgBouncer again.
a month ago
Hey Saarim (and others)
been a while! Since this happened a bit ago, we think we resolved it but we think there was something on one particular box that was torublesome that we don't have the full story just yet.
It lines up with a network-level incident that hit connectivity in US East over the weekend, and it landed on the host your services were running on. That's why you got client-login timeouts and stalled handshakes while Postgres itself looked totally fine.
We're putting together a quick RCA and I'll follow up here with the specifics, including the host-level impact and the timeline. (For you and others)
On pooling: the underlying issue has cleared, so you're good to route back through PgBouncer whenever you like. Nothing to change on your side.
Thanks again and sorry for the headache.
Angelo