9 days ago
Project: customized-staging (US East / Virginia)
Affected services: Postgres (public TCP proxy gondola.proxy.rlwy.net:15055), PgBouncer (public TLS ingress), and a second database behind shortline.proxy.rlwy.net:37117
All timestamps below are UTC.
SUMMARY
Since 2026-07-18 ~02:00 UTC, established TCP connections crossing the US East public edge to BOTH of our database entry points have been reset mid-stream at 3x to 9x our normal baseline. This has been ongoing for 36+ hours and is still elevated as I write this (roughly 900-1400 TLS EOFs/hour overnight 07-19 vs a ~200/hour baseline). The incident does not appear on your status page apart from 66L640AT (Outbound Static IPs, 18:03-19:26 on 07-18), which neither covers our onset nor our ongoing symptoms.
TIMELINE (error rates are per-hour counts from our service logs)
- Baseline (07-16, 07-17): PgBouncer inbound "unexpected eof" ~30-480/hr with nightly ~02:00 peaks; Postgres "connection reset by peer" 12-33/hr.
- 07-18 02:00-05:00: first wave. EOFs 578 / 527 / 913 / 347 per hour; Postgres resets 47 / 77 / 166 / 60. First application job failure 03:10:05 (Prisma: "Server has closed the connection").
- 07-18 04:22:47: a job could not reach our OTHER database at shortline.proxy.rlwy.net:37117 ("Can't reach database server"). Two independent proxy hostnames failing in the same window points at the edge layer, not one database.
- 07-18 06:00-14:00: sustained elevation (230-440 EOFs/hr, i.e. 3-5x those hours on prior days).
- 07-18 14:57 onward: severe wave. EOFs climbed 760 -> 1190/hr through the evening; Postgres resets 116-196/hr. Overnight it peaked at 1380/hr at 07-19 02:00 and remains elevated.
EVIDENCE THE CUT IS AT YOUR EDGE PROXY (not our app, not the DB)
- Both ends blame the peer on the SAME streams. Our hourly pg_dump (running on GitHub Actions, connecting directly to gondola.proxy.rlwy.net:15055) was killed mid-COPY of our largest table at 14:57, 18:00, 19:12, 20:54, and 22:54 on 07-18 with "server closed the connection unexpectedly" (exact failure moments 15:00:20 and 22:57:09 for the two runs we inspected in detail). Postgres logs show "could not send data to client: Connection reset by peer" for the same streams. Each endpoint saw the other vanish; the only component between them is your proxy.
- Private networking is clean. PgBouncer -> Postgres over railway.internal shows ZERO errors across the entire incident while the public ingress takes 1000+ EOFs/hour. Same database, same host, different path.
- The database is healthy. No restart in the whole window, no OOM, normal checkpoints, zero "too many connections" over 72 hours, currently 25 of 100 connection slots in use.
- Reproduced from three independent client networks: AWS (Vercel functions), Azure (GitHub Actions runners), and a residential ISP (held-connection probe from a local machine, which dropped mid pg_sleep on 07-18).
- The kills hit ESTABLISHED, aged connections. PgBouncer logs show TLS "error:0A000126:SSL routines::unexpected eof while reading" and "closing because: client unexpected eof" on connections up to 22 minutes old (example: age=1341s). These are mid-stream severs, not connect-time failures.
- We ruled ourselves out. We audited every deploy in the window: no ORM/driver version changes, no pool or connection-string changes, no new connection topology. The worst escalation (760 -> 1190 EOFs/hr) happened during a 7-hour window with zero deploys from us. There is no connection-exhaustion signature anywhere.
IMPACT
Customer-facing API 500s in bursts, ~45 failed cron jobs, and 6 failed database backups over the window.
REQUEST
Please escalate to the team that owns US East public TCP proxy / TLS ingress. The per-minute timestamps above (especially 03:10:05, 04:22:47, 15:00:20, 22:57:09 on 07-18) should correlate directly against your internal edge metrics. If a specific proxy instance serving our tenant is degraded, please advise whether you can migrate us off it or what mitigation you recommend. Happy to provide raw log excerpts for any of the above.
11 Replies
9 days ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • 9 days ago
9 days ago
You need to reroute all internal tenant traffic directly through Railway’s private network layer.
The service hosted within Railway must connect using the .railway.internal domain rather than public proxies like gondola.proxy.rlwy.net:15055 or shortline.proxy.rlwy.net:37117.
9 days ago
Thanks for the response, but this doesn't match our topology, as covered in the original post.
The clients experiencing the mid-stream connection resets are external to Railway and cannot use private networking:
- Our application runs on Vercel (AWS us-east-1), not inside Railway. Vercel functions cannot resolve .railway.internal; the public TCP proxy is the only supported path for them.
- Our backups run on GitHub Actions runners, also external.
- We also reproduced the drops from a residential ISP with a held-connection probe.
The only intra-Railway traffic we have (PgBouncer to Postgres) already connects over railway.internal, as stated in the original ticket, and it has had zero errors throughout the incident. That contrast is the core evidence: the same database is healthy on the private path while established connections through the US East public edge are being severed mid-stream at up to 9x baseline, for 36+ hours, from three independent external networks, on two different proxy hostnames (gondola.proxy.rlwy.net:15055 and shortline.proxy.rlwy.net:37117).
External clients connecting through the TCP proxy is exactly what the proxy product exists for. Could this please be escalated to the team that owns US East public proxy / TLS ingress? The per-minute timestamps in my earlier post (03:10:05, 04:22:47, 15:00:20, 22:57:09 UTC on 07-18) should correlate directly with internal edge metrics.
matt998759
Thanks for the response, but this doesn't match our topology, as covered in the original post. The clients experiencing the mid-stream connection resets are external to Railway and cannot use private networking: - Our application runs on Vercel (AWS us-east-1), not inside Railway. Vercel functions cannot resolve .railway.internal; the public TCP proxy is the only supported path for them. - Our backups run on GitHub Actions runners, also external. - We also reproduced the drops from a residential ISP with a held-connection probe. The only intra-Railway traffic we have (PgBouncer to Postgres) already connects over railway.internal, as stated in the original ticket, and it has had zero errors throughout the incident. That contrast is the core evidence: the same database is healthy on the private path while established connections through the US East public edge are being severed mid-stream at up to 9x baseline, for 36+ hours, from three independent external networks, on two different proxy hostnames (gondola.proxy.rlwy.net:15055 and shortline.proxy.rlwy.net:37117). External clients connecting through the TCP proxy is exactly what the proxy product exists for. Could this please be escalated to the team that owns US East public proxy / TLS ingress? The per-minute timestamps in my earlier post (03:10:05, 04:22:47, 15:00:20, 22:57:09 UTC on 07-18) should correlate directly with internal edge metrics.
9 days ago
Deploy Client-Side Connection Pooling (gateway Pooler) by running a localized PgBouncer instance on the external network's edge (inside AWS VPC).
Since Vercel and GitHub Actions are outside the Railway network, .railway.internal is completely inaccessible to them, and relying on gondola and shortline public proxies is the suggested path.
I hope edge engineering team has eyes on those logs for Envoy configuration issue, node state exhaustion, or aggressive TCP connection.
9 days ago
Corroborating from a different traffic direction: outbound container egress from US East is affected too, not just the inbound public proxy.
Our services (US East / Virginia) connect outbound to two separate Supabase projects in AWS us-east-2 (pooler, port 6543) via the Static Outbound IPs feature, plus server-side realtime WebSocket connections to Ably. All timestamps UTC.
07-18: Bursts of "Connection reset by peer" to both Supabase projects, 17:16–18:10. Overlaps incident 66L640AT (Outbound Static IPs, 18:03–19:26), but our onset preceded its stated start by ~47 minutes. Clean for 9+ hours after 18:10.
07-19 (ongoing, no status page coverage): Recurring bursts of "Can't reach database server" (connect failures to the pooler), transaction-start timeouts, and health-check timeouts:
- 03:40–03:54
- 04:24–04:25
- 05:07–05:09
- 05:52–05:55
Each burst lasts 1–3 minutes, recurring on a regular ~44-minute cycle (intervals: 44, 43, 45 min). The staging project failed in the same windows as production (03:48, 05:09, 05:54). The periodicity suggests a recurring infrastructure event (proxy recycling / connection-table sweep) rather than load, and may match the "same-instant sweep" behavior in the jfk1/hikari threads.
Evidence this isn't our app or the destinations:
- Both Supabase projects healthy throughout: normal checkpoints, no restarts, no connection exhaustion; Supabase status green. The only burst-correlated anomaly in the database's own logs is "there is no transaction in progress" warnings — clients re-issuing commands after mid-stream connection loss — appearing exclusively inside the burst windows.
- In the same windows, established WebSockets to Ably died mid-stream: idle-timer expiry after ~25s of no inbound data (03:47, 05:54–05:57), then failed resumes. Three destinations, two providers, two protocols, one shared egress path.
- We have no scheduled work at a ~44-minute period (job intervals are 60s/5min/1h, with jitter), and our staging and production services — started hours apart — fail in the same wall-clock windows, ruling out in-process timers.
- Our only deploy in the window was hours before onset and contained no runtime code changes. Between bursts, the same connections work flawlessly.
Happy to share the project ID and exact per-minute timestamps privately with support for correlation against edge metrics.
9 days ago
Corroborating jbeno's windows from the opposite traffic direction. We are a different tenant (unrelated stack) whose failures are INBOUND through the US East public TCP proxy / TLS ingress into Postgres and PgBouncer, while jbeno's are outbound egress. Per-minute error counts from our service logs, all timestamps UTC 07-19:
- 03:40 to 03:54 (jbeno window 1): our single worst cluster of the night, bookended exactly by this window. Postgres "connection reset by peer" peaked at 14/min at 03:40 (the highest minute in our entire 03:30-06:10 series), and PgBouncer TLS "unexpected eof" ran 46, 38, 48, 36, and 26 per minute at 03:40, 03:43, 03:46, 03:50, and 03:54.
- 04:24 to 04:25 (window 2): our spike lands at 04:26-04:27 (42 EOFs/min, 8 resets/min), within about two minutes.
- 05:07 to 05:09 (window 3): exact match. 32 and 16 EOFs/min plus 8 and 8 resets/min at 05:07-05:08.
- 05:52 to 05:55 (window 4): only a mild signal on our side (3-4 resets/min).
So: two tenants, opposite traffic directions, different destination stacks, synchronized worst moments. Two additional details for the edge team: the connections being severed on our side are long-established (TLS "unexpected eof" on connections aged 1300+ seconds, and pg_dump COPY streams killed ~2 minutes in), which is consistent with periodic state recycling or a connection-table sweep rather than load. And between the shared windows we carry a continuous lower-grade bleed with intensifications roughly every 20 minutes, so the ~44-minute cycle may not be the only mechanism in play.
Happy to share project IDs and raw per-minute logs privately for correlation against internal edge metrics.
9 days ago
+1 , we're seeing the identical issue from a second tenant, and notably on MySQL (not Postgres).
Setup: App on AWS ECS (us-east-1) → Railway MySQL, US East, connected over the public TCP proxy.
Onset: First error 2026-07-18 ~12:59 UTC, step-change from a flat baseline (literally 0 in the prior 6 days of our log retention). Still ongoing as of 07-19 ~16:00 UTC.
Exact error (thousands of them):
pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query ([Errno 104] Connection reset by peer)')
plus a smaller number of (2003, "Can't connect to MySQL server"). Rate went 0 → ~400/hr peak, sustained ~180–250/hr.
The DB itself is healthy (matches the OP): Railway metrics show CPU ~idle (<0.3 vCPU), memory flat, no restarts.
Ruled out on our side (this is the important part — it's not client tuning):
- No app deploy that day; traffic was not elevated.
- We raised
max_connections151→500 and added SQLAlchemypool_pre_ping+ bounded pool +pool_recycle, and deployed ; error rate was essentially unchanged.
This is a public-proxy edge issue, US East, starting 07-18. It's not on the status page and appears unresolved 36+ hrs in. Would appreciate an official acknowledgment + ETA - happy to share raw CloudWatch samples or a per-minute error timeline if useful.
9 days ago
Corroborating from another tenant. Local Next.js/Prisma → Railway Postgres in US East intermittently returns P1017 “Server has closed the connection.” Postgres logs show SSL error: unexpected eof while reading and Connection reset by peer. The database is healthy, and replacing it with a fresh restored Railway Postgres instance did not resolve it. Both old and new public proxy endpoints resolve within 66.33.22.0/24. Read retries are currently masking the customer-facing errors (as best as they could lol but not fully stable).
9 days ago
Follow-up from our project after overnight monitoring. Some context: our production traffic is very light, mostly health checks, with occasional bursts of real usage. Absolute error counts are therefore small compared with other reports, but during each burst a large share of real DB-bound requests failed (plus health checks and Ably).
Additional bursts after my earlier post, still UTC 07-19:
- 05:52 - 06:01 - window 4 ran longer than first reported, it shifted from connect failures to mid-stream
Connection reset by peer/Server has closed the connection - 07:23 - 07:24 - short production reset burst (about 91 min after 05:52, roughly two 45-min cycles)
- 07:54 - 07:57 - final clustered burst (31 min after 07:23)
- 08:50 - one isolated staging reset, then quiet
From 08:50 through 17:34 UTC (while our real usage was idle, mostly health checks, which are short/small queries) we saw no further Connection reset by peer / Can't reach database / Ably idle-timer events on either Supabase project. Health checks were failing during the earlier burst windows. Not sure if this indicates the situation improved or not. Will know more with real usage later today.
Notes for the edge team, given the other replies:
- matt998759's per-minute alignment with our 03:40 / 04:24 / 05:07 / 05:52 windows is the strongest cross-tenant signal: inbound public proxy and outbound Static Outbound IP egress failing in the same minutes, unrelated stacks.
- Anonymous (MySQL) and kapiushon8 (Postgres; fresh DB on a new proxy IP still fails) show this isn't Postgres-only and isn't fixed by replacing the database.
9 days ago
There was another thread https://station.railway.com/questions/intermittent-502s-on-web-socket-upgrades-826e8f70 that was marked as solved regarding this: "This service is a realtime multiplayer game server (Node, ws, long-lived WebSockets). Since at least 2026-07-17 evening, players are disconnected every few minutes." They were able to go around this issue by redeploying to US West.
I also have a app using live WebSockets, and I can corroborate seeing the same behavior with our server dropping out every few minutes on US East, with occassional 502s in Railway's HTTP logs. This began around the same time this all did.
Similarly to the other thread, I also for now have redeployed to US West, and the issue cleared up. Or at the very least I cannot reproduce the behavior again with my specific app after redeploying over there. Hoping this visibility helps Railway track down what's going on in US East, and gives others a temporary quick fix option if their setup allows for it.
8 days ago
Same issue here using Postgres on the public proxy. Intermittent 'Connection terminated unexpectedly' errors when making requests to the db through public url. Commenting here to help boost this issue
8 days ago
Hi,
I'm sorry for not responding sooner here and that you hit this issue.
I isolated the cause of the intermittent resets to a network hardware failure and completed mitigations over the last 2 hours to prevent this impacting your traffic.
You should be seeing recovery. I'll work with support to get more detailed shared here on Monday.

