a month ago
Since approximately 2026-07-18 23:00 UTC, TCP connections from outside Railway
to my Postgres service's public TCP proxy intermittently fail with either
"timeout expired" (connect never completes) or "server closed the connection
unexpectedly" (established connections reset mid-query).
Evidence gathered:
-
Reproduced across TWO different proxy assignments: the issue started on
nozomi.proxy.rlwy.net:11531 (66.33.22.249). I deleted and re-created the
TCP proxy; the new one, tokaido.proxy.rlwy.net:45428 (66.33.22.222),
shows the same failures — both resolve into the same 66.33.22.0/24 edge
range, so swapping proxies did not escape it.
-
Measured failure rates (repeated sequential connection attempts):
-
old proxy: ~30% of connection attempts failed
-
new proxy: ~4-8% still failing
-
failures occur at the raw TCP level (plain socket connect times out
before any Postgres handshake).
-
-
The Postgres service itself is healthy: I restarted it (no change),
connection count is ~17 of max 500, and successful connections complete
in ~0.55s and query normally.
-
My local network is not the cause: interleaved control connections from
the same machine to unrelated hosts (google.com:443 and a Government of
Canada site) show 0 failures across 50 attempts during the same window
the proxy attempts were failing.
Impact: a long-running batch job lost 563 of 1513 database writes during the
worst window on Jul 18-19.
Could you check the health of the edge/proxy infrastructure serving the
66.33.22.0/24 range, or move my service's proxy onto a healthy node?
3 Replies
a month ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • about 1 month ago
a month ago
Hi,
Yes. The issue can be reproduced without our application.
Environment
- Railway PostgreSQL via public TCP proxy
- Client: Windows 11
- Python 3.x
- psycopg2
Minimal reproduction
import psycopg2
DB_URL = "postgresql://<user>:<password>@tokaido.proxy.rlwy.net:45428/<database>"
for i in range(100):
try:
conn = psycopg2.connect(DB_URL, connect_timeout=6)
conn.close()
print(f"{i}: OK")
except Exception as e:
print(f"{i}: FAILED - {e}")The failures occur during the initial TCP connection before any SQL is executed.
To isolate the application completely, I also ran a raw TCP socket test that simply attempts to connect to the Railway proxy repeatedly:
import socket
for i in range(25):
try:
s = socket.create_connection(("tokaido.proxy.rlwy.net", 45428), timeout=6)
s.close()
print("OK")
except Exception as e:
print(e)Results:
- Railway proxy: 24 successful, 1 timeout
- google.com: 25/25 successful
- innovation.ised-isde.canada.ca: 25/25 successful
This suggests the intermittent failures occur before PostgreSQL receives the connection and appear to be related to the TCP proxy path rather than our application or SQL workload.
a month ago
Apparently, the issue has been resolved.
Status changed to Solved milo • about 1 month ago