Subject: TCP proxy accepts connection but never responds — reproducible, ~65% of attempts from one source IP
barradach
PROOP

a month ago

Hello,

We have a Postgres service exposed through a Railway TCP proxy. Since 2026-09-04, connections from one of our servers behave as follows: the TCP handshake completes normally, but the first PostgreSQL message receives no response at all — the connection simply hangs until our timeout. This happens on roughly 65% of attempts. The remaining 35% work perfectly, at completely normal speed.

Two client servers are involved. Both are at the same hosting provider, in different networks with different upstream transit:

Server A — affected

Server B — works normally, used as a control

REPRODUCTION

This sends an 8-byte PostgreSQL SSLRequest and expects a 1-byte reply:

printf '\x00\x00\x00\x08\x04\xd2\x16\x2f' | timeout 15 nc | xxd

Expected output: 00000000: 53 S

Actual output from Server A: empty, times out.

MEASUREMENT: 20 attempts from each server, taken 2.5 minutes apart

2026-09-05 15:05:18 UTC, from Server A:

7 of 20 attempts succeeded; successful ones averaged 0.50s

2026-09-05 15:07:48 UTC, from Server B:

20 of 20 attempts succeeded; averaged 0.53s

Same target, same command, same minute.

The most important detail: when a connection from Server A does succeed, it is exactly as fast as from Server B. There is no latency degradation and no packet loss pattern. The behaviour is binary — either served immediately, or never.

SOCKET STATES on Server A toward the proxy endpoint, at the time of measurement:

5 CLOSING

8 FIN-WAIT-1

4 TIME-WAIT

0 ESTABLISHED

Our FIN packets are not being acknowledged — connections accumulate half-closed instead of closing cleanly.

PATH COMPARISON (tracepath to the proxy IP, intermediate hops)

From Server A (affected):

185.215.163.29 -> 172.16.13.0 -> 87.245.230.96 -> 87.245.232.142 -> no reply

From Server B (control):

62.115.196.193 -> 62.115.139.186 -> 62.115.137.189 -> 62.115.196.223

The two servers leave through different transit providers.

WHAT WE HAVE ALREADY RULED OUT

  1. Not the database. max_connections is 500; pg_stat_activity shows about 10 client connections. Our API service, which connects over Railway's internal network, has worked flawlessly throughout the entire incident.
  2. Not general connectivity from Server A. From the same server at the same time: example.com returned 200 in 0.26s, api.telegram.org returned 302 in 0.25s, cloudflare.com returned 301 in 0.09s. Only this endpoint is affected.
  3. Not MTU. The payload that gets no response is 8 bytes.
  4. Our hosting provider reports no filtering on their side. They confirm MTR reaches the proxy IP with no packet loss, and that the TCP handshake completes. Their conclusion is that this looks like something on the receiving side reacting to our specific source IP.

QUESTIONS

  1. Can a TCP proxy of this kind accept the TCP handshake at the edge and then not forward the connection to the backend — for example due to per-source-IP rate limiting or connection limiting? If such a limit exists, are half-closed connections (FIN_WAIT / CLOSING) counted against it?
  2. If we delete and re-create the TCP proxy for this service, is it assigned a different edge IP, or the same one?
  3. Has anyone seen this pattern before — handshake completes, first data packet never answered, roughly half the attempts, successful ones at full speed?

We have root access to both servers and can run any diagnostic, including packet captures on our side.

Thank you.

$20 Bounty

1 Replies

Railway
BOT

a month ago

We've looked into this from our side and haven't found anything on the Railway platform that explains what you're seeing, so working it out means digging into your specific setup.

That's exactly what the Railway community is good at, so we'd like to open your thread as a community bounty. Railway pays a bounty to the community member who solves it, and threads like this usually get picked up quickly.

Opening it makes this entire thread public, including everything already posted. Nothing becomes public until you decide. Use the buttons below.

  • Open to the community - Before you click, take a moment to edit or remove anything you'd rather not share. The thread becomes publicly visible right away.
  • Keep it private and close the thread - Nothing becomes public and the thread closes.

Status changed to Awaiting User Response Railway • about 1 month ago


Railway
BOT

a month ago

This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.

Status changed to Open Railway • 30 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...