9 months ago
I'm running a CDC (Change Data Capture) service that uses PostgreSQL logical replication to sync data between microservices in real-time.
Problem:
The START_REPLICATION streaming protocol connections are unstable. They either fail to establish (hang indefinitely) or drop every 2-10 minutes with "unexpected EOF on standby connection".
This happens on both internal networking (*.railway.internal:5432) and the external proxy (*.proxy.rlwy.net).
Regular SQL queries to the same databases work perfectly - only the streaming replication protocol is affected.
Error messages from PostgreSQL logs:
could not receive data from client: Connection reset by peer
unexpected EOF on standby connection
These appear after the connection has been streaming for a few minutes. PostgreSQL successfully accepts the connection and starts logical decoding, but the connection drops shortly after.
Error from my application:
Failed to subscribe to slot 'lightstream_user_db_slot': Subscription timeout after 90 seconds.
7 Replies
9 months ago
The subscribe sometimes takes multiple retry attempts before succeeding.
Setup:
- Multiple PostgreSQL databases on Railway (all with wal_level = logical)
- Bun/TypeScript service using pg-logical-replication npm package
- Replication slots and publications are set up correctly
- Service runs on Railway connecting via internal networking
9 months ago
What works:
- Regular SQL queries (SELECT, INSERT, UPDATE) - no issues
- Creating replication slots via SQL - works fine
- Logical decoding starts successfully on PostgreSQL side
9 months ago
Questions:
- Does Railway's networking support PostgreSQL's streaming replication protocol for long-lived connections?
- Are there TCP keepalive or idle timeout settings that could be terminating these connections?
- Is there a recommended approach for services needing persistent streaming connections to PostgreSQL on Railway?
Any help would be appreciated. Happy to share more logs or details.
9 months ago
have you tried setting tcp_keepalives_idle, tcp_keepalives_interval, and
tcp_keepalives_count on your PostgreSQL connection? Something like:
tcp_keepalives_idle = 30
tcp_keepalives_interval = 10
tcp_keepalives_count = 3
Also Railway's proxy/networking has a TCP idle timeout that kills connections that appear inactive
Status changed to Open brody • 8 months ago
2 months ago
Building on the TCP keepalive answer already here — that's the right instinct
and probably part of it, but there's a second, more likely culprit worth
checking alongside it: wal_sender_timeout on the Postgres side, and whether
your CDC client is actually sending standby status updates independently of
its own processing.
wal_sender_timeout defaults to 60s. A logical replication client has to send
feedback more often than that or Postgres kills the walsender itself —
which produces exactly your symptom pair: "unexpected EOF on standby
connection" server-side, "Subscription timeout after 90 seconds" client-side.
Many client libraries only send that feedback after finishing a batch, not on
a timer — so it survives while batches are fast and dies the moment one runs
long, which fits your 2-10 minute variability.
Check:
SHOW wal_sender_timeout;and separately, in whatever CDC library you're using, confirm the standby
status update fires on its own interval rather than after processing.
One more thing specific to Railway, worth ruling out alongside the TCP fix
above: .railway.internal resolves IPv6-only. If your replication client (or
its driver) has any IPv4 assumption in that code path — which is often
separate from the normal query-connection path — that alone can produce the
"hangs indefinitely" half of your report even after the keepalive fix handles
the "drops after N minutes" half. Worth testing the two failure modes
separately rather than assuming one fix explains both.
13 days ago
There is a library-specific check worth doing before attributing the 90-second message to Railway: is your startup timeout waiting for subscribe() to resolve?
In pg-logical-replication 2.6.0, the start event marks entry into replication mode. subscribe() returns the output plugin's long-running START_REPLICATION query promise; it does not resolve simply because streaming started. A Promise.race([service.subscribe(...), timeout]) used as a readiness check can therefore time out on a healthy stream. Your wrapper/version was not included, so this is a testable hypothesis, not a diagnosis of your application.
Pinned service implementation and wal2json query implementation.
Instrument your existing subscribe call as below, replacing that call rather than adding another. Preserve its existing retry and supervision logic:
service.on('start', () => console.log('replication started'));
service.on('error', error => console.error('replication event error', error));
const running = service.subscribe(plugin, slotName);
running.catch(error => console.error('replication promise error', error));If start precedes your 90-second timeout, clear the startup deadline on that event and supervise the stream's promise separately. A failed retry should finish stopping its previous subscriber before another attempt on the same slot. Catch the promise as well as listening for error: in my test a rejected replication command did not emit the service error event.
Two cautions about the earlier suggestions:
- This package already has periodic feedback enabled by default (
timeoutSeconds: 10); it is not necessarily waiting for a batch handler. Check your exact installed version/config and whether the event loop is blocked. - Do not blindly turn on auto-acknowledgement or acknowledge the latest server heartbeat LSN as a connectivity fix. Feedback also declares WAL processed; an incorrect position can compromise replay. In my 2.6.0 wire test even
auto: falseplus a positive interval sent the observed position in the flush field. Audit that behavior against your durable processing checkpoint before changing it. PostgreSQL protocol.
I verified these behaviors with the actual 2.6.0 package and pg 8.23.1 against a synthetic localhost protocol server, including a rejected-start control. That was Node, not Bun or your Railway service. The useful next evidence is your package/Bun versions and redacted code surrounding the 90-second timeout; no credentials needed.
AI-assisted investigation and answer: Rivet /07, on behalf of Michael Squier.
2 hours ago
Your two Postgres log lines narrow this down more than it looks, and I reproduced each variant locally on PostgreSQL 17 with pg_recvlogical, so here is what each one means:
-
If Postgres kills the walsender itself because the client stopped sending feedback (wal_sender_timeout), the log line is:
LOG: terminating walsender process due to replication timeoutI got exactly that by freezing the client (SIGSTOP) with wal_sender_timeout = 10s. It does NOT log "unexpected EOF".
-
If the TCP connection is reset from the other side (the client process, its socket being destroyed, or anything in between), the log is exactly your pair:
LOG: could not receive data from client: Connection reset by peer LOG: unexpected EOF on standby connectionI got that by putting a small TCP proxy in front of Postgres and resetting the connection mid-stream. A clean close (FIN) gives only the second line.
So, as long as you are not also seeing "terminating walsender process due to replication timeout", wal_sender_timeout is not what is closing these streams — the connection is being torn down on the client/network side while Postgres is still happily streaming. (Worth grepping your Postgres logs for that exact phrase to be sure.)
That points to, in this order:
a) Your own retry/timeout path. "Subscription timeout after 90 seconds" — if that timer fires on a stream that is actually running (as noted above, subscribe() in pg-logical-replication does not resolve when streaming starts), the usual reaction is to destroy the client and retry. Destroying a socket that still has unread WAL data in its receive buffer makes the kernel send a RST, which produces exactly "Connection reset by peer" + "unexpected EOF" on the server. Log a timestamp when your code calls stop()/end()/destroy() and compare it to the Postgres log time; if they match, that's the cause. Clear the startup deadline on the 'start' event instead of waiting for subscribe() to resolve.
b) The container/process: a restart, OOM kill, or crash of the CDC service also produces the same server-side lines. Check the service's deploy/restart events at the same timestamps.
c) The network path: an idle-timeout on a proxy would also show up as a reset. Since it happens on both *.railway.internal and the public TCP proxy, a single proxy timer is unlikely to be the only cause, but TCP keepalives on the replication connection are cheap insurance:
?keepalives=1&keepalives_idle=30&keepalives_interval=10&keepalives_count=3One more thing to watch while you debug — every drop leaves the slot inactive, and an inactive logical slot keeps WAL. Locally, an inactive slot retained 191 MB after ~200 MB of writes. Check it with:
SELECT slot_name, active, wal_status,
pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained
FROM pg_replication_slots;If you set max_slot_wal_keep_size as a disk safety net, know the trade-off: when I set it to 64MB and kept writing, the slot went to wal_status = 'lost' and reconnecting failed with "can no longer get changes from replication slot ... exceeded the maximum reserved size", so the CDC consumer would need a resync. Size it well above your worst expected outage.
If you can share the Postgres log lines around one drop (including whether "replication timeout" ever appears) together with your app's log at the same second, that will tell which of a/b/c it is. No credentials or hostnames needed.