4 months ago
TLDR: using the internal pg host induced a 100ms penalty on queries compared to the external host.
I recently set up a PG Bouncer template instance and noticed weird query duration behavior when using it. Whenever I would load it up with several medium-sized queries it would tack on an additional 100-120ms latency per query, even for smaller 'SELECT 1' queries I ran after. I ended up diagnosing the issue as just to do with the internal TCP connection between the bouncer instance and my Postgres HA cluster. As soon as I switched the bouncer database host from the Postgres HA internal host to the external one (incurring extra egress costs) I noticed the problem go away instantly. All instances that are interacting are in US East region, so there is no cross-region penalty.
Given that I have solved this for my own use case and the egress costs are not currently a concern, I don't need immediate attention on this. However I consider this to be pretty concerning that internal networking is slower than external and wanted to open a ticket to eventually having this fixed.
3 Replies
Status changed to Awaiting Railway Response Railway • 4 months ago
4 months ago
Thanks for the detailed report, we'll track the issue as a data point for the networking team. One thing worth checking: if your environment was created before October 16, 2025, it uses IPv6-only private networking, which can compound this issue. Creating a new environment would give you dual-stack (IPv4/IPv6) private networking and may reduce the latency.
Status changed to Awaiting User Response Railway • 4 months ago
4 months ago
This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!
Status changed to Solved Railway • 4 months ago
4 months ago
Hi, sorry for the no reply I didn't see a notification for this response. It was created in 2026 so no concern about IPv6 only networking. I'm actually also experiencing this with another service besides PG Bouncer. I have a Go service running in a container that was connecting to the database URL via internal networking, and was incurring a 100-135ms penalty per query for cross-region and several ms per query for in-region. Now that I'm using external networking I see a significantly reduced penalty for cross-region (upwards of 80ms from limited data collection), and a slightly reduced penalty for same-region as well.
I would really love to be able to provide any more information for debugging or identifying the issue. I am happy to collect any data internal to our services to support identifying this issue.
Thanks again for your time and support!
Status changed to Awaiting Railway Response Railway • 4 months ago
4 months ago
This thread has been marked as public for community involvement, as it does not contain any sensitive or personal information. Any further activity in this thread will be visible to everyone.
Status changed to Open Railway • 4 months ago
13 days ago
The 2026 environment rules out the legacy IPv6-only explanation. The next useful step is to determine whether the extra time is paid once per connection or on every round trip, and whether it follows one private address family. That turns this from a general latency report into a route Railway can trace.
Run the same matrix from one fixed application replica against the same Postgres primary. Keep credentials in environment variables and do not paste connection strings:
-
Record the actual placement at runtime from both services, not only the configured region:
RAILWAY_REPLICA_REGION,RAILWAY_REPLICA_IDandRAILWAY_DEPLOYMENT_ID. Railway exposes these specifically for per-replica correlation. Confirm the PgBouncer/Go client and Postgres primary report the expected regions. -
Resolve the private hostname once with
getent ahosts <postgres>.railway.internal. In a new 2026 environment it can return both IPv4 and IPv6. Test each returned address separately only for diagnosis; with libpq you can sethost=<private-hostname>plushostaddr=<resolved-address>so hostname/TLS behavior is preserved while selecting the route. Do not hard-code a private address as the production fix because it can change. -
Split connection setup from query RTT. With
pgbenchavailable, run a persistent-session read-only test and then the same test with-C(new connection per transaction), first over private networking and then over the TCP proxy:pgbench "$PRIVATE_DATABASE_URL" -S -T 60 -P 5 -c 1 -j 1 pgbench "$PRIVATE_DATABASE_URL" -S -T 60 -P 5 -c 1 -j 1 -C pgbench "$PUBLIC_DATABASE_URL" -S -T 60 -P 5 -c 1 -j 1 pgbench "$PUBLIC_DATABASE_URL" -S -T 60 -P 5 -c 1 -j 1 -CUse sealed variables and redact command output that contains endpoints. If only
-Cis slow, the penalty is DNS/address selection, TCP/TLS or authentication/pool setup. If a single persistent session still adds ~100 ms to everySELECT 1, the private data path itself is adding RTT. -
During the persistent test, capture kernel TCP measurements for the established database socket with
ss -tin(redact credentials; IP/RTT/cwnd/retransmit data is enough). Repeat for private IPv4, private IPv6 and the public proxy. Also record PgBouncerSHOW STATS;/SHOW POOLS;before and after so queue wait or pool saturation is not mistaken for network RTT. -
Run the matrix once in-region and once cross-region with the same client image, pool settings and query count. Report median/p95, connection rate, TCP RTT/retransmits, selected address family and both replica IDs. A slow private family with the other private family fast identifies a dual-stack/WireGuard path; both private families slow on a persistent session while the public proxy is fast identifies the private-mesh route; only new connections slow identifies client/pool setup.
Railway currently documents private service traffic as an encrypted WireGuard mesh scoped to one project/environment and recommends it as the lower-latency, no-egress path. Therefore the public TCP proxy should not be the permanent workaround if this controlled persistent-session test proves the opposite. Send Railway the paired UTC window, deployment/replica IDs, actual source/destination regions, private address family, TCP RTT/retransmits and pgbench summaries. That is sufficient for the networking team to locate the tunnel/route without exposing database credentials or application data.
Official references: