a month ago
We're seeing very slow (~139mS) DNS resolution when on the private internal network, this is particularly problematic between the web layer and the DB and is adding substantial latency to requests.
See the dig below. This seems to be intermittent and sometimes is quick, but most of the time suffers the delay of ~140mS, I presume this is quick when cached and a cache miss is the problem, any suggestion on how to resolve this?
root@7b5dcbe9ed20:/usr/src/app# dig AAAA mysql.railway.internal
; <<>> DiG 9.18.39-0ubuntu0.24.04.2-Ubuntu <<>> AAAA mysql.railway.internal
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 47565
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: c57ce1bc1071c2d1 (echoed)
;; QUESTION SECTION:
;mysql.railway.internal. IN AAAA
;; ANSWER SECTION:
mysql.railway.internal. 10 IN AAAA fd12:608:9de4:0:1000:22:d877:2ebe
;; Query time: 139 msec
;; SERVER: fd12::10#53(fd12::10) (UDP)
;; WHEN: Fri Jan 09 15:40:07 UTC 2026
;; MSG SIZE rcvd: 113
root@7b5dcbe9ed20:/usr/src/app# dig AAAA mysql.railway.internal
; <<>> DiG 9.18.39-0ubuntu0.24.04.2-Ubuntu <<>> AAAA mysql.railway.internal
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55495
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: e87fcac4496af02b (echoed)
;; QUESTION SECTION:
;mysql.railway.internal. IN AAAA
;; ANSWER SECTION:
mysql.railway.internal. 10 IN AAAA fd12:608:9de4:0:1000:22:d877:2ebe
;; Query time: 0 msec
;; SERVER: fd12::10#53(fd12::10) (UDP)
;; WHEN: Fri Jan 09 15:40:08 UTC 2026
;; MSG SIZE rcvd: 113
6 Replies
a month ago
Hey there! We've found the following might help you get unblocked faster:
If you find the answer from one of these, please let us know by solving the thread!
a month ago
My working assumption is the DNS is doing an authoritative lookup in the US every 2 seconds and is only locally cached in the NL/EU region, so when this isn't in the cache we end up having to do the round trip to the US for the DB connection to be established. Is this the case, or is there anything we could do to mitigate the issue
a month ago
If we move the container we're running the dns latency tests from to California this drops to about 19mS lookup every 2s, and ~59mS if run from US east, which backs up this theory.
a month ago
Hello,
The source of truth for the private network's DNS resolver is indeed only located in US-West, though, we do have caching in every region.
We are already well underway on both making this a globally distributed system and generally speeding it up.
Until then, you would want to maintain a connection pool to your database so that your application does not have to perform a DNS lookup in the hot path.
Best,
Brody
Status changed to Awaiting User Response Railway • about 1 month ago
a month ago
Hmm there’s quite a few services we’d have to connection pool, so I’m not sure how feasible this will be. Are the public connections the same or is there any way to use static ups.
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
Public connections would involve egress fees, and static IPs are not available for the private network, as service IPs are dynamic.
Status changed to Awaiting User Response Railway • about 1 month ago
a month ago
This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!
Status changed to Solved Railway • 27 days ago