Private Network Instability: Internal Service Only Reachable 1/10 Times
sylver911
PROOP

2 months ago

Hey! I’m running into a really strange issue with two services on the private network in a brand-new Railway environment. My trading bot calls IBeam through:

http://ibeam-deploy.railway.internal:5000

IBeam is definitely listening on IPv4:5000 — the public IP works instantly every time. But internally, the private domain behaves completely inconsistently.

Sometimes the trading service can reach it with no issues at all. For example, here’s a log from a deploy where everything worked perfectly:

2025-12-03 00:18:25 [INFO] trading_service: IBKR URL: http://ibeam-deploy.railway.internal:5000
2025-12-03 00:18:25 [INFO] trading_service: Account: **********
2025-12-03 00:18:25 [INFO] trading_service: Queue stats: {'pending': 0, 'processing': 0, 'completed': 87, 'failed': 11}
2025-12-03 00:18:25 [INFO] trading_service: Starting consumer loop (BRPOP)...

But then — with the exact same config — literally two deploys earlier I got nothing but timeouts, as if the IBeam service wasn’t even reachable on the private network:
2025-12-02T23:22:27.860451108Z [inf] 2025-12-02 23:22:21 [INFO] trading_service: IBKR URL: http://ibeam-deploy.railway.internal:5000 2025-12-02 23:22:31.885405216Z [err] IbkrClient: Connection error detected, retrying attempt 1/3 :: HTTPConnectionPool(host='ibeam-deploy.railway.internal', port=5000): Max retries exceeded with url: /v1/api/tickle (Caused by ConnectTimeoutError(, 'Connection to ibeam-deploy.railway.internal timed out. (connect timeout=10)')) 2025-12-02 23:22:51.885723508Z [err] IbkrClient: Connection error detected, retrying attempt 2/3 :: HTTPConnectionPool(host='ibeam-deploy.railway.internal', port=5000): Max retries exceeded with url: /v1/api/tickle (Caused by ConnectTimeoutError(, 'Connection to ibeam-deploy.railway.internal timed out. (connect timeout=10)')) 2025-12-02 23:22:56.685309728Z [err] IbkrClient: Connection error detected, retrying attempt 3/3 :: HTTPConnectionPool(host='ibeam-deploy.railway.internal', port=5000): Max retries exceeded with url: /v1/api/tickle (Caused by ConnectTimeoutError(, 'Connection to ibeam-deploy.railway.internal timed out. (connect timeout=10)')) 2025-12-02 23:23:16.108250718Z [err] IBKR health check failed: IbkrClient: Connection error HTTPConnectionPool(host='ibeam-deploy.railway.internal', port=5000): Max retries exceeded with url: /v1/api/tickle (Caused by ConnectTimeoutError(, 'Connection to ibeam-deploy.railway.internal timed out. (connect timeout=10)')) for POST http://ibeam-deploy.railway.internal:5000/v1/api/tickle {} 2025-12-02 23:23:16.108256553Z [inf] 2025-12-02 23:23:11 [INFO] trading_service: Queue stats: {'pending': 0, 'processing': 1, 'completed': 86, 'failed': 11}

All I did between these deploys was restart the service and tweak an env var (the one containing the private domain). No actual configuration changes. Sometimes after that the service becomes reachable, sometimes it stays unreachable.

So the exact same internal hostname works once, then completely stops working, then works again after a random env-var update. I can’t find any pattern at all.

Is there any known issue where a *.railway.internal IPv4 service is only intermittently reachable?
Does railway.internal have any DNS or IP caching that could cause this kind of inconsistent behavior?

Any insights would be hugely appreciated — this one has been driving me crazy. 🙏

Solved

3 Replies

brody
EMPLOYEE

2 months ago

Hey @Sylver - We have just pushed a fix for this, could you please try again?


sylver911
PROOP

2 months ago

it seems like now it is more stable. ill do a couple of pushes now into my project to redeploy the service, let me test this. thank u very much for your fast response! ❤️


sylver911
PROOP

2 months ago

Hi today it is dying again. I can reach my service from public domain but internally i get timeout from every service


Status changed to Solved samgordon 2 months ago


Loading...