a month ago
Project: perfect-motivation (production environment)
Services affected: langfuse-web and clickhouse
Problem:
langfuse-web cannot reach clickhouse via private networking.
Both services are in the same project, same environment.
ClickHouse service is running and shows "Online" in the dashboard.
Connection attempts from langfuse-web to
clickhouse.railway.internal:8123 (HTTP) and
clickhouse.railway.internal:9000 (TCP) both time out.
Error from langfuse-web logs:
"error: failed to open database: dial tcp 10.209.29.64:9000:
i/o timeout"
And for HTTP queries:
"ClickHouse query failed with non-retryable error: Timeout error."
(via @clickhouse/client connecting to port 8123)
ClickHouse variables:
- PORT: 8123
- INTERNAL_PORT: 9000
- HOST: clickhouse.railway.internal
What I've tried:
- Restarted both services (no change)
- Redeployed both services (no change)
- ClickHouse shows clean startup logs (no errors)
- langfuse-web successfully connects to the Postgres
service in the same project (Prisma migrations succeed)
- Only ClickHouse connectivity fails
This was working previously (traces were recorded on April 16).
The issue appeared after a ClickHouse restart on April 17.
Could you investigate whether the private networking route
between these two services is properly established?
Pinned Solution
a month ago
Hello jturner-dev,
one thing stands out is your variables show PORT: 8123, INTERNAL_PORT: 9000, HOST: clickhouse.railway.internal, but there's no CLICKHOUSE_LISTEN_HOST set. by default clickhouse only binds to localhost, not to the railway private network interface. after a restart it won't accept connections from other services even if dns resolves correctly
try adding CLICKHOUSE_LISTEN_HOST=:: to your clickhouse service variables and redeploy it. the :: tells clickhouse to listen on all interfaces including the one railway assigns on the private network.
after redeploy check your clickhouse startup logs , you should see lines like "listening for http://[::]:8123" and "listening for native protocol tcp [::]:9000". if those aren't there, clickhouse isn't reachable from langfuse-web no matter what
Hope this help you :)
1 Replies
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
This thread has been marked as public for community involvement, as it does not contain any sensitive or personal information. Any further activity in this thread will be visible to everyone.
Status changed to Open Railway • about 1 month ago
a month ago
Hello jturner-dev,
one thing stands out is your variables show PORT: 8123, INTERNAL_PORT: 9000, HOST: clickhouse.railway.internal, but there's no CLICKHOUSE_LISTEN_HOST set. by default clickhouse only binds to localhost, not to the railway private network interface. after a restart it won't accept connections from other services even if dns resolves correctly
try adding CLICKHOUSE_LISTEN_HOST=:: to your clickhouse service variables and redeploy it. the :: tells clickhouse to listen on all interfaces including the one railway assigns on the private network.
after redeploy check your clickhouse startup logs , you should see lines like "listening for http://[::]:8123" and "listening for native protocol tcp [::]:9000". if those aren't there, clickhouse isn't reachable from langfuse-web no matter what
Hope this help you :)
Status changed to Solved brody • about 1 month ago