9 days ago
We're running the official Opik (self-hosted) It deploys as several services, including Opik Gateway (Caddy reverse proxy, public-facing) and Opik Backend (internal API service, no public domain, reached only via opik-backend.railway.internal:8080).
We had a custom domain attached to Opik Frontend and moved it to Opik Gateway via the dashboard on 2026-08-11 to put it behind Gateway's Basic Auth. Since that move, Opik Gateway's reverse proxy to Opik Backend over the private network has stopped completing every request to /api/* hangs until client timeout. Requests Caddy serves itself (root path) or proxies to Opik Frontend still work fine. Backend itself is healthy and receiving no request at all during the hang. Restarting both services (several times, with waits up to ~12 minutes) has not fixed it.
Error messages and descriptions:
No error is returned to the client — the connection simply hangs until timeout (curl exits with code 28, "Operation timed out after 15000 milliseconds"). No error appears in either service's logs for the failed request. There is no HTTP status code at all (curl reports HTTP:000).
Logs:
Gateway, root path request (works):
HTTP:200 time:0.748946s
Gateway, /api/is-alive/ping request (hangs, no server-side error, no log line at all for this request):
HTTP:000 time:15.006372s
curl: (28) Operation timed out after 15000 milliseconds
Backend logs during the same window only its own internal scheduled jobs, no incoming HTTP request logged:
INFO [2026-08-12 07:13:07,083] com.comet.opik.api.resources.v1.jobs.ExperimentDenormalizationJob: Starting experiment denormalization job - checking for pending experiments
INFO [2026-08-12 07:13:07,084] com.comet.opik.domain.AlertServiceImpl: Fetching all enabled alerts for workspace 'null', eventTypes '[TRACE_ERRORS, TRACE_FEEDBACK_SCORE, TRACE_THREAD_FEEDBACK_SCORE, TRACE_COST, TRACE_LATENCY]'
INFO [2026-08-12 07:13:07,119] com.comet.opik.api.resources.v1.jobs.TraceThreadsClosingJob: Successfully completed closing trace threads process
(This pattern repeats every 5s with no interruption or error, before, during, and after the failed request confirming the request never reaches Backend.)
Verified config (all correct, ruled out as cause):
OPIK_BACKEND_UPSTREAM = http://opik-backend.railway.internal:8080
Backend startup log confirms it listens on:
Started application@...{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
Template: Opik (self-hosted) official Railway template.
6 Replies
9 days ago
Having looked into this, the issue appears to be in your application code or configuration rather than the Railway platform itself, which puts it outside what Railway support can resolve directly.
This is exactly the kind of problem the Railway community is good at, so we'd like to open your thread as a community bounty. Railway pays a bounty to the community member who solves it, and threads like this usually get picked up quickly.
Opening it makes this entire thread public, including everything already posted. Nothing becomes public until you decide. Use the buttons below.
- Open to the community - Before you click, take a moment to edit or remove anything you'd rather not share. The thread becomes publicly visible right away.
- Keep it private and close the thread - Nothing becomes public. The thread closes, since this isn't something Railway support can take further.
Status changed to Awaiting User Response Railway • 9 days ago
9 days ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • 9 days ago
9 days ago
In the gateway service, are you able to resolve opik-backend.railway.internal:8080 via the terminal from the Console tab?
0x5b62656e5d
In the gateway service, are you able to resolve `opik-backend.railway.internal:8080` via the terminal from the Console tab?
9 days ago
DNS resolves fine opik-backend.railway.internal → Railway's private IPv6 network. So it's not a DNS problem.
Railway
Having looked into this, the issue appears to be in your application code or configuration rather than the Railway platform itself, which puts it outside what Railway support can resolve directly. This is exactly the kind of problem the Railway community is good at, so we'd like to open your thread as a community [bounty](https://docs.railway.com/community/bounties). Railway pays a bounty to the community member who solves it, and threads like this usually get picked up quickly. **Opening it makes this entire thread public**, including everything already posted. Nothing becomes public until you decide. Use the buttons below. - **Open to the community** - Before you click, take a moment to edit or remove anything you'd rather not share. The thread becomes publicly visible right away. - **Keep it private and close the thread** - Nothing becomes public. The thread closes, since this isn't something Railway support can take further.
9 days ago
Update: root-caused it. Ran nslookup opik-frontend.railway.internal from inside the Gateway container got two different answers for the same hostname:
Address: fd12:e686:8ca2:1:8000:ea:ff96:559c ← correct, live
Address: 10.150.85.156 ← stale, dead
Our proxy dials the stale IPv4 (A) address and fails (dial tcp 10.150.85.156:5173: connect: connection refused) instead of falling back to the working IPv6 (AAAA) one. Reran nslookup right after restarting both services same stale A record came back both times, so it's not something a restart clears on our end. Looks like your internal DNS is serving a stale A record for this private hostname after a reschedule.
9 days ago
Thanks checked that. Backend is bound to :: (IPv6 wildcard, dual-stack), not IPv4-only, so it accepts both address families correctly confirmed via /proc/net/tcp6 showing the LISTEN on port 8080, /proc/net/tcp empty. That part's fine.
The actual failure isn't Backend's bind address though it's a different service, opik-frontend, on port 5173. Our gateway is failing to connect to it with connection refused on a specific IP, and nslookup from the gateway container shows DNS returning two different addresses for that hostname: a correct/live IPv6 one and a stale IPv4 one that nothing's listening on anymore. So it looks like a stale DNS record for the frontend service, not a backend binding issue.
9 days ago
No I am not using go
Status changed to Solved bruh-codes • 9 days ago