a month ago
Hi Railway Support,
We are experiencing intermittent severe latency on our production project.
Project: risk-ai
Environment: production
Region shown in responses: europe-west4-drams3a
Affected services:
api
Custom domain: https://api.riskai.com.ar
Railway domain: https://risk-ai-production.up.railway.app
Frontend
Custom domain: https://app.riskai.com.ar
Railway domain: https://frontend-production-6d30.up.railway.app
Important context:
The issue is intermittent.
It affects multiple users/locations.
The API process is up and deployments show SUCCESS.
PostgreSQL was checked during the incident: no long-running queries, no locks, no obvious DB saturation at the sampled times.
Our /healthz endpoint is intentionally lightweight and does not touch the database. It only returns a JSON OK response from the API process.
Despite that, /healthz sometimes takes 4-16 seconds or times out.
The static frontend root also sometimes takes 4-17 seconds, including the Railway service domain directly.
Examples we captured:
API Railway domain:
https://risk-ai-production.up.railway.app/healthz
Observed timings:
total: 16.20s, code 200
total: 11.70s, code 200
sometimes timeout in probes
API custom domain:
https://api.riskai.com.ar/healthz
Observed timings:
total: 8.7s, code 200
total: 4-8s frequently during bad windows
Frontend Railway domain:
https://frontend-production-6d30.up.railway.app/
Observed timings:
total: 13.9s, code 200
total: 12.0s, code 200
total: 17.1s, code 200
Some response headers from slow API requests:
Server: railway-hikari
x-railway-edge: railway/europe-west4-drams3a
x-hikari-trace: waw1.qkma
x-railway-request-id: XIQVH9BqRwG2ElZVjUJq2g
x-railway-request-id: HyTc32UDRm-wUEhRjUJq2g
What we need help with:
Please check if there is an issue with Railway edge/proxy/routing for this project or region.
Please check whether the service instances are experiencing intermittent cold starts, network stalls, or request queueing despite showing SUCCESS.
Please confirm if there were any platform incidents or degraded routing around europe-west4-drams3a.
Please advise if moving regions or enabling multiple replicas is recommended.
This is production-impacting. Users are seeing timeouts and blank screens even in incognito/private browsing. Since both the lightweight API health endpoint and static frontend responses are slow, this does not look like a database-query-only issue.
Thanks.
7 Replies
a month ago
We can see both services in the risk-ai project are deployed and showing SUCCESS in production. To help us narrow down whether the latency is in the network/edge path or in the application itself, please check the HTTP logs for your API service in the Log Explorer using the filter @totalDuration:>1000 @path:/healthz and compare the totalDuration vs upstreamRqDuration values. If totalDuration is high but upstreamRqDuration is low, the latency is in the network path rather than your application, which would confirm a platform-side issue we need to investigate further. Please share those values so we can proceed with the right diagnosis.
Status changed to Awaiting User Response Railway • 27 days ago
a month ago
ymptom: The /healthz endpoint of our API service responds in 3–17 seconds (HTTP 200), intermittently timing out at 20–41s, while the container is idle on all resources. This is a trivial endpoint that does no DB access, no auth, no I/O — it just returns Ok({ status, timestamp }). Baseline for this endpoint/pipeline is ~600 ms.
Key point — it is NOT resource-bound on our side:
Metric (sampled concurrently, last 20 min) Value
API container CPU avg 9%, max 33%
API container Memory avg 0.94 GB, max 1.13 GB
Postgres active connections 1 (idle)
Postgres locks / slow queries 0 / 0
Timestamped /healthz latency (UTC) with concurrent Postgres state:
ts_utc | status | healthz_ms | pg_active | pg_locks
21:37:26 | 200 | 8131 | 1 | 0
21:37:38 | 200 | 6839 | 1 | 0
21:37:46 | 200 | 3760 | 1 | 0
21:37:59 | 200 | 9655 | 1 | 0
21:38:12 | 200 | 8391 | 1 | 0
21:38:40 | 200 | 14439 | 1 | 0
21:39:28 | 200 | 17338 | 3 | 0
21:39:37 | 200 | 5374 | 1 | 0
Context:
Persists across multiple deployments (code-independent) — including a rollback to a prior image.
Single replica (numReplicas = 1).
No healthcheck configured (healthcheckPath = null), so this is not healthcheck-induced restarting.
Question for Railway: With the container CPU/memory idle and Postgres idle, what is adding 3–17s to a no-op HTTP request? Please check edge/proxy routing latency, request queueing, or networking between the edge and this container.
IDs:
project: 017eff7f-d197-437b-9d8a-5223fc14f31e
environment (production): a13ed6da-17e4-42c9-8957-1e117be6f422
service (api): 5b063a17-33fa-4bf7-acd6-fac4fbcbdc5c
active deployment: f7d48725 (2026-06-10T21:28:31Z)
Status changed to Awaiting Railway Response Railway • 27 days ago
a month ago
Thanks for the detailed breakdown. With your container and database both idle and a no-op health endpoint plus static frontend still taking several seconds across both your Railway and custom domains, the delay is in the network path between our edge and your container, not in your application. Rolling back and redeploying would not change this, which lines up with what you are seeing. We are investigating this on our side, and your project, region, and request IDs are included in that work. Adding a second replica can reduce the odds that a single affected path is hit, so it is worth trying as a partial mitigation, but it will not fully resolve an edge-side delay. We will follow up on this thread as the investigation progresses.
Status changed to Awaiting User Response Railway • 26 days ago
a month ago
Thanks noahd, this matches our own findings — appreciate you investigating the edge-side path. To confirm on our end: we reproduced the no-op /healthz slowness from neutral external probes (check-host nodes worldwide returned 0.1–0.3s) while our local Argentine ISP saw 3–28s with the time spent in TCP/TLS handshake before reaching the container, and it hit the static frontend domain identically — consistent with an edge/transit issue rather than our app.
On the second replica: we want it both for this and for restart resilience, but our API service currently has a 5.5GB volume mounted (/app/uploads), and Railway blocks replicas while a volume is attached. We're planning to migrate that storage to object storage so we can enable replicas — once that's done we'll turn on the second replica as you suggested and report back whether it mitigates the edge latency.
Please keep us posted on the investigation. Project/region/request IDs are already on the thread. Thanks again
Status changed to Awaiting Railway Response Railway • 26 days ago
a month ago
That's correct, replicas cannot be used with volumes, so migrating /app/uploads to object storage is the right path to unblock scaling. We offer native S3-compatible Storage Buckets at $0.015/GB-month with free egress and unlimited API operations, which would be a direct fit for that uploads directory without needing an external provider.
Status changed to Awaiting User Response Railway • 26 days ago
Railway
That's correct, replicas cannot be used with volumes, so migrating `/app/uploads` to object storage is the right path to unblock scaling. We offer native S3-compatible [Storage Buckets](https://docs.railway.com/storage-buckets) at $0.015/GB-month with free egress and unlimited API operations, which would be a direct fit for that uploads directory without needing an external provider.
a month ago
Please keep us posted on the investigation
Status changed to Awaiting Railway Response Railway • 26 days ago
24 days ago
Your investigation is still active on our side, and your project, region, and request IDs are attached to it. We do not have a resolution timeline to share yet, but we will follow up on this thread as soon as we have an update. Migrating /app/uploads to object storage to unblock a second replica is still worth doing for resilience, though it will not by itself resolve the network-path delay you have isolated.
Status changed to Awaiting User Response Railway • 24 days ago