7 days ago
Service: tradingOS (api), project "tradingOS", service ID 11de9e83-6fc0-419f-932c-38685ea2d1c2
Database: Postgres plugin, same project, EU West (Amsterdam) — same region as the api service
Issue: The api service intermittently returns 502 "Application failed to respond" and,
when it does respond, takes 8-14+ seconds per request (confirmed via /health, which has
no logic beyond returning a static JSON body). This happens whether DATABASE_URL points
at the internal reference (${{Postgres.DATABASE_URL}}, postgres.railway.internal) or the
public proxy URL (switchyard.proxy.rlwy.net:28918) — same slowness/timeouts either way.
What I've ruled out:
-
App code: identical code, pointed at the same live Postgres database (via the public
proxy URL), starts in under 1 second and serves requests instantly when run locally
(outside Railway entirely) from my own machine.
-
Database health: connected directly via psql from my own machine to the same public
proxy URL — instant connection, clean pg_stat_activity (no stuck sessions/locks), all
122 tables present, alembic_version at the correct head revision (migrations succeeded).
-
Resources: Metrics tab shows flat ~90MB memory and ~0% CPU during the slow window — not
crashing, OOMing, or spinning.
-
Region mismatch: both the api service and Postgres are in EU West (Amsterdam).
-
Deploy logs: the container starts, runs the pre-deploy alembic migration successfully
(confirmed via direct DB check), but nothing further is logged even after several
minutes, and the app is unreachable from outside for that whole window.
Evidence from the request-metrics graphs on the tradingOS service (visible in-app):
Response Time graph shows a sustained p99 of ~9-14s over a ~30 minute window
(around 20:00-20:47 in the service's local deploy-log timestamps), not a single spike —
this was the consistent baseline, not an outlier.
This points at something in Railway's own network path for outbound connections
initiated from inside an app container in this project/region, not at application code,
database health, or configuration on our end. Happy to provide direct psql access
details or run any specific diagnostic you'd want from inside the container via the
Console tab.
6 Replies
7 days ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • 7 days ago
7 days ago
Is the port your application is listening to the same port your URL is mapped to?
7 days ago
Thanks for the quick response. I tested that specifically:
-
Explicitly added a PORT=8000 variable to the service (previously nothing named PORT
was set explicitly, though the app's Dockerfile CMD does read $PORT dynamically with
an ENV PORT=8000 fallback default).
-
Confirmed the Networking target port is also set to 8000, matching.
-
Redeployed (deployment ef7aee03, 2026-08-13 21:37 PDT).
Same result: GET /health (which has no logic beyond returning a static JSON body,
no database call) still times out after 12-15s from outside, and the deploy logs show
nothing past the pre-deploy migration step's two INFO lines -- no uvicorn startup lines
at all, even though the same container image runs and logs normally within under a
second when I run it locally against the same live database.
So port mismatch looks ruled out. Given what I already sent -- database confirmed
healthy via direct psql connection (instant, no locks, correct migration state), app
code confirmed correct (runs and logs normally outside Railway against the same
database), and now port config confirmed matching -- I think this points to something
in the container's actual outbound network path or process supervision on Railway's
side for this specific deployment, rather than app config. Could someone take a look
at the raw container logs/state directly, or let me know if there's a way to get more
verbose output from the platform side than what's in the Deploy Logs tab?
Service: tradingOS (11de9e83-6fc0-419f-932c-38685ea2d1c2)
Latest affected deployment: ef7aee03 (2026-08-13 21:37 PDT)
7 days ago
Can you share your Dockerfile? (As well as any custom pre-deploy/deploy commands you have set up in Railway)
7 days ago
Sure — here's the Dockerfile (apps/api/Dockerfile):
FROM python:3.14-slim AS builder
COPY --from=ghcr.io/astral-sh/uv:0.7 /uv /uvx /bin/
WORKDIR /app
ENV UV_COMPILE_BYTECODE=1 UV_LINK_MODE=copy UV_PYTHON_DOWNLOADS=never
COPY pyproject.toml uv.lock ./
RUN uv sync --frozen --no-install-project --no-dev
COPY . .
RUN uv sync --frozen --no-dev
FROM python:3.14-slim AS runtime
WORKDIR /app
RUN useradd --create-home --shell /bin/bash appuser
COPY --from=builder --chown=appuser:appuser /app /app
ENV PATH="/app/.venv/bin:$PATH"
USER appuser
EXPOSE 8000
ENV PORT=8000
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
CMD python -c "import os, urllib.request; urllib.request.urlopen(f'http://localhost:{os.environ.get(\"PORT\", \"8000\")}/health', timeout=3)" || exit 1CMD uvicorn tradingos_api.main:app --host 0.0.0.0 --port "$PORT"
Railway service settings for this service (tradingOS):
- Builder: Dockerfile (path: apps/api/Dockerfile)
- Pre-Deploy Command: alembic upgrade head
- Start Command: none set (uses the Dockerfile's own CMD above)
- Root Directory: apps/api
For context, apps/api/Dockerfile's HEALTHCHECK hits GET /health on the same container
(localhost), and that endpoint has zero logic beyond returning a static JSON body -- no
database call, no external dependency. So even the container's own internal healthcheck
should be near-instant if the process is actually listening and responsive.
7 days ago
I'd try removing Dockerfile's healthcheck entry for now and see what happens.
7 days ago
Tried that too — removed the Dockerfile's HEALTHCHECK entirely (deployment 9d2b7407,
2026-08-13 21:53 PDT, confirmed via a fresh git push + redeploy). Same result: GET /health
still times out after 15s from outside with no response at all, and the deploy logs still
show nothing past the pre-deploy migration's two INFO lines -- no uvicorn startup output.
So to summarize what's been ruled out on our end across this thread:
-
Database: confirmed healthy via direct psql connection (instant, no locks, correct
migration state, all tables present)
-
App code: confirmed correct -- runs and serves requests in under a second locally,
against this exact same live database
-
Port config: explicitly pinned PORT=8000 as a service variable, matching the
Networking target port -- no mismatch
-
Docker-level HEALTHCHECK: removed entirely -- no change in behavior
-
Region: api service and Postgres are both in EU West (Amsterdam) -- no mismatch
-
Resource usage: Metrics tab shows flat ~90MB memory, ~0% CPU during the failure
window -- not crashing, not OOMing, not spinning
At this point I don't have another app-level or config-level lever left to pull. Could
someone look at this from the platform side -- e.g. whatever's between the edge and the
container for this specific deployment/service?
Service: tradingOS (11de9e83-6fc0-419f-932c-38685ea2d1c2)
Latest deployment: 9d2b7407 (2026-08-13 21:53 PDT)