Sudden latency in the web interface service (MonitorJogosWeb) with no code changes.
raularaujofaria
HOBBYOP

10 days ago

We are experiencing unexpected HTTP latency on our MonitorJogosWeb service (Python Flask application) in production environment.

Problem Details:

Service started responding slowly from August 10th to August 11th without any code changes

HTTP response times are consistently high:

p50 (median): 5 seconds

p90: 7-11 seconds

p95: 7-11 seconds

p99: 7-11 seconds

Requests return HTTP 200 OK but take 7-11+ seconds to complete

Some specific API calls take up to ~11 seconds (e.g., dashboard queries with date_ini=2026-08-11)

Infrastructure Status (normal):

CPU usage: 0.001 vCPU (well below limits)

Memory usage: 0.042GB (normal)

Network: Normal traffic, no packet loss

Postgres database: 0.094GB of 8GB, CPU < 0.004 vCPU (healthy)

Background worker (monitor-jogos): Idle, no errors

No HTTP 5xx errors

No critical issues reported

Actions Taken:

Restarted Postgres service at 13:22 UTC on 2026-08-11

Problem persists after restart

Suspicion: This appears to be an infrastructure-level issue, possibly related to networking or connectivity between services, as application metrics and resources are normal.

Project: Monitor de jogos automático

Service: MonitorJogosWeb

Environment: Production

Date/Time: 2026-08-10 to 2026-08-11

Database Connectivity Suspicion: "We suspect the latency may be related to the database connection between MonitorJogosWeb and Postgres, as this service queries the database directly for all API endpoints. Despite Postgres showing normal CPU and memory metrics, there could be underlying connection pool exhaustion, network latency, or locking issues at the database level."

image.png

Attachments

$10 Bounty

8 Replies

Railway
BOT

10 days ago

This thread has been opened as a bounty so the community can help solve it.

Status changed to Open Railway 10 days ago


manuproject
HOBBY

10 days ago

worth checking your region first, quick question: is MonitorJogosWeb deployed in US West (California)? theres a confirmed railway incident there, "connectivity issues in us west," resolved 03:39 utc on aug 10 https://status.railway.com/incident/RL8FRJE6, right around when your latency started. their note says "a small number of services may need a redeploy to fully recover," and you only restarted postgres, not the web service itself, worth trying a full redeploy of MonitorJogosWeb if you are in that region.

if youre not in us west, this specific incident doesnt apply, and id look harder at your own connection pool theory instead, worth sharing your db pool size/timeout config


raularaujofaria
HOBBYOP

10 days ago

Thanks. I checked the deployment region and MonitorJogosWeb is deployed in US East (Virginia), so the US West incident does not apply to this service.

There were no code, configuration, or infrastructure changes on our side. The latency started suddenly from yesterday to today, without any changes to the application or database.

Do you have any other known or ongoing issue affecting web services or PostgreSQL connectivity in US East that could explain this sudden latency?


raularaujofaria

Thanks. I checked the deployment region and MonitorJogosWeb is deployed in US East (Virginia), so the US West incident does not apply to this service. There were no code, configuration, or infrastructure changes on our side. The latency started suddenly from yesterday to today, without any changes to the application or database. Do you have any other known or ongoing issue affecting web services or PostgreSQL connectivity in US East that could explain this sudden latency?

manuproject
HOBBY

10 days ago

checked railways full status history for august, the only logged incident all month was that us west one, nothing for us east in this window. so no, i cant point you to a confirmed platform incident for this.

given that, your own connection pool theory is probably the better lead now. worth checking pg_stat_activity on postgres for any lock waits or long-running idle-in-transaction connections, and your app's pool size/timeout settings, if the pool got exhausted or a connection is stuck waiting on a lock, youd see exactly this pattern: normal cpu, normal memory, but consistent added latency per request. also worth checking if dns resolution to your postgres host is being cached or re-resolved on every connection, a slower dns path would add fixed latency without touching cpu


raularaujofaria
HOBBYOP

10 days ago

Thanks for the suggestions!

I tried both Restarting and Redeploying the MonitorJogosWeb service, but the issue persists.

Here are additional technical symptoms occurring right now in the Railway dashboard:

Exec Container Failure: When attempting to open the Shell console for MonitorJogosWeb, it fails with the error: WebSocket Disconnection: The Railway console repeatedly shows WebSocket connection failed when trying to establish a terminal session.

image.png

Latency: HTTP requests remain consistently slow (4-6 seconds), while CPU (0.001 vCPU) and Memory (0.042 GB) usage remain completely flat/low.

Since restarting/redeploying didn't clear the process block or restore access to the web shell, could this indicate an underlying host/node-level issue on US East, or a container process limit (NPROC/PID limit) lockup at the Railway runtime level?

Any insights on how to unblock the container shell session or check active connection threads directly would be greatly appreciated.

Attachments


raularaujofaria

Thanks for the suggestions! I tried both Restarting and Redeploying the MonitorJogosWeb service, but the issue persists. Here are additional technical symptoms occurring right now in the Railway dashboard: Exec Container Failure: When attempting to open the Shell console for MonitorJogosWeb, it fails with the error: WebSocket Disconnection: The Railway console repeatedly shows WebSocket connection failed when trying to establish a terminal session. ![image.png](https://station-server.railway.com/attachments/att_01kzrprh48fzzsdm9y0tskp9sd) Latency: HTTP requests remain consistently slow (4-6 seconds), while CPU (0.001 vCPU) and Memory (0.042 GB) usage remain completely flat/low. Since restarting/redeploying didn't clear the process block or restore access to the web shell, could this indicate an underlying host/node-level issue on US East, or a container process limit (NPROC/PID limit) lockup at the Railway runtime level? Any insights on how to unblock the container shell session or check active connection threads directly would be greatly appreciated.

manuproject
HOBBY

10 days ago

the redeploy resetting everything but not fixing it, plus the shell also failing, points away from your app and toward railways host/runtime for this specific deployment. worth escalating directly with these exact symptoms, thats not something fixable from the app side.

while you wait, add a temp endpoint that returns your db pool's connection counts so you can check state without needing the broken console


raularaujofaria
HOBBYOP

10 days ago

Hey, thanks for the detailed insight, that makes a lot of sense.

I just checked the database directly via DBeaver and can confirm the pool is not exhausted on the DB side. There are only 4 connections from my application, all in idle state with no locks or long-running queries. So as you pointed out, the issue is not on the app or database side, but on the Railway host/runtime for this specific deployment.

image.png

Given that the redeploy doesn't fix it and even the shell is failing, I'd like to escalate this directly to the Railway team with these exact symptoms. However, when I go to the "Get Help" option in the dashboard, the only choice I see is "Ask the Community". Is there another way to open a direct ticket or reach the infrastructure team? I want to make sure this reaches the right people since it's not something fixable from my side.

Thanks again for the help!

Attachments


raularaujofaria

Hey, thanks for the detailed insight, that makes a lot of sense. I just checked the database directly via DBeaver and can confirm the pool is not exhausted on the DB side. There are only 4 connections from my application, all in idle state with no locks or long-running queries. So as you pointed out, the issue is not on the app or database side, but on the Railway host/runtime for this specific deployment. ![image.png](https://station-server.railway.com/attachments/att_01kzrr4bwkfq8abysaksg0kpyb) Given that the redeploy doesn't fix it and even the shell is failing, I'd like to escalate this directly to the Railway team with these exact symptoms. However, when I go to the "Get Help" option in the dashboard, the only choice I see is "Ask the Community". Is there another way to open a direct ticket or reach the infrastructure team? I want to make sure this reaches the right people since it's not something fixable from my side. Thanks again for the help!

manuproject
HOBBY

10 days ago

normally they do come inside comments too help like the mods and the employee i think you just have to wait one them will surely comment soon


raularaujofaria
HOBBYOP

10 days ago

Thanks!! I will wait for them. 😭


Welcome!

Sign in to your Railway account to join the conversation.

Loading...