2 days ago
Project: PiCell-Works (70e5c890-52ee-4eff-afce-4c2c2f89d8ae)
Service: picell-works (a3bceeb6-cd63-44e8-a06d-a920286c3ff2)
Environment: production (42eaffdb-c10e-4dc4-9636-24ccbcacfb6f)
Region: asia-southeast1-eqsg3a
Example deployment/instance: f137af6f-060a-4e5a-a178-572b897359ce / instance 9a236a6f-e2ff-4254-8212-f5abd6b5a659
For a while now, a significant fraction of requests to our service intermittently
take 1–7 seconds, while the app itself is idle and healthy. We have instrumented
the app extensively and can show the delay happens **before the request ever
reaches our Node process**:
-
Edge HTTP logs show e.g. 28 requests ≥ 500 ms (up to 4.8 s) between
2026-07-03 16:21–16:25 UTC (
upstreamRqDuration≈totalDuration). -
In-app instrumentation hooked at the earliest possible point (Node
http.Serverrequestevent → response close) with a 500 ms thresholdlogged zero slow requests in the same window. Route handlers, the pg
connection pool, and Postgres (
pg_stat_statementsmax_exec_time < 0.7 sover the whole day) are all fast. Event-loop delay stays < 60 ms.
-
Latencies cluster around ~1 s / ~3 s / ~7 s, which matches TCP
retransmission back-off (1 s, +2 s, +4 s) after dropped packets.
-
Your network flow logs recorded a
PROTO_MEMdrop on our container'straffic (kernel dropping TCP data due to protocol memory pressure):
-
flowId
408e2aca-dd88-405c-8edc-c87d70c92ba2,2026-07-03T16:22:28Z, ingress from Postgres (
:5432) to the appcontainer, 80 packets / 142 KB dropped.
Our container uses ~400 MB of its 24 GB limit, so this looks like
host-level TCP memory pressure (noisy neighbor), not our workload.
-
Could you check the host our service instance is placed on and migrate us to a
healthier host? Happy to provide more instrumentation data if useful.
1 Replies
Status changed to Awaiting Railway Response Railway • 2 days ago
Status changed to Solved dwander • 2 days ago
2 days ago
I accidentally clicked Resolved. This thread is not resolved yet.
Status changed to Awaiting Railway Response Railway • 2 days ago
2 days ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • 2 days ago