Intermittent 1–7 s request latency caused by kernel-level packet drops (PROTO_MEM) — app itself confirmed healthy
dwander
PROOP

2 days ago

Project: PiCell-Works (70e5c890-52ee-4eff-afce-4c2c2f89d8ae)

Service: picell-works (a3bceeb6-cd63-44e8-a06d-a920286c3ff2)

Environment: production (42eaffdb-c10e-4dc4-9636-24ccbcacfb6f)

Region: asia-southeast1-eqsg3a

Example deployment/instance: f137af6f-060a-4e5a-a178-572b897359ce / instance 9a236a6f-e2ff-4254-8212-f5abd6b5a659

For a while now, a significant fraction of requests to our service intermittently

take 1–7 seconds, while the app itself is idle and healthy. We have instrumented

the app extensively and can show the delay happens **before the request ever

reaches our Node process**:

  1. Edge HTTP logs show e.g. 28 requests ≥ 500 ms (up to 4.8 s) between

    2026-07-03 16:21–16:25 UTC (upstreamRqDurationtotalDuration).

  2. In-app instrumentation hooked at the earliest possible point (Node

    http.Server request event → response close) with a 500 ms threshold

    logged zero slow requests in the same window. Route handlers, the pg

    connection pool, and Postgres (pg_stat_statements max_exec_time < 0.7 s

    over the whole day) are all fast. Event-loop delay stays < 60 ms.

  3. Latencies cluster around ~1 s / ~3 s / ~7 s, which matches TCP

    retransmission back-off (1 s, +2 s, +4 s) after dropped packets.

  4. Your network flow logs recorded a PROTO_MEM drop on our container's

    traffic (kernel dropping TCP data due to protocol memory pressure):

    • flowId 408e2aca-dd88-405c-8edc-c87d70c92ba2,

      2026-07-03T16:22:28Z, ingress from Postgres (:5432) to the app

      container, 80 packets / 142 KB dropped.

    Our container uses ~400 MB of its 24 GB limit, so this looks like

    host-level TCP memory pressure (noisy neighbor), not our workload.

Could you check the host our service instance is placed on and migrate us to a

healthier host? Happy to provide more instrumentation data if useful.

$20 Bounty

1 Replies

Status changed to Awaiting Railway Response Railway 2 days ago


Status changed to Solved dwander 2 days ago


dwander
PROOP

2 days ago

I accidentally clicked Resolved. This thread is not resolved yet.


Status changed to Awaiting Railway Response Railway 2 days ago


Railway
BOT

2 days ago

This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.

Status changed to Open Railway 2 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...