Repeatable Python wake-up jitter in one Singapore service after same-image restart
x4u20
HOBBYOP

3 hours ago

Hi Railway team,

We are investigating repeatable latency differences between two services in asia-southeast1-eqsg3a.

Concurrent diagnostic probes run via SSH on 2026-09-26 at approximately 18:27 UTC showed:

  1. Python time.sleep(0.01) wake-up overshoot — 100 samples:

    Control: p95 0.140 ms, max 0.517 ms

    Candidate: p95 16.918 ms, max 23.708 ms

  2. Persistent read-only PostgreSQL SELECT 1 — 100 samples:

    Control round-trip: p95 2.570 ms, max 4.718 ms

    Candidate round-trip: p95 5.658 ms, max 28.137 ms

  3. EXPLAIN ANALYZE SELECT 1 — 50 samples:

    Control round-trip p95: 2.272 ms

    Candidate round-trip p95: 19.297 ms

    PostgreSQL executor p95: 0.028 ms and 0.019 ms respectively.

The sleep probe does not import application code, access the database, or use the network. The DB probes bypass the application connection pool.

Both containers report:

  • cpu.max: 800000 100000
  • cpu.weight: 100
  • Timer slack: 50000 ns
  • Normal scheduling policy
  • Clocksource: kvm-clock
  • Zero CPU quota-throttling increments during these probes

Observed kernel versions:

  • Control: 6.18.15+deb13-cloud-amd64
  • Candidate: 6.12.22+bpo-cloud-amd64

We are not claiming the kernel difference causes the problem. The application images also have three differing Python dependency versions, so we are not claiming identical runtime builds.

We restarted only the candidate using its exact existing image. Its deployment SHA/image remained unchanged, but the no-network wake-up jitter persisted.

Could Railway investigate scheduling/placement for the candidate, or advise how to move only that candidate to an equivalent healthy runtime in the same Singapore region while preserving its image?

Please leave the control service, shared database, and normal traffic unchanged. This is an isolated candidate diagnostic, not a production outage.

We can provide exact service/deployment identifiers through an appropriate private channel.

Thank you.

Awaiting Railway Response

0 Replies

Status changed to Awaiting Railway Response Railway • about 3 hours ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...