Intermittent TCP_AOFAILURE packet drops between service and Postgres (Hobby plan)
pratibhavanave
HOBBYOP

16 days ago

I'm experiencing intermittent connectivity failures between my backend service and its Postgres database on the Hobby plan.

Symptom: Requests that hit the database occasionally fail with a network/connection error, then succeed on retry seconds later with no code or data changes. No clear pattern tied to load or time of day.

Root cause found in network flow logs — a dropped packet matching this behavior:

flowId: 81b5ffa2-10c8-42d1-8606-6b54d6d5fc1f

captureStart/End: 2026-06-20T17:18:34.711484020Z

srcAddr: fd12:2cd3:db30:1:9000:29:c695:3b0d (Postgres, port 5432)

dstAddr: fd12:2cd3:db30:1:9000:29:108f:c3ac (port 33802)

direction: egress

dropCause: TCP_AOFAILURE

serviceId: ab64794c-ceed-43ac-80ec-31eb85f941b6

deploymentId: 041560b5-3cdb-4e81-872d-546ee55fdbc4

deploymentInstanceId: bed00744-3f65-40d8-a3bc-b2aba598a4e0

Corroborating evidence in Postgres logs (same period):

could not receive data from client: Connection reset by peer (multiple connections, 2026-06-21 09:00:33 UTC)

Unexpected restart: database system was interrupted; last known up at 2026-06-21 09:02:53 UTC, followed by automatic WAL recovery (data was recovered successfully, no loss)

Questions for support:

What causes a TCP_AOFAILURE drop on the internal service-to-service network mesh?

Is this expected/known behavior, or specific to this project/instance/deployment?

Is there a mitigation available on the Hobby plan, or does this require action on Railway's side?

Happy to provide additional logs, timestamps, or project/service IDs as needed.

Solved

1 Replies

Status changed to Awaiting Railway Response Railway 16 days ago


Thanks for the unusually thorough report, the flow log and Postgres correlation are exactly what we need. A TCP_AOFAILURE drop on the private network points to an intermittent drop in the internal mesh rather than anything in your code or service config, so there's no Hobby-plan limitation or app-side change driving this. We're routing it to our networking team to trace the flow directly.

To help them narrow it down, could you share how often you see it (roughly per hour or per day), whether it correlates with redeploys or scaling events, and a few more drop timestamps if you have them. The flow ID, both service IDs, and the capture window you already provided are great.

As interim resilience, add connection-pool retry with short backoff so a transient drop is retried transparently instead of surfacing as a request failure. We'll follow up here as the team investigates.


Status changed to Awaiting User Response Railway 15 days ago


Railway
BOT

8 days ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway 8 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...