Intermittent AU Railway pre-deploy connectivity timeouts to Neon PostgreSQL
cryptocryptocryptocrypto
PROOP

a month ago

Hello,

We have been experiencing recurring production deployment failures for the past several days. The failures are AU-specific and occur during the Railway pre-deploy database migration phase, when the pre-deploy container intermittently cannot establish a PostgreSQL connection to our AU Neon pooler endpoint.

Deployment details

Service: eclipse-api

Environment: production

Database region: AWS ap-southeast-2

Destination port: 5432

Endpoint type: Neon pooled PostgreSQL endpoint

Affected period: Ongoing for several days

Representative failure:

Deployment ID: c4433801-0a0b-4202-8191-ff4a127f19eb

Release: 360aed555dbfc571daa135aa7322684bff7c47d0

Failure window: September 9, 2026, approximately 04:41–04:42 UTC

Observed errors

The migration runner reports timeouts to all resolved IPv4 addresses:

ETIMEDOUT connect 54.153.234.207:5432

ETIMEDOUT connect 54.206.85.193:5432

ETIMEDOUT connect 3.106.69.51:5432

It also reports unreachable IPv6 routes:

ENETUNREACH connect 2406:da1c:b7c:d0e:9858:9714:c422:b17b:5432

ENETUNREACH connect 2406:da1c:b7c:d11:b952:e163:87c0:5972:5432

ENETUNREACH connect 2406:da1c:b7c:d20:2c57:4c4d:e935:9480:5432

The initial connection-check phase succeeds, but the subsequent migration process opens a separate connection for the advisory lock and fails. A second attempt approximately six seconds later produces the same result.

Neon-side observations

Neon monitoring for the AU compute during the relevant period shows:

Compute active

Autosuspend disabled

Low CPU utilization

No deadlocks

Pooler wait time of 0.00s

Pooler connections well below the configured maximum

No failed control-plane operations

The Neon endpoint is serving other clients, so this does not appear to be a database suspension, capacity, or pooler saturation issue.

Assistance requested

Because this has been recurring for several days, could you investigate the outbound network path used by the production pre-deploy container, specifically:

1 - Whether the container’s NAT or egress path had a routing problem to the listed Neon IPv4 addresses.

2 - Whether there was an AU-region or Railway egress incident during 04:41–04:42 UTC.

3 - Whether the pre-deploy container uses a different egress path from the running service.

4 - Whether there were ephemeral port, NAT exhaustion, firewall, or address-family routing issues.

5 - Whether Railway can provide the container’s egress/NAT identity and relevant network diagnostics for this deployment.

We are not requesting that the database migration gate be bypassed. The deployment must remain fail-closed when the AU database cannot be reached. We need help identifying why the Railway pre-deploy container has been intermittently unable to reach a healthy Neon AU endpoint over multiple days.

Thank you in advance.

$10 Bounty

7 Replies

Railway
BOT

a month ago

This thread has been opened as a bounty so the community can help solve it.

Status changed to Open Railway • about 1 month ago


pepperoney
HOBBY

a month ago

I think your Railway ticket is pointed in the right direction, but there are two things I would isolate before concluding that this is purely a Railway AU→Neon routing issue.

First, the IPv6 ENETUNREACH errors are probably not the important part. Railway currently disables outbound IPv6 by default, and their documentation explicitly states that IPv6 attempts will fail with ENETUNREACH while it is disabled. The significant signal is that all resolved IPv4 addresses are timing out on TCP/5432.

Second, Railway confirms that pre-deploy commands run in a separate container from the application. Therefore, successful connectivity from the currently running service does not prove that the pre-deploy container is using the same outbound/NAT path. I would keep that question very prominently in the support request.

There is also one application-side variable worth removing from the equation: you are running migrations against a Neon pooled endpoint. For migrations—particularly runners that use session-dependent behaviour or advisory locks—I would test a dedicated direct/unpooled Neon connection.

For example:

  • DATABASE_URL → Neon pooled endpoint for the running API
  • MIGRATION_DATABASE_URL → Neon direct/unpooled endpoint for the pre-deploy migration

The deployment can remain fully fail-closed: if the direct database connection is unavailable, the migration exits non-zero and Railway must abort deployment.

I would also temporarily instrument the pre-deploy step to capture, without logging credentials:

  • timestamp
  • Railway deployment/region metadata
  • resolved A records
  • resolved AAAA records
  • public IPv4 egress address
  • TCP reachability of each resolved IPv4 on port 5432
  • TCP reachability of the Neon hostname on port 5432

Then run the same diagnostics from the currently running service.

That comparison would be particularly valuable:

running container → egress IP/path → Neon:5432

versus

pre-deploy container → egress IP/path → Neon:5432

If one succeeds and the other fails, Railway should be able to trace the NAT/egress path for the deployment ID you provided.

I would also verify that Neon IP Allow is either disabled or contains all applicable Railway egress addresses.

One wording change for the support ticket: Railway currently documents Singapore as its APAC deployment region rather than an Australia deployment region, so I would ask whether there was an egress/routing incident from the Railway region hosting the deployment toward AWS ap-southeast-2, rather than calling it an “AU Railway region” incident.

Given that DNS succeeds, Neon compute/pooler health is normal, and all three IPv4 destinations time out, I agree that Railway-side egress/routing is a credible hypothesis. But testing the direct Neon endpoint and comparing pre-deploy versus runtime egress should make the case substantially stronger.


cryptocryptocryptocrypto
PROOP

a month ago

Thank you. We agree that the IPv6 ENETUNREACH messages are probably expected when IPv6 is disabled; the important failure is that all resolved IPv4 addresses timed out on TCP/5432.

The application’s initial connection check succeeds, then the migration opens a second PostgreSQL connection for its transaction-level advisory lock. That second connection is where the failure occurs.

Please confirm whether the Railway pre-deploy container for deployment c4433801-0a0b-4202-8191-ff4a127f19eb uses the same outbound NAT/egress path as the running eclipse-api service. We would also appreciate any available source egress identity and network-path diagnostics for the failure window of approximately 04:41–04:42 UTC.

Please investigate possible differences in:

pre-deploy versus runtime egress/NAT;

routing to the resolved Neon IPv4 addresses;

ephemeral-port or NAT exhaustion;

firewall or IP-allowlist behavior;

connectivity from the Railway Southeast Asia region (Singapore) connecting to Neon in AWS ap-southeast-2 (Sydney).


cryptocryptocryptocrypto
PROOP

a month ago

Please inspect the historical network flows for deployment c4433801-0a0b-4202-8191-ff4a127f19eb, especially the pre-deploy deployment instance, during 2026-09-09 04:40–04:43 UTC. We need egress TCP/5432 flow details to the resolved Neon IPv4 addresses, including the source/NAT identity, deployment instance ID, packet counts, latency, and any drop cause.


cryptocryptocryptocrypto
PROOP

a month ago

.


cryptocryptocryptocrypto
PROOP

a month ago

14:41:39

=== Railway Pre-Deploy: Starting ===

14:41:39

Environment: production

14:41:39

=== Skipping Playwright Chromium install (CRAWL_NO_SANDBOX_OK not set) ===

14:41:39

[predeploy] event=regional_database_policy deploymentTier=production requiresAllRegions=true missingOptionalRegions=none

14:41:39

[predeploy] phase=regional-database-endpoint-contract state=checking service=eclipse-api deployment=c4433801-0a0b-4202-8191-ff4a127f19eb release=360aed555dbfc571daa135aa7322684bff7c47d0

14:41:39

npm warn config production Use --omit=dev instead.

14:41:39

Starting Container

14:41:41

[regional-database-endpoint-contract] passed contractVersion=neon-endpoint-contract-2026-09-08.3 service=eclipse-api deployment=c4433801-0a0b-4202-8191-ff4a127f19eb release=360aed555dbfc571daa135aa7322684bff7c47d0 DATABASE_URL:category=neon_pooler,representation=aws_region_suffix,region=ap-southeast-2,port=5432,hostFingerprint=115d349c9b804b74 DATABASE_URL_AU:category=neon_pooler,representation=aws_region_suffix,region=ap-southeast-2,port=5432,hostFingerprint=115d349c9b804b74 DATABASE_URL_US:category=neon_pooler,representation=aws_region_suffix,region=us-east-1,port=5432,hostFingerprint=b960d94e12b0a576 DATABASE_URL_EU:category=neon_pooler,representation=aws_region_suffix,region=eu-central-1,port=5432,hostFingerprint=b9de6e8a61c70c6c

14:41:42

[predeploy] phase=connection database="primary DB" attempt=1/2 timeoutSeconds=30

14:41:42

[predeploy] phase=regional-database-endpoint-contract state=passed

...and 59 more logs

I can now see the root cause and the critical network issue. Let me update your working memory and provide you the findings.

Summary

I've opened the deployment and found the critical failure:

Deployment c4433801-0a0b-4202-8191-ff4a127f19eb failed during PRE_DEPLOY_COMMAND on 2026-09-09 04:42:04–04:42:26 UTC.

Root Cause: Network Connectivity Failure to Neon Database (TCP/5432)

The pre-deploy script could not reach the Neon pooler endpoints on port 5432:

IPv4 Address Error Timestamp

54.153.234.207:5432 ETIMEDOUT (attempt 1 & 2) 04:41:42–04:42:04

54.206.85.193:5432 ETIMEDOUT (attempt 1 & 2) 04:41:42–04:42:04

3.106.69.51:5432 ETIMEDOUT (attempt 1 & 2) 04:41:42–04:42:04

All IPv6 addresses returned ENETUNREACH errors.

Migration attempts: 2 retries with 6s delay between them; both failed with transient connection errors.

Network Telemetry Limitation

The detailed egress TCP flow data you requested (source/NAT identity, packet counts, latency, drop cause per connection) is not available through Railway's standard logging APIs. This level of visibility (VPC NetFlow logs, security group rule hits, per-socket kernel state) requires:

Railway infrastructure-level access – VPC flow logs, container networking stack inspection

Container packet capture – requires privileged access or tcpdump during deployment

Kernel socket inspection (/proc/net/tcp, connection tracking) – would need to be captured in real-time

To investigate further, open a support thread with Railway to request:

VPC flow logs for the pre-deploy instance during 04:40–04:43 UTC

Whether security group rules, network policies, or region-scoped firewall changes affected egress to Neon

Whether a Neon infrastructure incident occurred in AU/US regions at that time

The deployment did not get to the point where container-level diagnostics are available.


cryptocryptocryptocrypto
PROOP

a month ago

We have confirmed from the pre-deploy logs that deployment c4433801-0a0b-4202-8191-ff4a127f19eb failed while establishing TCP connections from the pre-deploy container to all three resolved IPv4 addresses for the Neon endpoint on port 5432.

The attempts failed with ETIMEDOUT during approximately 2026-09-09 04:41:42–04:42:04 UTC. IPv6 also returned ENETUNREACH, which we understand may be expected when outbound IPv6 is disabled.

This confirms the immediate failure mode, but not the underlying infrastructure cause. Could Railway please inspect the failed pre-deploy deployment instance for:

  • the host and deployment-instance identity;
  • the outbound/NAT path used by that container;
  • any dropped or unanswered egress flows to the three Neon IPv4 addresses on TCP/5432;
  • security-group, firewall, routing, or NAT-pool events during 04:40–04:43 UTC;
  • whether the pre-deploy container used a different egress path from the running service.

pepperoney
HOBBY

a month ago

Thanks — I agree with the distinction in your last message: the logs establish the immediate failure mode, but not yet the infrastructure root cause.

One detail now seems particularly important: the initial PostgreSQL connectivity check from the pre-deploy container succeeds, and then a newly opened PostgreSQL connection for the migration/advisory-lock phase fails only seconds later.

That means I would specifically try to determine what differs between the successful and failed connections.

If you can reproduce this on another deployment, I would add temporary diagnostics that record, for every successful PostgreSQL connection:

  • timestamp;
  • resolved IPv4 address;
  • actual remote IPv4 selected;
  • local socket address/port where available;
  • Railway region/deployment metadata;
  • public egress IPv4 observed from that pre-deploy container.

Then log the equivalent information immediately before the migration connection is opened.

This could distinguish several cases:

  1. Same pre-deploy container, same NAT identity, same Neon destination, second connection fails

    → points toward a transient routing/NAT/host/network issue.

  2. Same container but different public egress/NAT identity between connections

    → very important if Neon IP Allow or another upstream firewall is in use.

  3. Different Neon IP selected from the DNS pool

    → could identify a bad path toward one subset of Neon/AWS addresses.

  4. Pooler endpoint fails but Neon direct endpoint succeeds from the same container

    → would move suspicion toward the Neon pooler/path rather than Railway outbound connectivity generally.

I would also explicitly verify whether Neon IP Allow is disabled. If it is enabled, confirm that every possible Railway outbound IP is permitted.

The most useful answer Railway staff could provide now is not another reading of the deployment logs, but whether the successful connection at roughly 04:41:42 and the failed connections immediately afterward traversed the same host, NAT gateway/pool, and outbound route.

In particular, I would ask them:

“Can you correlate the successful TCP/PostgreSQL connection immediately preceding the migration with the failed TCP SYN attempts from the same pre-deploy deployment, and confirm whether their source NAT identity and host/network path were identical?”

That comparison may be more informative than looking only at the failed flows.

Also, Railway's public status page does not show a general incident for September 9, so a host-, NAT-, or path-specific issue remains plausible even in the absence of a published regional incident.


Welcome!

Sign in to your Railway account to join the conversation.

Loading...