a month ago
Hello,
We have been experiencing recurring production deployment failures for the past several days. The failures are AU-specific and occur during the Railway pre-deploy database migration phase, when the pre-deploy container intermittently cannot establish a PostgreSQL connection to our AU Neon pooler endpoint.
Deployment details
Service: eclipse-api
Environment: production
Database region: AWS ap-southeast-2
Destination port: 5432
Endpoint type: Neon pooled PostgreSQL endpoint
Affected period: Ongoing for several days
Representative failure:
Deployment ID: c4433801-0a0b-4202-8191-ff4a127f19eb
Release: 360aed555dbfc571daa135aa7322684bff7c47d0
Failure window: September 9, 2026, approximately 04:41–04:42 UTC
Observed errors
The migration runner reports timeouts to all resolved IPv4 addresses:
ETIMEDOUT connect 54.153.234.207:5432
ETIMEDOUT connect 54.206.85.193:5432
ETIMEDOUT connect 3.106.69.51:5432
It also reports unreachable IPv6 routes:
ENETUNREACH connect 2406:da1c:b7c:d0e:9858:9714:c422:b17b:5432
ENETUNREACH connect 2406:da1c:b7c:d11:b952:e163:87c0:5972:5432
ENETUNREACH connect 2406:da1c:b7c:d20:2c57:4c4d:e935:9480:5432
The initial connection-check phase succeeds, but the subsequent migration process opens a separate connection for the advisory lock and fails. A second attempt approximately six seconds later produces the same result.
Neon-side observations
Neon monitoring for the AU compute during the relevant period shows:
Compute active
Autosuspend disabled
Low CPU utilization
No deadlocks
Pooler wait time of 0.00s
Pooler connections well below the configured maximum
No failed control-plane operations
The Neon endpoint is serving other clients, so this does not appear to be a database suspension, capacity, or pooler saturation issue.
Assistance requested
Because this has been recurring for several days, could you investigate the outbound network path used by the production pre-deploy container, specifically:
1 - Whether the container’s NAT or egress path had a routing problem to the listed Neon IPv4 addresses.
2 - Whether there was an AU-region or Railway egress incident during 04:41–04:42 UTC.
3 - Whether the pre-deploy container uses a different egress path from the running service.
4 - Whether there were ephemeral port, NAT exhaustion, firewall, or address-family routing issues.
5 - Whether Railway can provide the container’s egress/NAT identity and relevant network diagnostics for this deployment.
We are not requesting that the database migration gate be bypassed. The deployment must remain fail-closed when the AU database cannot be reached. We need help identifying why the Railway pre-deploy container has been intermittently unable to reach a healthy Neon AU endpoint over multiple days.
Thank you in advance.
7 Replies
a month ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • about 1 month ago
a month ago
I think your Railway ticket is pointed in the right direction, but there are two things I would isolate before concluding that this is purely a Railway AU→Neon routing issue.
First, the IPv6 ENETUNREACH errors are probably not the important part. Railway currently disables outbound IPv6 by default, and their documentation explicitly states that IPv6 attempts will fail with ENETUNREACH while it is disabled. The significant signal is that all resolved IPv4 addresses are timing out on TCP/5432.
Second, Railway confirms that pre-deploy commands run in a separate container from the application. Therefore, successful connectivity from the currently running service does not prove that the pre-deploy container is using the same outbound/NAT path. I would keep that question very prominently in the support request.
There is also one application-side variable worth removing from the equation: you are running migrations against a Neon pooled endpoint. For migrations—particularly runners that use session-dependent behaviour or advisory locks—I would test a dedicated direct/unpooled Neon connection.
For example:
DATABASE_URL→ Neon pooled endpoint for the running APIMIGRATION_DATABASE_URL→ Neon direct/unpooled endpoint for the pre-deploy migration
The deployment can remain fully fail-closed: if the direct database connection is unavailable, the migration exits non-zero and Railway must abort deployment.
I would also temporarily instrument the pre-deploy step to capture, without logging credentials:
- timestamp
- Railway deployment/region metadata
- resolved A records
- resolved AAAA records
- public IPv4 egress address
- TCP reachability of each resolved IPv4 on port 5432
- TCP reachability of the Neon hostname on port 5432
Then run the same diagnostics from the currently running service.
That comparison would be particularly valuable:
running container → egress IP/path → Neon:5432
versus
pre-deploy container → egress IP/path → Neon:5432
If one succeeds and the other fails, Railway should be able to trace the NAT/egress path for the deployment ID you provided.
I would also verify that Neon IP Allow is either disabled or contains all applicable Railway egress addresses.
One wording change for the support ticket: Railway currently documents Singapore as its APAC deployment region rather than an Australia deployment region, so I would ask whether there was an egress/routing incident from the Railway region hosting the deployment toward AWS ap-southeast-2, rather than calling it an “AU Railway region” incident.
Given that DNS succeeds, Neon compute/pooler health is normal, and all three IPv4 destinations time out, I agree that Railway-side egress/routing is a credible hypothesis. But testing the direct Neon endpoint and comparing pre-deploy versus runtime egress should make the case substantially stronger.
a month ago
Thank you. We agree that the IPv6 ENETUNREACH messages are probably expected when IPv6 is disabled; the important failure is that all resolved IPv4 addresses timed out on TCP/5432.
The application’s initial connection check succeeds, then the migration opens a second PostgreSQL connection for its transaction-level advisory lock. That second connection is where the failure occurs.
Please confirm whether the Railway pre-deploy container for deployment c4433801-0a0b-4202-8191-ff4a127f19eb uses the same outbound NAT/egress path as the running eclipse-api service. We would also appreciate any available source egress identity and network-path diagnostics for the failure window of approximately 04:41–04:42 UTC.
Please investigate possible differences in:
pre-deploy versus runtime egress/NAT;
routing to the resolved Neon IPv4 addresses;
ephemeral-port or NAT exhaustion;
firewall or IP-allowlist behavior;
connectivity from the Railway Southeast Asia region (Singapore) connecting to Neon in AWS ap-southeast-2 (Sydney).
a month ago
Please inspect the historical network flows for deployment c4433801-0a0b-4202-8191-ff4a127f19eb, especially the pre-deploy deployment instance, during 2026-09-09 04:40–04:43 UTC. We need egress TCP/5432 flow details to the resolved Neon IPv4 addresses, including the source/NAT identity, deployment instance ID, packet counts, latency, and any drop cause.
a month ago
.
a month ago
14:41:39
=== Railway Pre-Deploy: Starting ===
14:41:39
Environment: production
14:41:39
=== Skipping Playwright Chromium install (CRAWL_NO_SANDBOX_OK not set) ===
14:41:39
[predeploy] event=regional_database_policy deploymentTier=production requiresAllRegions=true missingOptionalRegions=none
14:41:39
[predeploy] phase=regional-database-endpoint-contract state=checking service=eclipse-api deployment=c4433801-0a0b-4202-8191-ff4a127f19eb release=360aed555dbfc571daa135aa7322684bff7c47d0
14:41:39
npm warn config production Use --omit=dev instead.
14:41:39
Starting Container
14:41:41
[regional-database-endpoint-contract] passed contractVersion=neon-endpoint-contract-2026-09-08.3 service=eclipse-api deployment=c4433801-0a0b-4202-8191-ff4a127f19eb release=360aed555dbfc571daa135aa7322684bff7c47d0 DATABASE_URL:category=neon_pooler,representation=aws_region_suffix,region=ap-southeast-2,port=5432,hostFingerprint=115d349c9b804b74 DATABASE_URL_AU:category=neon_pooler,representation=aws_region_suffix,region=ap-southeast-2,port=5432,hostFingerprint=115d349c9b804b74 DATABASE_URL_US:category=neon_pooler,representation=aws_region_suffix,region=us-east-1,port=5432,hostFingerprint=b960d94e12b0a576 DATABASE_URL_EU:category=neon_pooler,representation=aws_region_suffix,region=eu-central-1,port=5432,hostFingerprint=b9de6e8a61c70c6c
14:41:42
[predeploy] phase=connection database="primary DB" attempt=1/2 timeoutSeconds=30
14:41:42
[predeploy] phase=regional-database-endpoint-contract state=passed
...and 59 more logs
I can now see the root cause and the critical network issue. Let me update your working memory and provide you the findings.
Summary
I've opened the deployment and found the critical failure:
Deployment c4433801-0a0b-4202-8191-ff4a127f19eb failed during PRE_DEPLOY_COMMAND on 2026-09-09 04:42:04–04:42:26 UTC.
Root Cause: Network Connectivity Failure to Neon Database (TCP/5432)
The pre-deploy script could not reach the Neon pooler endpoints on port 5432:
IPv4 Address Error Timestamp
54.153.234.207:5432 ETIMEDOUT (attempt 1 & 2) 04:41:42–04:42:04
54.206.85.193:5432 ETIMEDOUT (attempt 1 & 2) 04:41:42–04:42:04
3.106.69.51:5432 ETIMEDOUT (attempt 1 & 2) 04:41:42–04:42:04
All IPv6 addresses returned ENETUNREACH errors.
Migration attempts: 2 retries with 6s delay between them; both failed with transient connection errors.
Network Telemetry Limitation
The detailed egress TCP flow data you requested (source/NAT identity, packet counts, latency, drop cause per connection) is not available through Railway's standard logging APIs. This level of visibility (VPC NetFlow logs, security group rule hits, per-socket kernel state) requires:
Railway infrastructure-level access – VPC flow logs, container networking stack inspection
Container packet capture – requires privileged access or tcpdump during deployment
Kernel socket inspection (/proc/net/tcp, connection tracking) – would need to be captured in real-time
To investigate further, open a support thread with Railway to request:
VPC flow logs for the pre-deploy instance during 04:40–04:43 UTC
Whether security group rules, network policies, or region-scoped firewall changes affected egress to Neon
Whether a Neon infrastructure incident occurred in AU/US regions at that time
The deployment did not get to the point where container-level diagnostics are available.
a month ago
We have confirmed from the pre-deploy logs that deployment c4433801-0a0b-4202-8191-ff4a127f19eb failed while establishing TCP connections from the pre-deploy container to all three resolved IPv4 addresses for the Neon endpoint on port 5432.
The attempts failed with ETIMEDOUT during approximately 2026-09-09 04:41:42–04:42:04 UTC. IPv6 also returned ENETUNREACH, which we understand may be expected when outbound IPv6 is disabled.
This confirms the immediate failure mode, but not the underlying infrastructure cause. Could Railway please inspect the failed pre-deploy deployment instance for:
- the host and deployment-instance identity;
- the outbound/NAT path used by that container;
- any dropped or unanswered egress flows to the three Neon IPv4 addresses on TCP/5432;
- security-group, firewall, routing, or NAT-pool events during 04:40–04:43 UTC;
- whether the pre-deploy container used a different egress path from the running service.
a month ago
Thanks — I agree with the distinction in your last message: the logs establish the immediate failure mode, but not yet the infrastructure root cause.
One detail now seems particularly important: the initial PostgreSQL connectivity check from the pre-deploy container succeeds, and then a newly opened PostgreSQL connection for the migration/advisory-lock phase fails only seconds later.
That means I would specifically try to determine what differs between the successful and failed connections.
If you can reproduce this on another deployment, I would add temporary diagnostics that record, for every successful PostgreSQL connection:
- timestamp;
- resolved IPv4 address;
- actual remote IPv4 selected;
- local socket address/port where available;
- Railway region/deployment metadata;
- public egress IPv4 observed from that pre-deploy container.
Then log the equivalent information immediately before the migration connection is opened.
This could distinguish several cases:
-
Same pre-deploy container, same NAT identity, same Neon destination, second connection fails
→ points toward a transient routing/NAT/host/network issue.
-
Same container but different public egress/NAT identity between connections
→ very important if Neon IP Allow or another upstream firewall is in use.
-
Different Neon IP selected from the DNS pool
→ could identify a bad path toward one subset of Neon/AWS addresses.
-
Pooler endpoint fails but Neon direct endpoint succeeds from the same container
→ would move suspicion toward the Neon pooler/path rather than Railway outbound connectivity generally.
I would also explicitly verify whether Neon IP Allow is disabled. If it is enabled, confirm that every possible Railway outbound IP is permitted.
The most useful answer Railway staff could provide now is not another reading of the deployment logs, but whether the successful connection at roughly 04:41:42 and the failed connections immediately afterward traversed the same host, NAT gateway/pool, and outbound route.
In particular, I would ask them:
“Can you correlate the successful TCP/PostgreSQL connection immediately preceding the migration with the failed TCP SYN attempts from the same pre-deploy deployment, and confirm whether their source NAT identity and host/network path were identical?”
That comparison may be more informative than looking only at the failed flows.
Also, Railway's public status page does not show a general incident for September 9, so a host-, NAT-, or path-specific issue remains plausible even in the absence of a published regional incident.