4 hours ago
Hello,
We have been experiencing recurring production deployment failures for the past several days. The failures are AU-specific and occur during the Railway pre-deploy database migration phase, when the pre-deploy container intermittently cannot establish a PostgreSQL connection to our AU Neon pooler endpoint.
Deployment details
Service: eclipse-api
Environment: production
Database region: AWS ap-southeast-2
Destination port: 5432
Endpoint type: Neon pooled PostgreSQL endpoint
Affected period: Ongoing for several days
Representative failure:
Deployment ID: c4433801-0a0b-4202-8191-ff4a127f19eb
Release: 360aed555dbfc571daa135aa7322684bff7c47d0
Failure window: September 9, 2026, approximately 04:41–04:42 UTC
Observed errors
The migration runner reports timeouts to all resolved IPv4 addresses:
ETIMEDOUT connect 54.153.234.207:5432
ETIMEDOUT connect 54.206.85.193:5432
ETIMEDOUT connect 3.106.69.51:5432
It also reports unreachable IPv6 routes:
ENETUNREACH connect 2406:da1c:b7c:d0e:9858:9714:c422:b17b:5432
ENETUNREACH connect 2406:da1c:b7c:d11:b952:e163:87c0:5972:5432
ENETUNREACH connect 2406:da1c:b7c:d20:2c57:4c4d:e935:9480:5432
The initial connection-check phase succeeds, but the subsequent migration process opens a separate connection for the advisory lock and fails. A second attempt approximately six seconds later produces the same result.
Neon-side observations
Neon monitoring for the AU compute during the relevant period shows:
Compute active
Autosuspend disabled
Low CPU utilization
No deadlocks
Pooler wait time of 0.00s
Pooler connections well below the configured maximum
No failed control-plane operations
The Neon endpoint is serving other clients, so this does not appear to be a database suspension, capacity, or pooler saturation issue.
Assistance requested
Because this has been recurring for several days, could you investigate the outbound network path used by the production pre-deploy container, specifically:
1 - Whether the container’s NAT or egress path had a routing problem to the listed Neon IPv4 addresses.
2 - Whether there was an AU-region or Railway egress incident during 04:41–04:42 UTC.
3 - Whether the pre-deploy container uses a different egress path from the running service.
4 - Whether there were ephemeral port, NAT exhaustion, firewall, or address-family routing issues.
5 - Whether Railway can provide the container’s egress/NAT identity and relevant network diagnostics for this deployment.
We are not requesting that the database migration gate be bypassed. The deployment must remain fail-closed when the AU database cannot be reached. We need help identifying why the Railway pre-deploy container has been intermittently unable to reach a healthy Neon AU endpoint over multiple days.
Thank you in advance.
1 Replies
4 hours ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • about 4 hours ago
2 hours ago
I think your Railway ticket is pointed in the right direction, but there are two things I would isolate before concluding that this is purely a Railway AU→Neon routing issue.
First, the IPv6 ENETUNREACH errors are probably not the important part. Railway currently disables outbound IPv6 by default, and their documentation explicitly states that IPv6 attempts will fail with ENETUNREACH while it is disabled. The significant signal is that all resolved IPv4 addresses are timing out on TCP/5432.
Second, Railway confirms that pre-deploy commands run in a separate container from the application. Therefore, successful connectivity from the currently running service does not prove that the pre-deploy container is using the same outbound/NAT path. I would keep that question very prominently in the support request.
There is also one application-side variable worth removing from the equation: you are running migrations against a Neon pooled endpoint. For migrations—particularly runners that use session-dependent behaviour or advisory locks—I would test a dedicated direct/unpooled Neon connection.
For example:
DATABASE_URL→ Neon pooled endpoint for the running APIMIGRATION_DATABASE_URL→ Neon direct/unpooled endpoint for the pre-deploy migration
The deployment can remain fully fail-closed: if the direct database connection is unavailable, the migration exits non-zero and Railway must abort deployment.
I would also temporarily instrument the pre-deploy step to capture, without logging credentials:
- timestamp
- Railway deployment/region metadata
- resolved A records
- resolved AAAA records
- public IPv4 egress address
- TCP reachability of each resolved IPv4 on port 5432
- TCP reachability of the Neon hostname on port 5432
Then run the same diagnostics from the currently running service.
That comparison would be particularly valuable:
running container → egress IP/path → Neon:5432
versus
pre-deploy container → egress IP/path → Neon:5432
If one succeeds and the other fails, Railway should be able to trace the NAT/egress path for the deployment ID you provided.
I would also verify that Neon IP Allow is either disabled or contains all applicable Railway egress addresses.
One wording change for the support ticket: Railway currently documents Singapore as its APAC deployment region rather than an Australia deployment region, so I would ask whether there was an egress/routing incident from the Railway region hosting the deployment toward AWS ap-southeast-2, rather than calling it an “AU Railway region” incident.
Given that DNS succeeds, Neon compute/pooler health is normal, and all three IPv4 destinations time out, I agree that Railway-side egress/routing is a credible hypothesis. But testing the direct Neon endpoint and comparing pre-deploy versus runtime egress should make the case substantially stronger.