a month ago
My Postgres service (ID: 82f30c0a-91ad-4ade-ba50-569225b92850) in project e7039ff6 is broken. The Railway Postgres DB UI shows "We are unable to connect to the database via SSH" with error "Invalid Record Length: columns length is 1, got 2 on line 4". All connections from my app fail. This started May 23, 2026. I need the service repaired or restored from backup. My app depends on this database and is down. #✋|help
41 Replies
a month ago
Could you provide the errors you get when trying to connect from your app?
Tell them exactly this:
From the app (cozy-truth, Node.js 18 Alpine, pg@8.20.0):
With ssl: false: "Connection terminated unexpectedly"
With ssl: { rejectUnauthorized: false }: "There was an error establishing an SSL connection"
With NODE_TLS_REJECT_UNAUTHORIZED=0 set: same errors persist regardless of SSL config.
The database host resolves correctly (postgres-lwgx.railway.internal), credentials are confirmed correct via railway run environment inspection, but every connection attempt fails immediately. This started May 23, 2026.
Railway's own Postgres DB UI also shows: "We are unable to connect to the database via SSH — Failed to parse psql output: Error: Invalid Record Length: columns length is 1, got 2 on line 4"
The corruption appears to be at the PostgreSQL data file level, not an application issue.
Attachments
Attachments
a month ago
The error visible in the console here is of your app
a month ago
I would think that your app is crashing and then unexpectedly closing the database connection as a result of it
a month ago
And what logs do you see in the Postgres service?
(no markdown available for this content)
Attachments
(no markdown available for this content)
Attachments
a month ago
Postgres service
Attachments
(no markdown available for this content)
Attachments
a month ago
Could you try suffixing ?sslmode=require&sslnegotiation=postgres to your database connection URL in your app?
a month ago
Can do that from the variables tab
a month ago
Also consider downgrading pg, to something like 8.15.x
a month ago
might also fix it
a month ago
same error on the app and postgres service?
Still getting "There was an error establishing an SSL connection". The sslnegotiation=postgres URL suffix did not fix it. The Node.js pg library is a pure JavaScript implementation — it doesn't read the sslnegotiation URL parameter from libpq. The Postgres server logs show "received direct SSL connection request without ALPN protocol negotiation extension."
Can you either:
Tell us the correct DATABASE_PUBLIC_URL for our Postgres service so we can connect through the external proxy instead of the internal hostname? or
Confirm whether the internal Postgres (postgres-lwgx.railway.internal) accepts plain non-SSL connections, and if not, what TLS configuration is required for Node.js clients?
a month ago
Can you show me which image and version the Postgres service is on?
a month ago
Can see this in the service settings tab
a month ago
remove this query string as it's obvious it doesn't work
a month ago
and consider disabling ssl ssl: false in pg as you are (should be) using railway's private networking
a month ago
And also do confirm that your app's env looks similar to this
Attachments
We set DATABASE_URL to ${{Postgres-lwGX.DATABASE_URL}} as shown in your screenshot. With ssl: false, we still get Connection terminated unexpectedly. The injected DATABASE_URL shows postgres-lwgx.railway.internal:5432 as the host but the password appears stale. Could you verify the Postgres service credentials on your side and confirm the DATABASE_URL variable reference is resolving to current/valid credentials? We suspect the Postgres password may have rotated and the reference isn't reflecting the new valu
Done — ssl: false, query string removed. Still getting Connection terminated unexpectedly on every connection attempt. The DATABASE_URL resolves to postgres-lwgx.railway.internal:5432 with the credentials from ${{Postgres-lwGX.DATABASE_URL}}. Is it possible the Postgres password is out of sync? The Railway DB UI is also showing "unable to connect via SSH" with an invalid record length error — it looks like the Postgres service itself may have a credentials or data issue on your side.
ENOTFOUND base is fixed. Now connecting to postgres-lwgx.railway.internal correctly with ssl: false. Still getting Connection terminated unexpectedly immediately. The Railway Postgres DB UI still shows "Unable to connect via SSH — Invalid Record Length: columns length is 1, got 2 on line 4". The Postgres service itself appears to have a config or data integrity issue on your side. Can you investigate the Postgres service health directly
We've tried every SSL configuration. ssl: false → "Connection terminated unexpectedly". ssl: { rejectUnauthorized: false } → "There was an error establishing an SSL connection". ssl: { rejectUnauthorized: false, ciphers: 'DEFAULT@SECLEVEL=0' } → same SSL error. The Postgres server logs earlier showed "no shared cipher" and "bad key share" which indicates TLS cipher/version incompatibility at the infrastructure level. Can you fix the Postgres SSL configuration or confirm which port/TLS config is correct for Node.js 18 clients? Alternatively, can you provide the correct DATABASE_PUBLIC_URL (external proxy) for this service as a workaround?
BREAKTHROUGH! Decoding the hex response:
485454502f312e31203430302042616420526571756573740d0a436f6e6e656374696f6e3a20636c6f73650d0a0d0a
= HTTP/1.1 400 Bad Request\r\nConnection: close\r\n\r\n
Port 5432 at postgres-lwgx.railway.internal is NOT PostgreSQL — it's an HTTP proxy. Railway updated their internal networking to route through an HTTP layer. PostgreSQL protocol gets rejected as "bad request". This is why NOTHING has worked all session — we've never been talking to the actual database.
The external proxy (turntable.proxy.rlwy.net:23306) IS real PostgreSQL. We need to fix the TLS handshake on that path. Switch to external URL and let the diagnostic tell us the actual TLS error:
a month ago
Private networking is TCP (incl HTTP) and UDP
a month ago
Railway did not update their internal networking at all
Found the root cause. Our cozy-truth Node.js app was accidentally deployed to the Postgres-lwGX service (deploy 206a318b). The Postgres container is running our Express server instead of PostgreSQL. The postgres-volume-0cYy volume should still have the data files intact. Can you restore the Postgres-lwGX service to a proper PostgreSQL image? The volume needs to stay attached."