issue with database
guler9-ops
PROOP

5 hours ago

Service ID: 39422adc-4110-4b1a-82a4-7c7c638955d9

Volume ID: b354fc4a-042d-4552-ad9a-ac1dedb4acaf

Project: jubilant-heart

Issue: Postgres database is offline and shows WAL corruption. The alldesk_backups table filled the original 500MB volume. After expanding to 4GB, the database still crashes with "invalid record length" errors. Database needs restoration or volume recovery.

$20 Bounty

13 Replies

Railway
BOT

5 hours ago

Our databases are unmanaged, so automatic snapshots, WAL recovery, and PITR aren't available unless you set up volume backups or PITR yourself. We don't recover data lost to user-initiated actions. Going forward, enable volume backups on your stateful services so you can self-restore if this happens again.


Status changed to Awaiting User Response Railway about 5 hours ago


guler9-ops
PROOP

3 hours ago

Project: jubilant-heart

Postgres Service: 39422adc-4110-4b1a-82a4-7c7c638955d9

DATABASE_URL interpolates correctly (93 chars) but pg library HANGS on connect.

No timeout error, just hangs forever.

Connection works nowhere - pg library, psql CLI all hang.

Container file access returns "WS connection closed".

CRITICAL: Only copy of business data (alldesk_store table) - no backups exist.


Status changed to Awaiting Railway Response Railway about 3 hours ago


Railway
BOT

3 hours ago

Having looked into this, the issue appears to be in your application code or configuration rather than the Railway platform itself, which puts it outside what Railway support can resolve directly.

This is exactly the kind of problem the Railway community is good at, so we'd like to open your thread as a community bounty. Railway pays a bounty to the community member who solves it, and threads like this usually get picked up quickly.

Opening it makes this entire thread public, including everything already posted. Nothing becomes public until you decide. Use the buttons below.

  • Open to the community - Before you click, take a moment to edit or remove anything you'd rather not share. The thread becomes publicly visible right away.
  • Keep it private and close the thread - Nothing becomes public. The thread closes, since this isn't something Railway support can take further.

Status changed to Awaiting User Response Railway about 3 hours ago


Railway
BOT

3 hours ago

This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.

Status changed to Open Railway about 3 hours ago


Did you redeploy your Postgres service after you increased the volume size?


Also, invalid record length logs are mostly normal. Can you provide the full error logs of your database?


guler9-ops
PROOP

2 hours ago

@darseen thx! I just redeployed the Volume Resize - no answer Postgres !

Logs:

  • 2 failed Deployments (8f0337a0... und 153573a1...)
  • Connection timeout expired (10s timeout)
  • Container file access: WS connection closed
  • Postgres zeigt "Online" aber ist unreachable

Volume Info:

  • Volume ID: b354fc4a-042d-4552-ad9a-ac1dedb4acaf
  • War: 500MB → Jetzt: 4GB
  • Mount Path: /var/lib/postgresql/data

Can you share the database logs?


guler9-ops
PROOP

2 hours ago

@darseen Danke für die Anfrage. Hier sind die Logs:

OLD POSTGRES IMAGE ERROR (postgres-ssl:18):

"root" execution of the PostgreSQL server is not permitted.

The server must be started under an unprivileged user ID.

TESTED SOLUTIONS:

  1. Removed bad startCommand (recovery_target='immediate') ❌ Still failed
  2. Changed to official postgres:18 image ❌ Still failed
  3. Restarted container ❌ Still failed

CURRENT STATUS:

  • 3 failed deployments (8f0337a0..., 153573a1..., 11f2d86c...)
  • Postgres won't start with ANY image
  • Connection timeout from other services
  • Volume b354fc4a-042d-4552-ad9a-ac1dedb4acaf is unreachable
  • Data directory cannot be accessed (WS connection closed)

CONCLUSION:

Database appears permanently corrupted and unrecoverable. Is there a way to:

  1. Restore from volume snapshots?
  2. Access raw volume data?
  3. Force PostgreSQL recovery mode?

This is our only copy of critical business data (alldesk_store table). Any help appreciated!


The database logs are located in Deployments > View logs > Deploy logs. Can you share those please?

image.png

Attachments


guler9-ops
PROOP

2 hours ago

@darseen The deploy logs are completely EMPTY - the container doesn't start at all. The build logs are also empty. This suggests the deployment isn't even beginning to build properly.

This appears to be an infrastructure issue rather than just a Postgres problem. The 4GB volume may not be mounting correctly after the resize.

Additional details:

Service shows "Online" but is unreachable (timeout after 10s)

Container file access returns "WS connection closed"

Tested with both old image (postgres-ssl:18) and official postgres:18 - both fail

Removed problematic startCommand but deployment still fails

Could this be a volume mounting issue after the resize? The fact that NO logs are generated suggests the container can't even initialize. Any insights would be greatly appreciated - this is critical business data.


The logs can't be empty if the database is crashing. logs might take some time to load. So if you see "Your logs are loading", just wait for them to show up, and provide a screenshot or paste them here. Unfortunately, without any logs, we can't determine the cause of the issue.


guler9-ops
PROOP

2 hours ago

@darseen Thanks for the clarification! I'm checking the logs via the API and they're still empty/not loading. I'll check the Railway dashboard directly to see if they appear there. Let me go to Deployments > View logs > Deploy logs and send you a screenshot if the logs are loading there.


guler9-ops
PROOP

an hour ago

Project: jubilant-heart (ID 5d8a7adb-69d3-4a34-a83a-d8f6531a0c9f)

Service: "Postgres" (ID 39422adc-4110-4b1a-82a4-7c7c638955d9), image postgres:18

Env: production

Problem: The Postgres service is stuck in a crash loop. Deploy logs show, on every start:

Starting Container

"root" execution of the PostgreSQL server is not permitted.

The server must be started under an unprivileged user ID...

So the postgres server process is being launched as root and immediately exits.

This started during a Postgres major-version migration (I also have

postgres-recovery-qmJM on pg15 holding the real data volume "postgres-volume",

and postgres-dump on pg16).

Questions:

  1. Is a custom start command on this service causing postgres to run as root

    instead of going through docker-entrypoint.sh (which drops to the postgres user)?

    What is the correct start command / image config for this service?

  2. What is the recommended safe path to get back online without losing the data

    on the "postgres-volume" (currently attached to postgres-recovery-qmJM, pg15)?


Welcome!

Sign in to your Railway account to join the conversation.

Loading...