5 hours ago
Service ID: 39422adc-4110-4b1a-82a4-7c7c638955d9
Volume ID: b354fc4a-042d-4552-ad9a-ac1dedb4acaf
Project: jubilant-heart
Issue: Postgres database is offline and shows WAL corruption. The alldesk_backups table filled the original 500MB volume. After expanding to 4GB, the database still crashes with "invalid record length" errors. Database needs restoration or volume recovery.
13 Replies
5 hours ago
Our databases are unmanaged, so automatic snapshots, WAL recovery, and PITR aren't available unless you set up volume backups or PITR yourself. We don't recover data lost to user-initiated actions. Going forward, enable volume backups on your stateful services so you can self-restore if this happens again.
Status changed to Awaiting User Response Railway • about 5 hours ago
3 hours ago
Project: jubilant-heart
Postgres Service: 39422adc-4110-4b1a-82a4-7c7c638955d9
DATABASE_URL interpolates correctly (93 chars) but pg library HANGS on connect.
No timeout error, just hangs forever.
Connection works nowhere - pg library, psql CLI all hang.
Container file access returns "WS connection closed".
CRITICAL: Only copy of business data (alldesk_store table) - no backups exist.
Status changed to Awaiting Railway Response Railway • about 3 hours ago
3 hours ago
Having looked into this, the issue appears to be in your application code or configuration rather than the Railway platform itself, which puts it outside what Railway support can resolve directly.
This is exactly the kind of problem the Railway community is good at, so we'd like to open your thread as a community bounty. Railway pays a bounty to the community member who solves it, and threads like this usually get picked up quickly.
Opening it makes this entire thread public, including everything already posted. Nothing becomes public until you decide. Use the buttons below.
- Open to the community - Before you click, take a moment to edit or remove anything you'd rather not share. The thread becomes publicly visible right away.
- Keep it private and close the thread - Nothing becomes public. The thread closes, since this isn't something Railway support can take further.
Status changed to Awaiting User Response Railway • about 3 hours ago
3 hours ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • about 3 hours ago
2 hours ago
Did you redeploy your Postgres service after you increased the volume size?
2 hours ago
Also, invalid record length logs are mostly normal. Can you provide the full error logs of your database?
2 hours ago
@darseen thx! I just redeployed the Volume Resize - no answer Postgres !
Logs:
- 2 failed Deployments (8f0337a0... und 153573a1...)
- Connection timeout expired (10s timeout)
- Container file access: WS connection closed
- Postgres zeigt "Online" aber ist unreachable
Volume Info:
- Volume ID: b354fc4a-042d-4552-ad9a-ac1dedb4acaf
- War: 500MB → Jetzt: 4GB
- Mount Path: /var/lib/postgresql/data
2 hours ago
@darseen Danke für die Anfrage. Hier sind die Logs:
OLD POSTGRES IMAGE ERROR (postgres-ssl:18):
"root" execution of the PostgreSQL server is not permitted.
The server must be started under an unprivileged user ID.
TESTED SOLUTIONS:
- Removed bad startCommand (recovery_target='immediate') ❌ Still failed
- Changed to official postgres:18 image ❌ Still failed
- Restarted container ❌ Still failed
CURRENT STATUS:
- 3 failed deployments (8f0337a0..., 153573a1..., 11f2d86c...)
- Postgres won't start with ANY image
- Connection timeout from other services
- Volume b354fc4a-042d-4552-ad9a-ac1dedb4acaf is unreachable
- Data directory cannot be accessed (WS connection closed)
CONCLUSION:
Database appears permanently corrupted and unrecoverable. Is there a way to:
- Restore from volume snapshots?
- Access raw volume data?
- Force PostgreSQL recovery mode?
This is our only copy of critical business data (alldesk_store table). Any help appreciated!
2 hours ago
The database logs are located in Deployments > View logs > Deploy logs. Can you share those please?
Attachments
2 hours ago
@darseen The deploy logs are completely EMPTY - the container doesn't start at all. The build logs are also empty. This suggests the deployment isn't even beginning to build properly.
This appears to be an infrastructure issue rather than just a Postgres problem. The 4GB volume may not be mounting correctly after the resize.
Additional details:
Service shows "Online" but is unreachable (timeout after 10s)
Container file access returns "WS connection closed"
Tested with both old image (postgres-ssl:18) and official postgres:18 - both fail
Removed problematic startCommand but deployment still fails
Could this be a volume mounting issue after the resize? The fact that NO logs are generated suggests the container can't even initialize. Any insights would be greatly appreciated - this is critical business data.
2 hours ago
The logs can't be empty if the database is crashing. logs might take some time to load. So if you see "Your logs are loading", just wait for them to show up, and provide a screenshot or paste them here. Unfortunately, without any logs, we can't determine the cause of the issue.
2 hours ago
@darseen Thanks for the clarification! I'm checking the logs via the API and they're still empty/not loading. I'll check the Railway dashboard directly to see if they appear there. Let me go to Deployments > View logs > Deploy logs and send you a screenshot if the logs are loading there.
an hour ago
Project: jubilant-heart (ID 5d8a7adb-69d3-4a34-a83a-d8f6531a0c9f)
Service: "Postgres" (ID 39422adc-4110-4b1a-82a4-7c7c638955d9), image postgres:18
Env: production
Problem: The Postgres service is stuck in a crash loop. Deploy logs show, on every start:
Starting Container
"root" execution of the PostgreSQL server is not permitted.
The server must be started under an unprivileged user ID...
So the postgres server process is being launched as root and immediately exits.
This started during a Postgres major-version migration (I also have
postgres-recovery-qmJM on pg15 holding the real data volume "postgres-volume",
and postgres-dump on pg16).
Questions:
-
Is a custom start command on this service causing postgres to run as root
instead of going through docker-entrypoint.sh (which drops to the postgres user)?
What is the correct start command / image config for this service?
-
What is the recommended safe path to get back online without losing the data
on the "postgres-volume" (currently attached to postgres-recovery-qmJM, pg15)?
