Postgres DB full, can't connect

wimgz

FREEOP

9 months ago

Hello,

I'm hosting n8n on Railway and my PostgreSQL database volume is completely full. The database is now inaccessible and stuck in a crash/recovery loop.

The execution_entity table filled up the entire storage over a couple days. I had setup pruning variables to manage this table automatically, but now it can't execute because the volume is full - classic catch-22.

I cannot access the database through any method:

Railway CLI connection fails
Web interface shows connection errors
n8n app is down since it can't reach the DB

Here's what I'm seeing in the PostgreSQL logs:

PANIC: could not write to file "pg_logical/replorigin_checkpoint.tmp": No space left on device
LOG: checkpointer process was terminated by signal 6: Aborted
FATAL: the database system is in recovery mode
Failed to hard-delete executions

I have these pruning settings configured but they can't run:

N8N_EXECUTIONS_DATA_PRUNE=true
N8N_EXECUTIONS_DATA_MAX_AGE=1
EXECUTIONS_DATA_PRUNE_MAX_COUNT=1000

I tried connecting via CLI and the web interface but both fail due to the full disk. I can't increase storage on my current plan.

Is there any way Railway support could help clear some space in the database so I can get back in? Even just deleting old records from execution_entity table would probably fix it.

This is my production instance so I'd really appreciate any help to recover it rather than having to start over.

Thanks

Solved

6 Replies

Railway

BOT

9 months ago

Hey there! We've found the following might help you get unblocked faster:

If you find the answer from one of these, please let us know by solving the thread!

parmstar

EMPLOYEE

9 months ago

Hi - I would suggest upgrading to Pro to get the volume size increased. We would not delete data in your instance.

Status changed to Awaiting User Response Railway • 10 months ago

Status changed to Solved parmstar • 10 months ago

wimgz

FREEOP

9 months ago

Hello, thanks for your reply.

No, i dont't need the extra volume:

on 5GB i probably have 3-4GB of useless data that should have been pruned, and that I will delete as soon as i can access the DB.

If Railway team really cannot modify data in the database, I could temporarily upgrade the volume, do my fixes then scale back.

Do you confirm this is possible?

Or could I access a dump of the database somehow (didn't manage to get it via CLI)?

Thanks

Status changed to Awaiting Railway Response Railway • 10 months ago

parmstar

EMPLOYEE

9 months ago

If Railway team really cannot modify data in the database, I could temporarily upgrade the volume, do my fixes then scale back.

Do you confirm this is possible?

Yes, but you will have one month of Pro for that - you can upgrade, scale, cancel and revet back to Hobby in the next billing cycle.

Status changed to Awaiting User Response Railway • 10 months ago

Status changed to Solved parmstar • 10 months ago

wimgz

FREEOP

9 months ago

OK I upgraded.

Now I can't access the DB to do a VACUUM FULL.

CLI times out

GUI in Dashboard tries to connect indefinitely

Is there something else I can try or is it something weird on my setup ?

Project ID : 3ee58a81-7f2c-489b-b393-5e1d6e35993a

Thanks!

Attachments

image.png

Status changed to Awaiting Railway Response Railway • 9 months ago

wimgz

OK I upgraded. Now I can't access the DB to do a VACUUM FULL. CLI times out GUI in Dashboard tries to connect indefinitely ![](https://station-server.railway.com/attachments/att_01k2831m6he4fs8ffpnqcjy51y)Is there something else I can try or is it something weird on my setup ? Project ID : 3ee58a81-7f2c-489b-b393-5e1d6e35993a Thanks!

wimgz

FREEOP

9 months ago

Somehow I was able to reduce disk usage by cleaning N8N execution data with this query in a N8N Postgres Query node:

TRUNCATE TABLE public.execution_data RESTART IDENTITY;

SELECT

  pg_size_pretty(pg_total_relation_size('public.execution_data')) AS total_after,

  pg_size_pretty(pg_total_relation_size((

    SELECT reltoastrelid

    FROM pg_class c

    JOIN pg_namespace n ON n.oid = c.relnamespace

    WHERE n.nspname = 'public' AND c.relname = 'execution_data'

  )::regclass)) AS toast_after;

Status changed to Solved parmstar • 9 months ago

Welcome!