Hi,

I am seeing recursive restart of Postgres container with these logs.
These logs are the initial ones when a series of error starts to occur.
Is there any option to have more storage for the database just in order to launch the container so that we could remove unnecessary data through CLI or our application user interface?

Actually, I am not sure the limit of database storage or volumes attached to it with our current plan.
https://help.railway.app/questions/database-storage-limit-5e8c8fd7
seeing this, 50gb is the limit though I am not able to find any documentation about it.

2024-10-31 14:10:00.276 UTC [26] LOG: checkpoint starting: time

Mounting volume on: /var/lib/containers/railwayapp/bind-mounts/f6f62a63-9bd9-4072-9343-6eb1bbe554b2/vol_q9uxujbsd9cofez6

container event container died

2024-10-31 14:11:45.772 UTC [74923] PANIC: could not write to file "pg_wal/xlogtemp.74923": No space left on device

2024-10-31 14:11:45.776 UTC [5] LOG: server process (PID 74923) was terminated by signal 6: Aborted

2024-10-31 14:11:45.776 UTC [5] DETAIL: Failed process was running: INSERT INTO "public"."observations" ("id","traceid","projectid","type","starttime","name","metadata","level","createdat","updatedat","model","internalmodel","internalmodelid","modelParameters","input","prompttokens","completiontokens","totaltokens","unit") VALUES ($1,$2,$3,CAST($4::text AS "public"."ObservationType"),$5,$6,$7,CAST($8::text AS "public"."ObservationLevel"),$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19) ON CONFLICT ("id") DO UPDATE SET "name" = $20, "starttime" = $21, "metadata" = $22, "model" = $23, "modelParameters" = $24, "input" = $25, "unit" = $26, "internalmodel" = $27, "internalmodelid" = $28, "updatedat" = $29 WHERE ("public"."observations"."id" = $30 AND 1=1) RETURNING "public"."observations"."id", "public"."observations"."traceid", "public"."observations"."projectid", "public"."observations"."type"::text, "public"."observations"."starttime", "public"."observations"."endtime", "public"."observations"."name", "public"."observations"."metadata", "public"."observations"."

2024-10-31 14:11:45.776 UTC [5] LOG: terminating any other active server processes

2024-10-31 14:11:45.791 UTC [5] LOG: all server processes terminated; reinitializing

2024-10-31 14:11:45.848 UTC [74926] LOG: database system was interrupted; last known up at 2024-10-31 12:06:08 UTC

2024-10-31 14:11:45.892 UTC [74926] LOG: database system was not properly shut down; automatic recovery in progress

2024-10-31 14:11:45.895 UTC [74926] LOG: redo starts at 0/55949168

2024-10-31 14:11:46.079 UTC [74926] LOG: redo done at 0/58FFFAD0 system usage: CPU: user: 0.08 s, system: 0.07 s, elapsed: 0.18 s

2024-10-31 14:11:46.173 UTC [74926] FATAL: could not write to file "pg_wal/xlogtemp.74926": No space left on device

2024-10-31 14:11:46.202 UTC [5] LOG: startup process (PID 74926) exited with exit code 1

2024-10-31 14:11:46.202 UTC [5] LOG: terminating any other active server processes

2024-10-31 14:11:46.203 UTC [5] LOG: shutting down due to startup process failure

2024-10-31 14:11:46.262 UTC [5] LOG: database system is shut down

Certificate will not expire

PostgreSQL Database directory appears to contain a database; Skipping initialization

2024-10-31 14:11:48.732 UTC [5] LOG: starting PostgreSQL 16.3 (Debian 16.3-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit

2024-10-31 14:11:48.733 UTC [5] LOG: listening on IPv4 address "0.0.0.0", port 5432

2024-10-31 14:11:48.733 UTC [5] LOG: listening on IPv6 address "::", port 5432

2024-10-31 14:11:48.748 UTC [5] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"

2024-10-31 14:11:48.755 UTC [28] LOG: database system was interrupted while in recovery at 2024-10-31 14:11:45 UTC

2024-10-31 14:11:48.755 UTC [28] HINT: This probably means that some data is corrupted and you will have to use the last backup for recovery.

View Deploy details

ⓘ Deployment information is only viewable by project members and Railway employees.