Severity: P1 — production down, customers unable to use the app

Project: courageous-endurance

Environment: production

Affected service: Postgres (plugin, with attached "postgres-volume")

Related services: fichaje (API, api.fichme.com), backend-worker, PgBouncer, Redis

Project ID / Service ID: [PEGA AQUÍ los IDs desde Settings de cada servicio]

Plan: [tu plan, p.ej. Pro]

Time window (UTC): 2026-07-14, from ~11:56 UTC and ongoing

SUMMARY

Our production Postgres instance has become unresponsive. Trivial queries

(primary-key lookups) are taking 30–90+ seconds, and the database is now

refusing/hanging new connections entirely. This persists even after we removed

ALL application load (scaled the API to 0 replicas and stopped the worker), which

tells us the problem is with the Postgres instance itself, not our connection pool.

Critically: Railway's own "Data" tab for the Postgres service also fails with

"Attempting to connect to the database…", and external clients (pgAdmin) cannot

connect either. So even with zero application traffic, nothing can open a

connection to this database.

TIMELINE (UTC, 2026-07-14)

• ~11:56 Query latency began climbing under normal peak load (queries 5–8s).

• ~12:1x We triggered a redeploy of the Postgres service. It got STUCK on

      "Deploying" for ~6m40s and never completed; we aborted it.

• ~12:13 Query latency escalated to 30–90s; Prisma connection-pool timeouts.

• ~12:1x Scaled API (fichaje) down to 0/1 replicas and stopped the worker to

      shed all load.

• now Database still not accepting connections. Railway Data tab + pgAdmin

      both fail to connect.

SYMPTOMS / EVIDENCE

• Postgres shows "Online" in the canvas, but does not accept connections.

• Application logs (Prisma):

  "Timed out fetching a new connection from the connection pool

  (Current connection pool timeout: 30, connection limit: 3)"

  "Slow query: Company.findUnique took 91886ms WHERE {slug: ...}"

  "Slow query: SessionLog.findFirst took 91650ms ..."

  (These are primary-key / indexed lookups that normally take <5ms.)

• Railway Postgres "Data" tab: stuck on "Attempting to connect to the database…"

• pgAdmin (via the public proxy, superuser): cannot connect / times out.

WHAT WE'VE ALREADY TRIED

• Aborted the stuck Postgres redeploy.

• Scaled the API to 0/1 replicas and stopped the background worker (all app load removed).

• Attempted to connect via pgAdmin (public proxy, postgres superuser) and via the

  Railway Console — both fail.

WHAT WE NEED FROM YOU (URGENT)

1. Is the Postgres instance/container actually healthy, or did the aborted

   deployment leave it wedged / in crash-recovery? Please check its internal state.

2. Is the "postgres-volume" full or is there any disk/IO issue on the instance?

   (We suspect a possible disk-full condition — please confirm current usage.)

3. Is the instance hitting a CPU/RAM limit, or is it in an OOM/restart loop?

4. Is the internal networking / TCP proxy to this Postgres service functioning?

5. If the instance is wedged, can you help perform a clean restart of the Postgres

   service and confirm it comes back accepting connections?

IMPACT

Full production outage. All employees across all tenant companies are unable to

clock in/out or log in. This is business-critical and time-sensitive.