Stuck lock prevents the app from starting
sharatpotukuchi
PROOP

11 days ago

Service: vedic-astro-backend (ID: c7550e5a-4799-4331-9622-624eb2b106e2)

Project: vedicastro-io

Environment: vedicastro-production

My app runs database migrations on startup. A migration got stuck mid-execution and left an advisory lock (lock_key: 147501) held in Postgres.

Every deployment attempt times out trying to acquire this lock after 5 seconds:

[MIGRATION_EVENT] {"event": "migration_lock_timeout", "lock_key": 147501, "strict_mode": false, "waited_ms": 5022}

The database is responsive, but this stuck lock prevents the app from starting.

Current deployment: aaa5bb3a-3e74-42d6-bd7f-3ea9ad4fa0db (stuck on healthcheck)

Can you please manually clear the advisory lock in my Postgres database (lock_key: 147501)? A SELECT pg_advisory_unlock_all(); should do it.

Once cleared, my deployment should proceed normally.

$20 Bounty

2 Replies

Railway
BOT

11 days ago

This thread has been opened as a bounty so the community can help solve it.

Status changed to Open Railway 11 days ago


Try going to the Console tab of your database, run psql, then run:


SELECT pid, usename, state, age(clock_timestamp(), query_start), query

FROM pg_stat_activity

WHERE query LIKE '%147501%' 

   OR query LIKE '%pg_advisory%'

   OR state = 'active';

Then: SELECT pg_terminate_backend(<PID_NUMBER>); where <PID_NUMBER> is the PID of that query


Anonymous
FREE

11 days ago

The suggested pg_advisory_unlock(147501) / pg_advisory_unlock_all() from a fresh Console session will not clear a lock held by another backend: advisory locks are session-owned, so those functions only release locks held by the session executing them.

First identify the actual holder (and confirm it is a stale migration, not a migration still doing useful work):

SELECT l.pid, l.mode, l.granted, l.classid, l.objid, l.objsubid,
       a.usename, a.application_name, a.state,
       a.backend_start, a.query_start, a.query
FROM pg_locks AS l
JOIN pg_stat_activity AS a ON a.pid = l.pid
WHERE l.locktype = 'advisory'
ORDER BY l.granted DESC, a.query_start;

For a single-bigint lock such as pg_advisory_lock(147501), the low 32-bit portion commonly appears as objid = 147501; inspect the lock tuple rather than assuming the representation.

After verifying the PID is the abandoned migration connection, release its session by terminating that backend:

SELECT pg_terminate_backend(<stale_migration_pid>);

That automatically releases its advisory locks. It can also roll back that session's uncommitted transaction, so check the migration table/logs and take a backup or snapshot first; do not terminate an active migration. If your Railway DB role cannot call pg_terminate_backend, ask Railway support to end the identified backend, or restart Postgres only as a last resort because it disconnects all clients.

For prevention: run migrations as a single release/one-off job rather than in every replica's startup path, use a bounded advisory-lock wait/retry, and ensure the migration process closes its DB connection cleanly on SIGTERM.


Welcome!

Sign in to your Railway account to join the conversation.

Loading...