3 months ago
Hi,
I have a postgres set up in my environemt where I've connected my main service and external service trigger.dev and it works fine. Tried to add a PgBouncer via this template https://railway.com/deploy/OpUzwe that coonects to my postgres and have trigger connection to the PgBouncer instead. Looking at my architecure page here on Railway it seems like I've done it correctly but trigger tasks that worked with a direct connection to the postgres does not work with connection to the PgBouncer. Below are logs from PgBouncer:
pgbouncer 12:06:04.07 INFO ==>
pgbouncer 12:06:04.10 INFO ==> Welcome to the Bitnami pgbouncer container
pgbouncer 12:06:04.10 INFO ==> Subscribe to project updates by watching https://github.com/bitnami/containers
pgbouncer 12:06:04.11 INFO ==> NOTICE: Starting August 28th, 2025, only a limited subset of images/charts will remain available for free. Backup will be available for some time at the 'Bitnami Legacy' repository. More info at https://github.com/bitnami/containers/issues/83267
pgbouncer 12:06:04.11 INFO ==>
pgbouncer 12:06:04.44 INFO ==> Starting PgBouncer setup
pgbouncer 12:06:04.46 INFO ==> Validating settings in PGBOUNCER_* env vars...
pgbouncer 12:06:04.47 INFO ==> Initializing PgBouncer...
pgbouncer 12:06:04.62 INFO ==> Waiting for PostgreSQL backend to be accessible
2025-12-03 12:06:04.694 UTC [1] LOG got SIGTERM, shutting down, waiting for all clients disconnect
2025-12-03 12:06:04.818 UTC [1] LOG client connections dropped, exiting
pgbouncer 12:06:05.50 INFO ==> Backend pgvector.railway.internal:5432 accessible
pgbouncer 12:06:05.51 INFO ==> Configuring credentials
pgbouncer 12:06:05.79 INFO ==> Creating configuration file
pgbouncer 12:06:08.36 INFO ==> Loading custom scripts...
pgbouncer 12:06:08.39 INFO ==> PgBouncer setup finished!
pgbouncer 12:06:08.40 INFO ==> Starting PgBouncer
2025-12-03 12:06:08.917 UTC [1] LOG kernel file descriptor limit: 1048576 (hard: 1048576); max_client_conn: 120, max expected fd use: 172
2025-12-03 12:06:08.950 UTC [1] LOG listening on 0.0.0.0:6432
2025-12-03 12:06:08.951 UTC [1] LOG listening on [::]:6432
2025-12-03 12:06:08.951 UTC [1] LOG listening on unix:/tmp//.s.PGSQL.6432
2025-12-03 12:06:08.951 UTC [1] LOG process up: PgBouncer 1.24.1, libevent 2.1.12-stable (epoll), adns: c-ares 1.18.1, tls: OpenSSL 3.0.17 1 Jul 2025
Stopping Container
2025-12-03 12:06:59.626 UTC [1] LOG C-0x55be6991edd0: railway/postgres@100.64.0.2:62500 login attempt: db=railway user=postgres tls=no replication=no
2025-12-03 12:07:08.916 UTC [1] LOG stats: 0 xacts/s, 0 queries/s, 0 client parses/s, 0 server parses/s, 0 binds/s, in 0 B/s, out 0 B/s, xact 0 us, query 0 us, wait 0 us
2025-12-03 12:07:14.836 UTC [1] LOG S-0x55be699278e0: railway/postgres@10.214.96.254:5432 closing because: connect failed (age=15s)
2025-12-03 12:07:29.836 UTC [1] LOG S-0x55be69927ba0: railway/postgres@10.214.96.254:5432 closing because: connect failed (age=14s)
2025-12-03 12:07:32.644 UTC [1] LOG C-0x55be6991f090: railway/postgres@100.64.0.3:54642 login attempt: db=railway user=postgres tls=no replication=no
2025-12-03 12:07:45.053 UTC [1] LOG S-0x55be69927ba0: railway/postgres@10.214.96.254:5432 closing because: connect failed (age=15s)
2025-12-03 12:07:59.915 UTC [1] LOG C-0x55be6991edd0: railway/postgres@100.64.0.2:62500 closing because: client_login_timeout (server down) (age=60s)
I probably haven't connected PgBouncer to postgres correctly but I can't figure out where I've might added wrong variables.
Best,
Jonas
3 Replies
3 months ago
Hey there! We've found the following might help you get unblocked faster:
If you find the answer from one of these, please let us know by solving the thread!
3 months ago
This thread has been marked as public for community involvement, as it does not contain any sensitive or personal information. Any further activity in this thread will be visible to everyone.
Status changed to Open ray-chen • 3 months ago
3 months ago
Hello Jonas!
I saw this thread has been quiet for over a week, so I wanted to step in and do a proper check on your setup to get this sorted for you.
Looking at your architecture, your logic is actually sound. However, the combination of pgvector (Postgres 16+) and the behavior of Bitnami images on the internal network can create a few invisible hurdles.
The error closing because: connect failed in your logs points to a Stale DNS / IP Mismatch, likely compounded by a hidden authentication conflict. PgBouncer is likely connecting using a host/IP (10.214...) that no longer matches the current database allocation, which can happen if the host was hard-coded earlier.
Here are 3 steps to stabilize the connection and fix the auth protocol:
1. Prevent Auth & Client Failures
Since you are using pgvector (which enforces SCRAM encryption) and
Trigger.dev (Node.js), we need to explicitly tell PgBouncer how to handle them.
Go to your PgBouncer Service Variables and add these:
- PGBOUNCER_AUTH_TYPE = scram-sha-256(Helps avoid auth issues when Postgres is configured with SCRAM (as in PG16+))
- PGBOUNCER_IGNORE_STARTUP_PARAMETERS = extra_float_digits(Fixes connection drops common with Node.js/Trigger clients)
2. Establish Reference Variables
This is the most common issue with templates versus Railway's dynamic infrastructure. We need to ensure Railway automatically updates PgBouncer if the database moves.
Check your POSTGRESQL_HOST variable in PgBouncer:
- If it looks like plain white text: Please delete it.
- You should reference your service variable, like:POSTGRESQL_HOST="${{pgvector.RAILWAY_PRIVATE_DOMAIN}}"
This creates a dynamic link that prevents the "Stale IP" issue you are seeing in the logs.
3. The Chain Restart (Crucial)
To flush the DNS cache and apply these changes, please restart the services in this specific order:
a) Redeploy Postgres -> Wait for it to be green/active.
b) Redeploy PgBouncer -> This forces it to resolve the new IP address of the database.
For your peace of mind, here is the verified config that works for this stack:# Connection (Reference Variables)
POSTGRESQL_HOST="${{pgvector.RAILWAY_PRIVATE_DOMAIN}}"
POSTGRESQL_PORT="5432"
POSTGRESQL_USERNAME="${{pgvector.PGUSER}}"
POSTGRESQL_PASSWORD="${{pgvector.PGPASSWORD}}"
POSTGRESQL_DATABASE="${{pgvector.PGDATABASE}}"
# PgBouncer Settings
PGBOUNCER_AUTH_TYPE="scram-sha-256"
PGBOUNCER_IGNORE_STARTUP_PARAMETERS="extra_float_digits"
PGBOUNCER_POOL_MODE="session"
PGBOUNCER_DATABASE="*"
# Service Port
PORT="6432"
(Note on PGBOUNCER_DATABASE="*": I set this to allow all connections for simplicity, but you can optionally set it to ${{pgvector.PGDATABASE}} later if you prefer stricter pooling.)
Give that a spin. I'll keep an eye on this thread in case you need anything else!
majewskibartosz
Hello Jonas!I saw this thread has been quiet for over a week, so I wanted to step in and do a proper check on your setup to get this sorted for you.Looking at your architecture, your logic is actually sound. However, the combination of pgvector (Postgres 16+) and the behavior of Bitnami images on the internal network can create a few invisible hurdles.The error closing because: connect failed in your logs points to a Stale DNS / IP Mismatch, likely compounded by a hidden authentication conflict. PgBouncer is likely connecting using a host/IP (10.214...) that no longer matches the current database allocation, which can happen if the host was hard-coded earlier. Here are 3 steps to stabilize the connection and fix the auth protocol:1. Prevent Auth & Client FailuresSince you are using pgvector (which enforces SCRAM encryption) andTrigger.dev (Node.js), we need to explicitly tell PgBouncer how to handle them.Go to your PgBouncer Service Variables and add these:- PGBOUNCER_AUTH_TYPE = scram-sha-256(Helps avoid auth issues when Postgres is configured with SCRAM (as in PG16+))- PGBOUNCER_IGNORE_STARTUP_PARAMETERS = extra_float_digits(Fixes connection drops common with Node.js/Trigger clients)2. Establish Reference VariablesThis is the most common issue with templates versus Railway's dynamic infrastructure. We need to ensure Railway automatically updates PgBouncer if the database moves.Check your POSTGRESQL_HOST variable in PgBouncer:- If it looks like plain white text: Please delete it.- You should reference your service variable, like:POSTGRESQL_HOST="${{pgvector.RAILWAY_PRIVATE_DOMAIN}}"This creates a dynamic link that prevents the "Stale IP" issue you are seeing in the logs.3. The Chain Restart (Crucial)To flush the DNS cache and apply these changes, please restart the services in this specific order:a) Redeploy Postgres -> Wait for it to be green/active.b) Redeploy PgBouncer -> This forces it to resolve the new IP address of the database.For your peace of mind, here is the verified config that works for this stack:# Connection (Reference Variables)POSTGRESQL_HOST="${{pgvector.RAILWAY_PRIVATE_DOMAIN}}"POSTGRESQL_PORT="5432"POSTGRESQL_USERNAME="${{pgvector.PGUSER}}"POSTGRESQL_PASSWORD="${{pgvector.PGPASSWORD}}"POSTGRESQL_DATABASE="${{pgvector.PGDATABASE}}"# PgBouncer SettingsPGBOUNCER_AUTH_TYPE="scram-sha-256"PGBOUNCER_IGNORE_STARTUP_PARAMETERS="extra_float_digits"PGBOUNCER_POOL_MODE="session"PGBOUNCER_DATABASE="*"# Service PortPORT="6432"(Note on PGBOUNCER_DATABASE="*": I set this to allow all connections for simplicity, but you can optionally set it to ${{pgvector.PGDATABASE}} later if you prefer stricter pooling.)Give that a spin. I'll keep an eye on this thread in case you need anything else!
3 months ago
Hi there,
Changed to this configuration, except that I run in transaction mode instead of session, and initial testing shows it's running smoothly.
Thanks a ton!
Status changed to Solved itsrems • 3 months ago