Postgresql + Railway + Strapi (Node.js)
genai_works
PROOP

2 years ago

I always have this issue in Postgresql logs

2024-04-30 02:43:15.804 UTC [66070] FATAL: sorry, too many clients already

with this database.js config on Strapi side

module.exports = ({ env }) => ({
connection: {
client: 'postgres',
connection: {
host: env('PGHOST'),
port: env('PGPORT'),
database: env('PGDATABASE'),
user: env('PGUSER'),
password: env('PGPASSWORD'),
ssl: {
rejectUnauthorized: false,
ca: env("DATABASE_CA", undefined)
},
charset: 'utf8mb4',
},
debug: false,
pool: {
min: 0,
max: 100,
acquireTimeoutMillis: 30000,
createTimeoutMillis: 30000,
destroyTimeoutMillis: 5000,
idleTimeoutMillis: 30000,
reapIntervalMillis: 500,
createRetryIntervalMillis: 100,
propagateCreateError: false,
},
},
});

The connection works well for the application, but I cannot connect to Postgresql via "Data" tab in railway or via 3rd party service (i.e. service for creating backups for AWS S3).

Am I able to configure max_connections variable? How can I debug this issue?

17 Replies

brody
EMPLOYEE

2 years ago

Postgres has a default max connection of 100 (Postgres itself, not Railway) so if you don't require 100, I would significantly lower that, Strapi has a default of max 10 fwiw.

Am I able to configure max_connections variable?

Yes, you can, but 100 is far more than you would likely need.


genai_works
PROOP

2 years ago

Could you please provide a way to increase this number?

Our maximum spotted online was 100 people at the same time (40k people per week). So I am wondering how to calculate how many connections we need to handle big online.


genai_works
PROOP

2 years ago

I was able to setup creating backups at the moment only by setting to check postgresql availability each 5 minutes, and sometimes it have success to connect. But it doesn't look good as even when online is less than 5 people, it says that "sorry, too many clients already". Probably the thing is that they send a lot of API requests, that's why all 100 connections may be used at the same time


brody
EMPLOYEE

2 years ago

Increasing the connection limit of Postgres would only be a bandaid solution, the correct solution here would be to put pgbouncer in front of your database, then have Strapi connect to pgbouncer so that connections can be reused.

We have a pgbouncer template available that you can easily deploy into your project from the + New button in the top right of your project canvas.


genai_works
PROOP

2 years ago

Thank you, I will try and let you know


genai_works
PROOP

2 years ago

I have setup pgbouncer, which I believe helps, but the problem hasn't gone

Attachments


brody
EMPLOYEE

2 years ago

Are you making sure Strapi is connecting to pgbouncer instead of directly to Postgres?


genai_works
PROOP

2 years ago

You are right

Strange, when I try to connect (via Strapi) to pgbouncer, I receive knex mistake

There seems to be an unexpected error, try again with --debug for more information

KnexTimeoutError: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call


genai_works
PROOP

2 years ago

Strapi

PGDATABASE=railway
PGHOST=monorail.proxy.rlwy.net
PGPASSWORD=real_password
PGPORT=48270
PGUSER=postgres

Bouncer
postgres://postgres:real_password@monorail.proxy.rlwy.net:48270/railway

Postgresql

postgresql://postgres:real_password@viaduct.proxy.rlwy.net:24875/railway

I was able to connect to both via PgAdmin


genai_works
PROOP

2 years ago

Number of sessions reach 100 for both (idle sessions)

Attachments


genai_works
PROOP

2 years ago

After all, it seems to me that Strapi has a built-in "pgbouncer" (knex pooling system), and don't need it


genai_works
PROOP

2 years ago

I have tried to configure 2nd pgbouncer template (by Jonathan), and the same. I am able to connect via pgAdmin, but not via Strapi (connection timeout)


brody
EMPLOYEE

2 years ago

Let go back to my original proposed solution, don't set the pool max at 100.


genai_works
PROOP

2 years ago

Maybe some important detail to notice: server has 6 replicas (+ 1 original server, if I am correct in counting). Not sure if it influences a lot on number of idle connections.

So, I tried to decrease max pool number from 100 to 10. And you are right - it helped. If I understand correctly, each replica creates 10 idle connections by default (that's why 66 connections opened, so it means that original server is included in replicas number and 6 servers are working, not 7). And after all, I don't see this mistake in postgresql logs "sorry, too many clients already". Now it's good.

So, my next assumption is that I need to configure max number based on number of replicas that I have. In case I would have 50 replicas, I should set max value as 2 or even 1.


genai_works
PROOP

2 years ago

My only concern that I do not understand how I can, in the future, upscale database to handle bigger online using Railway tools. If I don't need pgbouncer, and don't have built-in-solutions to do it on my own, probably the only way is contacting support for some kind of horizontal scailing of postgresql.


brody
EMPLOYEE

2 years ago

Railway can not horizontal scale a database but that's not needed as increasing the connection limit only involves running a few simple queries on your database.


genai_works
PROOP

2 years ago

As far as I understood, in case of need I can ask Railway support to do this queries, not on my own


Loading...