a month ago
We're experiencing intermittent asyncio.TimeoutError errors on our backend service when connecting to PostgreSQL. The issue started this morning and is recurring periodically (morning and evening). The backend service becomes unable to execute database queries, causing login and other database-dependent endpoints to fail.
Error messages:
asyncio.exceptions.TimeoutError
Stack trace shows the error occurs during SSL connection handshake:
File: /usr/local/lib/python3.10/site-packages/asyncpg/connect_utils.py, line 744, in _create_ssl_connection
The timeout happens in asyncpg.connection.connect() when trying to establish an SSL connection to PostgreSQL
Logs:
Deploy logs show repeated timeout errors at:
2026-06-11T21:07:53Z
2026-06-11T21:10:02Z
2026-06-11T21:11:02Z
2026-06-11T21:12:12Z
2026-06-11T21:12:42Z
All errors follow the same pattern: SSL handshake timeout during connection creation.
What we've verified:
CPU and memory usage on both backend and PostgreSQL are normal
Both services are marked as "Online" in the dashboard
PostgreSQL is responding (service is running)
The issue is NOT related to code changes (unaccent() function was deployed days ago and worked fine)
GitHub repo:
https://github.com/sistemacep-collab/cep-controle-alunos
Backend service: backend/ directory
Database configuration: backend/app/db/session.py
Config: backend/app/core/config.py
Project details:
Project: CEP CONTROLE ALUNOS
Environment: Production
Backend: FastAPI + Python with SQLAlchemy async
Database: PostgreSQL on Railway with SSL enabled
Could you please investigate the PostgreSQL SSL certificate status and any potential SSL-related issues on the Railway infrastructure?
3 Replies
a month ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • 26 days ago
25 days ago
Try adding pool_pre_ping=True and pool_recycle=259 to your create_async_engine() call. So that it detects and replaces dead connections before your app tries to use them.
25 days ago
Your occasional asyncio.TimeoutError during SSL Handshakes are occurring due to Railway infrastructure bandwidth throttling/connection pooling limits being reached, not because your SSL certificate expired.
Your FastAPI backend and PostgreSQL DBs are both hosted on Railway so when you get these timeouts it means your backend cannot successfully make an SSL tunnel to your Postgres DB and the packet is being dropped during network handshake. Because this occurs regularly during morning/evening hours, you are likely experiencing peak traffic to your application. When your application scales out to multiple async workers all trying to connect to the DB at the same time, your Railway internal network or single Postgres instance will reach a concurrency limit for open connections and begin dropping them.
How to fix/make better:
Connection Limits: Login to your Railway Postgres dashboard and check your current plan's maximum amount of allowed concurrent connections are not being hit during peak request times.
Client Pool Settings: Open backend/app/db/session.py and double check your asyncpg pool settings are not too aggressively opening minimum connections or thrashing connections. Tune your max_ size.
SSL Parameters: Since the connection cannot successfully complete the SSL handshake you can adjust ssl parameters on your connection string (i.e. ssl=require) or even try adjusting SSL-handshake TCP timeouts within asyncpg itself).
Railway? : If your DB is receiving normal levels of requests/responses then latency in routing between your app <-> DB on Railway is also possible. This will likely require opening a support ticket.
asyncpg does connection pooling natively so misconfiguration here can easily cause timeout cascades.
boinapellyavaneesh-coder
Your occasional asyncio.TimeoutError during SSL Handshakes are occurring due to Railway infrastructure bandwidth throttling/connection pooling limits being reached, not because your SSL certificate expired. Your FastAPI backend and PostgreSQL DBs are both hosted on Railway so when you get these timeouts it means your backend cannot successfully make an SSL tunnel to your Postgres DB and the packet is being dropped during network handshake. Because this occurs regularly during morning/evening hours, you are likely experiencing peak traffic to your application. When your application scales out to multiple async workers all trying to connect to the DB at the same time, your Railway internal network or single Postgres instance will reach a concurrency limit for open connections and begin dropping them. How to fix/make better: Connection Limits: Login to your Railway Postgres dashboard and check your current plan's maximum amount of allowed concurrent connections are not being hit during peak request times. Client Pool Settings: Open backend/app/db/session.py and double check your asyncpg pool settings are not too aggressively opening minimum connections or thrashing connections. Tune your max_ size. SSL Parameters: Since the connection cannot successfully complete the SSL handshake you can adjust ssl parameters on your connection string (i.e. ssl=require) or even try adjusting SSL-handshake TCP timeouts within asyncpg itself). Railway? : If your DB is receiving normal levels of requests/responses then latency in routing between your app <-> DB on Railway is also possible. This will likely require opening a support ticket. asyncpg does connection pooling natively so misconfiguration here can easily cause timeout cascades.
25 days ago
We investigated the issue and implemented several improvements to the database connection management:
- Increased the connection pool size from the default 5 connections to 20 connections, with an additional 10 overflow connections available when needed.
- Increased the connection timeout from 10 seconds to 30 seconds.
- Enabled
pool_pre_pingto automatically detect and replace stale or closed connections before they are used. - Enabled
pool_recycleto refresh connections every hour and prevent long-lived connection issues.
Since applying these changes, the intermittent connection timeout errors have not reoccurred, and the application has been operating normally. The issue appears to have been related to connection pool exhaustion and/or stale database connections under load.
