5 months ago
Postgres DB keeps crashing on redeploys. Is the DB server having issues opening up port 5432?
Asking to Community since my plan is Hobby.
- Had been working, and all of a sudden started failing with no deployments from my side.
- Plenty of disk space, using 120MB and the quota is 500 MB
- Using private networking. Django settings.py file contains:
PROD_DB = {'ENGINE': 'django.db.backends.postgresql','NAME': "railway",'USER': "postgres",'PASSWORD': "$PASSWORD",'HOST': "postgres.railway.internal",'PORT': 5432} - Unable to ssh or connect using psql command
- Haven't changed any DB configs
- Have only redeployed changes to Django web app, which had breaking schema changes. However, the DB should be independent of the web app.
- Trying to connect from my local dev using the TCP proxy server and port.
Logs:
File "/app/.venv/lib/python3.11/site-packages/django/db/backends/base/base.py", line 269, in connect
self.connection = self.get_new_connection(conn_params)
^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
connection = self.Database.connect(**conn_params)
^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.11/site-packages/psycopg/connection.py", line 127, in connect
raise type(last_ex)("\n".join(lines)).with_traceback(None)
django.db.utils.OperationalError: connection failed: connection to server at "127.0.0.1", port 5432 failed: Connection refused
Is the server running on that host and accepting TCP/IP connections?
Multiple connection attempts failed. All failures were:
- host: 'localhost', port: '5432', hostaddr: '::1': connection failed: connection to server at "::1", port 5432 failed: Connection refused
Is the server running on that host and accepting TCP/IP connections?
- host: 'localhost', port: '5432', hostaddr: '127.0.0.1': connection failed: connection to server at "127.0.0.1", port 5432 failed: Connection refused
Is the server running on that host and accepting TCP/IP connections?Unable to SSH, it dies right away:
% railway ssh --project=$PROJECT --environment=$ENVIRONMENT --service=$SERVICE
✓ Connected to interactive shell root@42fefbb18d69:/app# %
# Dies right awayUnable to connect using PSQL or pgAdmin4:
Public networking TCP proxy is enabled on this server and port.
% PGPASSWORD=$PASSWORD psql -h turntable.proxy.rlwy.net -U postgres -p 55568 -d railway
psql: error: connection to server at "turntable.proxy.rlwy.net" (66.33.22.229), port 55568 failed: server closed the connection unexpectedly
This probably means the server terminated abnormally before or while processing the requestSame error using pgAdmin4 (see screenshot).
Attachments
2 Replies
5 months ago
Your Postgres container itself is crashed, that’s why nothing can connect. The Django error showing 127.0.0.1 is a red herring from psycopg’s connection retry logic.
Click on the Postgres service directly and check its logs (not the Django deploy logs) to see why it crashed. Common causes are OOM or corrupted data from an interrupted operation. Try restarting the Postgres service from the three-dot menu. If it keeps crash-looping, the logs should show the actual error, share those and we can help diagnose further.
Also worth checking if your volume is actually at 120MB or if that’s stale. click on the postgres-volume to see current metrics.
5 months ago
the issue is django is ignoring your PROD_DB config and defaulting to localhost.
to fix this in railway django service add env var: DATABASE_URL=${{Postgres.DATABASE_PRIVATE_URL}}
then in settings.py use dj_database_url.config() instead of hardcoded dict
and add sleep 3 && before your start command (dns needs time to resolve)
also if using alpine image add env var: ENABLE_ALPINE_PRIVATE_NETWORKING=true
the logs show it's trying 127.0.0.1 not postgres.railway.internal which means your DATABASE_URL isn't set right. once you reference the private url properly it should work