a year ago
Hello Railway Community and Devs,
I am currently running a Directus CMS instance backed by a PostgreSQL database on Railway, and I've encountered a performance issue that seems to be related to PostgreSQL's checkpoint behavior. After running for a period, my PostgreSQL logs show lengthy checkpoint durations, which I believe are causing 503 errors in my Directus application due to the database becoming temporarily unresponsive.
Here are the specific log entries:
2024-03-11 00:42:39.357 UTC [76] LOG: checkpoint starting: time
2024-03-11 00:42:43.579 UTC [76] LOG: checkpoint complete: wrote 44 buffers (0.3%); 0 WAL file(s) added, 0 removed, 0 recycled; write=4.211 s, sync=0.005 s, total=4.222 s; sync files=17, longest=0.003 s, average=0.001 s; distance=158 kB, estimate=205 kB; lsn=0/4829510, redo lsn=0/481DDD8Given the above, I am looking for guidance on how to optimize my PostgreSQL setup to mitigate these issues. Specifically, I am interested in adjusting parameters such as checkpointtimeout, maxwalsize, and checkpointcompletion_target to improve performance. However, I am unsure how to apply these settings in the Railway environment, as direct access to the postgresql.conf file may not be available.
Furthermore, I have considered using Railway's "Start Command" feature to execute custom initialization scripts or commands that might help, but I'm uncertain if this is the best approach or how to properly implement it for PostgreSQL configuration purposes.
Could anyone provide advice on how to effectively configure PostgreSQL in this context, or share if there's a way to apply advanced configuration settings through Railway's platform? Any examples or guidance on utilizing "Start Command" for this purpose, if feasible, would be greatly appreciated.
Thank you in advance for your support and insights!
4 Replies
a year ago
checkpoints are asynchronous as to not block any read / write access to the database, your issue is very likely elsewhere.
when you are seeing these 503 errors from the directus app, what errors are you getting in the deployment logs?
a year ago
Thank you for your response and for clarifying the nature of checkpoints in PostgreSQL. Following your advice, I revisited the deployment logs around the time when the 503 errors were occurring in the Directus API. The 503 errors happened when trying to access specific API endpoints, and the Directus application logs indicated "503 Service Unavailable" without much detail about the underlying cause related to the database.
However, I did notice PostgreSQL error logs mentioning lengthy checkpoint operations around the same time frames, which initially led me to suspect that these might be related to the 503 issues. Since checkpoints are asynchronous and shouldn't block read/write access, I understand now that the root cause might be different.
Unfortunately, I couldn't find more specific error messages in the PostgreSQL logs directly pointing to the reason for these 503 errors, other than the general checkpoint information. Is there any other type of log or detail I should look for within the Railway deployment environment or Directus logs that might help pinpoint the issue? Additionally, if there are common pitfalls or settings in Directus or PostgreSQL that I should double-check or optimize, I'd appreciate any insights.
Thank you again for your help. Looking forward to any suggestions or guidance you can provide to resolve these service interruptions.
a year ago
Like this one
Attachments
a year ago
definitely look into the service logs of directus itself, I strongly believe there is no issues with postgres.