17 days ago
I have a Node.js program running on Railway.
Just now, it gave me a message saying it couldn't connect to the database, which is very unusual. I'm actually connecting to a paid Supabase resource. I'm using a database client locally, and Supabase works fine. I'm pretty sure the problem is with the Railway platform.
Here is log:
SequelizeConnectionError: connect ETIMEDOUT 3.0.121.111:5432 at Client._connectionCallback (/opt/outline/node_modules/sequelize/lib/dialects/postgres/connection-manager.js:145:24) at Client._handleErrorWhileConnecting (/opt/outline/node_modules/pg/lib/client.js:379:19) at Client._handleErrorEvent (/opt/outline/node_modules/pg/lib/client.js:389:19) at Connection.emit (node:events:509:28) at Connection.emit (node:domain:489:12) at Socket.reportStreamError (/opt/outline/node_modules/pg/lib/connection.js:56:12) at Socket.emit (node:events:509:28) at Socket.emit (node:domain:489:12) at emitErrorNT (node:internal/streams/destroy:170:8) at emitErrorCloseNT (node:internal/streams/destroy:129:3) at process.processTicksAndRejections (node:internal/process/task_queues:90:21)
Strangely, in the same service area, my other services also connect to the Supabase service without any problems. I've confirmed it's not a problem on my end; my services have been running stably for over two years.Please fix this issue immediately and help me troubleshoot it—I’m really at my breaking point… This is a core internal collaboration service for the company, and we’re still in working hours!
14 Replies
17 days ago
We have not done anything on our end; this looks to be an external issue.
17 days ago
All good on our end over here.
@Brody Error log is: [onAuthenticate] connect ETIMEDOUT 3.0.149.190:5432
It looks like a network problem. Could you please check this out? I tested the database connection in my office and it works. Do you need me to provide some information?
17 days ago
It's a networking problem, but not one on our end; all is good on our end.
Thank you—I found the cause. It might be that my database connection pool size is set too small.
I'm feeling quite frustrated because these database connection drops are intermittent. I've already investigated the Supabase side, and everything seems fine there. I've also verified that I can connect to Supabase from other networks without any issues.
I suspect that the Railway platform might be intermittently blocking requests to the IP address 3.0.149.190. Could you please look into this for me?
@Brody
I'm fairly certain this isn't a connection pool exhaustion issue, as I've thoroughly investigated that end. My other services are running smoothly, which makes this even more puzzling. I'm not sure if this is a localized network glitch, but for context, all my services are deployed in the Singapore region.
17 days ago
We have monitors for outbound networking availability on every host in our fleet that will page an engineer, and those monitors have been green since the incident yesterday. If there were an issue on our end, we would know about it.
Assuming your platform's monitoring can capture these anomalies, you should be able to see the failed requests originating from my service, right? It should be possible to check the logs for those specific short-lived failures.
I’m using the open-source Outline application, which is extremely stable and straightforward. The error originates directly from the database connection component, which strongly indicates an issue with outbound network connectivity. If the remote service were actually rejecting my requests, I would be receiving a 'Connection Refused' error, not a 'Timeout.' This suggests the network is dropping packets before they even reach the destination.
If such logs are accessible on your end, would you mind helping me investigate these specific instances? I would greatly appreciate it if you could cross-reference the timestamps of my connection timeouts with your outbound traffic logs to see if there's any pattern. @Brody
17 days ago
You have access to the same network flow logs I do.
I would also highly appreciate not being pinged 🙏