Unreachable over the private or public network?

over private networking as it seems but weirdely enough I can reach it over Tailscale

does this help?

What is the source service that is trying to access the database

https://railway.com/project/357a7a66-a372-47f0-b2ae-2d8e2b6f1f32/service/8812873e-b82c-46c5-82d3-e121bbc8580e?environmentId=86853589-64b5-48b3-9eda-2174a7ce26b2

Can the data tab access it if you add a TCP proxy

Yeah I'm able

moving it over to us-west also didn't make any difference, healthchecks don't even go through

I'm not seeing other reports, and the database and backend aren't using the beta IPv4 networking, so I'm not sure of the issue.

I'm also not seeing any errors in the logs besides the failing health check?

let me try railway ssh

psql is able to connect, yeah might be our fault

will investigate more

psql over the private network?

did ssh into the service container, installed psql there and a connection was made

just weird that we're getting these errors from the database

even tho no deploy was made and the postgres metrics is normal

What is your timeout set to?

whatever typeorm uses by default

i'll try increasing it but doubt its that

even satellite services, with totally different source code than ours, are also unable to connect to our databse

I don't know if it's related but I am having something somewhat similar, one of my services stopped working and when restarting the deploy fails on the health check. it looks like it might be unable to connect to the pg db that I have running, but I can connect to it over public network, (maybe private network issue?) nothing have changed in the service in the last few days no new deployments no changes. Any support would be appreaciated

same here, still unable to debug

seems like that some connections go through

now the problem is also affecting our other project, completely unrelated

One strange thing I noticed is that the "Architecture" UI for a PG DB usually show how much of the db storage is used, and it does for the project that I still have running fine, but not longer does that for the one that is having the problem
see the difference in the screenshots

ohh same here

wish I could dettach the volume re-attach to another service

tried to do a backup and restore from it, still having issues

all of our major providers are still up and no issues whatsover

can anyone from the Railway team confirm that they're looking into it? would keep me calm

dumping the database and restoring it into another service solved my issue for one of my projects
volume size appears ok without any problems

When did you all first see errors?

14:30-14:50 Brazilian time

my only issue now is with this database:

I gotta start asking for timestamps in UTC

in your timezone:

Please provide a direct link to your database.

I'm sorry but that's not quite what I asked for, please provide the URL of your browser's omni bar while opened to the database.

I've rasied this to the infra team.

people that highly depends on their service, do a pgdump and pgrestore to another service, I'm in the middle of doing it for another project of ours.

also, use an ubuntu container and railway ssh for a faster dump

just did a pgrestore and pgdump for both of our databases and they're back up again, feel free to do anything to those services (well, as long as you don't delete them)

make sure to increase your connections count to a really high value and then try to connect

our connections were pilling up and thus we were getting too many clients

We are actively looking into the cause.

and obviously, run a railway backup just to be sure

Hi, can I know what happened?

A host's networking locked up.

great to know, would a high availability pg cluster prevent that from happening in the future or was that happening on the service itself? looking for ways to prevent that from happening again.

Unlikely, since something could go wrong with the pooler service, there's still a single point of failure.

there's probably someway to replicate that, for the service would replicas do the trick? i dont know if they're deployed to the same host

They are not deployed on the same host, but then your own code would have to handle fallback to another pooler if one isn't available, since we don't handle that on the private network

probably i would also need an API gateway to automatically fail over in case a service replica goes down, damn HA is hard 💀

fair enough, will look into ways, thanks brody

thread can be closed

No problem!

!s