Does the Redis service show any logs?

nothing that seems notable. I am not sure the request is even making it there

I suspect a network issue

Are you using the private service URL to connect to it?

yes

What exactly are the error messages you're getting?

timeouts

Could you send the exact error message?

well that was weird. it's working now despite me not doing anything??

hmm it is likely to time out again though

it worked once in prod and now it's taking forever in dev

BTW the outage broke both my environments

I can't find redis failures anymore after all the redeployments

I hate doing this from my phone

I haven't been home all day

lol only one message made it through. we're back to timeouts

it's still broken. ugh

I was hoping it would be a transient thing that would self heal but nope it's been hoooours

stopped working a bit after lunchtime

I'll work on adding better logging in the meantime

Railway nightmare today

grrr now my production Postgres is hanging and the redeploy is too

today has been a bad day

it finally deployed but I still can't connect to it. wtf

doesn't even work on the website

about the Postgres service, do you see any kind of error logs?

I redeployed it like twice. it says active but nothing shows up here

can't even connect via the website

can you remove your current deployment and then try a fresh deployment by hitting CTRL + K and then deploy latest commit

Can I get a link to that service?

https://railway.com/project/ed0106a8-e1db-4e84-863b-86f3ebd4a385/service/6dabbbbc-a680-4f5e-87a5-e3b86c8d637e?environmentId=ae9806dc-d287-49a4-97f6-b617a44b79c4

it doesn't give me that option but I did deploy source image or whatever

That option doesnt exist on a postgres image. It's alright

waiting for it to come online and looking at it

Thats indeed odd, I'll wait for the team response on that, sorry for any issue.

it's taking its sweet time

this is like, my third attempt at redeploying it

The creating containers step randomly might take some time. We are working on a project that will fix that and a few other issues to make things much faster

I'm really frustrated that both my dev and prod environments stopped working after lunch today and have been hosed for hours

even now the dev bot isn't working

the incident earlier broke my app

The degraded performance in US-East?

yes. everything was fine until that incident

afterwards? it has been broken all day

tbh it might also be partly a Qdrant issue. my app logs show it hanging a lot. or it's a networking thing idk. AWS has been unreliable lately

Is your Qdrant hosted on AWS?

I don't see qdrant on this project

I hosted it externally because I didn't know any better. does Railway offer hosting for it?

it's on qdrant.io

and they use us-east-1 for my cluster

Railway does! I host my qdrant instances on Railway always

hmm then it would be good to just migrate

Direct link for it:

https://railway.com/deploy/i1tz3T

beat me to it

it shows as active but the db connection still isn't working

Interesting... Let me check something internally

lol it's still unable to be connected to. I'm working on setting up Qdrant on Railway and I hope it actually works

Need to bring this up to the right team. You should be good to connect with an external DB viewer like tableplus

going to get this fixed though

it's not working

I use Jetbrains and their DB connectivity stuff

I can connect to my DEV postgres but not PROD

Fixed (Paulo the goat)

Reload yor web page and it should be good!

the web page works but the external connection is still not working

tbh I don't like using the web page because it's so limited

hard to do any serious database management

it's handy when I'm on my phone but that's about it

External connections to Postgres are working.

my dev one works fine, but prod doesn't

I double checked the password and port on the proxy and everything. it was working before today's mess

Prod works fine with regard to both private and public connections; we have verified that.

as I said, one works, the other fails

Make sure that your username credential is correct, from what I remember Railway uses the railway username.

they use postgres for the username actually

I double checked all the credentials. which didn't change since yesterday, when everything was fine

my suspicion is that some networking stuff broke badly during the incident earlier

as a separate example, for some reason I can't write to Qdrant.io but I can read. bizarre

Any connection issues would be on your end. We have verified that the database is accessible via both private and public connections.

the proxy domain might be broken idk

I guess I can try generating a new one

There could be a firewall on your end blocking the current TCP port that is in use. Generating a new TCP proxy will get you a different port.

tried a new domain and same issue

again, this was working before the incident. I had zero problems. so it's not my firewall

Again, we have verified that the database is accessible via both private and public connections.

if it was my firewall I would have trouble accessing both dev and prod. and since nothing changed in my firewall between yesterday and now, it is not the issue

basic process of elimination

I'm sorry, but I don't know what to tell you at this point. The database is accessible to the public internet without issue.

how exactly are you verifying that?

I am using the TCP proxy feature

We have internal tools that I couldn't disclose, but I can show you Telnet being able to communicate.

Is that another database? As it differs from her screenshot (the proxy domain)

telnet doesn't tell me anything really. a real test would be to connect from outside using the same approach (via the postgres connection)

*her screenshot

and I changed it already as a debugging step

sorry!

it's the correct proxy

I am most certainly outside. I am not anywhere near a Railway data center.

When using a database client, I get a "wrong password" error instead of a connection failure like you're experiencing.

still fails on the new proxy

I have pasted the password so many times

Also, I can see that DBeaver is showing some information about the database (like versions) so I'm guessing the issue here are credentials. I also believe that you're copying the right password.

maybe I should regen the password?

That would be the best option, but I'm unsure if changing the environment variable would make a difference. You would need to SSH into the container and change it manually.

We have a way to regen the password in our UI now.

it didn't work lol

I tried regen and it didn't make a new password

so not even the UI for the password regen is working

it worked earlier when I generated a new dev password

Do you get any kind of errors when trying it?

nope, just doesn't actually update it

Maybe the password regeneration feature uses the current credentials to reset it?

oh actually it popped up an error that said failed to fetch

this time it didn't say "failed to fetch"

the issue with even regenerating the password may be a hint about what's happening under the hood

I was an idiot and accidentally leaked my dev password on Github today and had no issue making a new password for my dev DB via the same UI

I tried again and same issue

I'm trying to use this in my development environment and I either get refused connections or timeouts (the former for the internal networking, the latter for the public URL). not very promising, unless I'm doing something dumb

trying the TCP proxy now I guess

proxy works, but it's dumb that I have to use that for internal services talking to each other

I'll have to check the status of my app when I have a chance but it would be helpful to know if anyone is looking into my issues or if I have to pull teeth and do most of the debugging myself

hmm, I was able to regen the password and can connect locally again. thanks to whoever fixed it

having issues with creating collections on Railway Qdrant now

ugh

if it's not one thing it's another

nvm it's listing collections that keeps timing out

either way

and deleting apparently

I'm gonna just give up on Qdrant tbh. too much hassle

at least my Postgres is working. I can switch to pgvector and call it a day

ok new issue - how do I rotate credentials with pgvector?

that particular image doesn't give me access to the database tab like a regular Postgres instance

that's expected as it isn't a database template made by Railway

how do I do it then?

that popup is wrong because there is no such tab

I can't confirm it right now but can't you install the pgvector extensions directly onto the official template?

I tried

it's not a thing

I wouldn't have made a new db if I could have just intalled that

you would need to do it via SQL then, that modal is detecting it as an official template when it's not

yeah that's what I ended up doing

this is a usability problem though

that modal is not applicable to this template

don't you love it when you're vibe coding and the AI leaks your password 🙃 happened twice now

a #🤗｜feedback thread about it is more than welcome :)

can't you tell your vibe coding tool to ignore .env files?

.env is already ignored

it committed it to a todo list file 🤦‍♀️

the credentials are rotated now. but man that was annoying

done: #pgvector template doesn't allow UI-based password rotation

alright I'm good to close this thread

got my stuff fixed after a very stressful couple of days

!s