Project ID is 750a6ff7-b90f-4885-88b9-e69f7ba105be

Hmmm, I wonder if this is now something you could technically change yourself using SSH with the Railway CLI.

@Crafter could you try to SSH into your service and run cat /var/lib/postgresql/data/pgdata/[postgresql.auto](postgresql.auto).conf and see if timescaledb,pg_stat_statements is listed there?

I am also going to escalate this so that it doesn't happen with other users.

!t

Unable to SSH

the container crashes entirely

Another note, I did change the image from the old railway timescale image that was formerly the default to the new one to see if that fixed it but it did not

The boot loop happened as soon as I confirmed that the container would be restarted

One way to SSH into the container would be to attach the volume to another running service that doesn't have a volume and SSH into the container this way. (You might as well just use the file browser template at this point.)

Oh I see the issue, really sorry that happened, please try to remove the inner quotes and it should work again. I will fix it

🫡

I asked in the PB thread, but how did you get timescale in the binary and preloaded in the db if you use ghcr.io/railwayapp-templates/postgres-ssl:15 as the source?

That should not be possible

Or at least not without some source trickery and a very old service

This is quite an old service

https://discord.com/channels/713503345364697088/1467207292100415771/1467295626763960463

The former default was the timescale image

But did you change the source?

That's really risky tbh

Ahh

Yep

Honestly, surprised it worked

I think I deployed this in 2023?

It's been on Railway for quite a while. I've kind of been in limbo in terms of updating the database it because it's so damn old.

Looks like I have to revert back now as it can't find Timescale

Yeah, the new source doesnt have it

What was the old source?

lemme find it

It's on the Railway app templates Github

⁨ghcr.io/railwayapp-templates/timescale-postgis-ssl:pg15-ts2.12⁩

Cause timescale shouldnt have the Data UI working, at least new deployments dont have (although I would love to support, its not something we test)

interesting

I don't actually use any of the Timescale features

If that matters

so you can just remove the timescale from preloaded libraries tbh

actually dont do that

im not sure how timescale affects the data folder, so let's try to decrease entropy and return to the original state

https://github.com/railwayapp-templates/timescale-postgis-ssl

Uhhhh

the UI won't let me change the image

I press enter and it doesn't update

I'm stupid, nevermind

erm

Any clue on how to fix that 😅

i think you mounted the wrong path

but honestly unsure

⁨/var/lib/postgresql/data⁩

<:hmm:1228339204048814080>

It said it was used 6 months ago

¯_(ツ)_/¯

I will note, this only happened after changing the image back

Sorry I'm not sure what is going on here, I don't have a lot of details on how the timescale template worked, so I may have to leave it for oncall

You could try removing timescale from preloaded, but honestly please take a backup before doing any risky operation on your volume

Should I just download the entire data directory? I'm not on Pro so I don't have backups

Changing the source has a high risk of corrupting the data

I could also just upgrade tbf

Is the "create backup" button under the database service sufficient?

yes! that said it wont auto recover service config

so that backup will work on the specific image source + volume mount path it was made to work on, if you further change things, the backup is only guaranteed to work if you revert

ah okay

Alrighty I removed Timescale from the shared libraries and switched to the newer image. I had to purge the certs directory as it was causing permission issues, Postgres regenerated them.

!s

oh is that not a thing anymore

P.S does this make me eligible for Bug Basher 🥺

Ahhh makes sense that the certs were conflicting between the two sources

Great to know it's fixed, I've updated the logic to prevent that

And Brody gave you the role!