Production Postgres down after redeploy stuck on TimeZone: UTC error
eyenalxai
PROOP

25 days ago

We hit a strange failure in production: new connections to Postgres fail with invalid value for parameter "TimeZone": "UTC". I tried redeploying Postgres to fix it, but the redeploy is stuck. Prod is down

27 Replies

eyenalxai
PROOP

25 days ago

Stuck at:

Database Connection

Attempting to connect to the database...

eyenalxai
PROOP

25 days ago

Both build and deploy logs are empty <:mildpanic:804271964587819059>


eyenalxai
PROOP

25 days ago

What will happen if I abort


eyenalxai
PROOP

25 days ago

Can't upgrade either


janheussner
PRO

25 days ago

I have the same issue "Level: Error - invalid value for parameter "TimeZone": "UTC".

Also the connection fails with Session Error. How to proceed?


janheussner

I have the same issue "Level: Error - invalid value for parameter "TimeZone": "UTC". Also the connection fails with Session Error. How to proceed?

eyenalxai
PROOP

25 days ago

Does it fail only on complex queries for you? Parts of our app are still working, which is weird.


eyenalxai

Does it fail only on complex queries for you? Parts of our app are still working, which is weird.

janheussner
PRO

25 days ago

I am not sure. For me it seems like thats not only the case. Sometimes it happens for the same query and sometimes not. Are you also using Django?


janheussner

I am not sure. For me it seems like thats not only the case. Sometimes it happens for the same query and sometimes not. Are you also using Django?

eyenalxai
PROOP

25 days ago

Nope, typescript with drizzle orm.


eyenalxai

Nope, typescript with drizzle orm.

janheussner
PRO

25 days ago

Means we know its only related to postgres itself.


eyenalxai
PROOP

25 days ago

select count(*) from pg_timezone_names;

could not stat file "/usr/share/zoneinfo/Australia/Currie": Input/output error


janheussner

Means we know its only related to postgres itself.

eyenalxai
PROOP

25 days ago

Just to confirm, you encountered this error around the time I reported it here, right?


eyenalxai

Just to confirm, you encountered this error around the time I reported it here, right?

janheussner
PRO

25 days ago

Yes right. First time 18:46 was the first time. I can see that in my Sentry Logs and also in my Monitoring a little bit earlier:

Attachments


janheussner
PRO

25 days ago

The strange part is that i have round about 25 instances of the same application but only one is failing.


eyenalxai
PROOP

25 days ago

I don't even see a working Postgres deploy now, even though some parts of the app are still working.

Attachments


eyenalxai

I don't even see a working Postgres deploy now, even though some parts of the app are still working. ![](https://station-server.railway.com/attachments/att_01kq80y0nheh4ak8ss4twdg77b)

janheussner
PRO

25 days ago

Thats why i was wating for someone from the support. I was also trying to create a new backup but with all the regular backups it is already at the limit of backups (10) which doesn't make much sense for me. So because of that and because i cannot create a ssh connection i would be happy if someone from the support could look at that. (Service ID: 2eb353b4-8abe-4044-86ca-fda27279ef3a)


eyenalxai
PROOP

25 days ago

Fixed on my end by adding the max_parallel_workers_per_gather=0 option.

I also want to say that 2 hours without a response is crazy work. Something simple like 'looking into this' would have been much appreciated.

P.S. This is not a fix but a hack. Some queries take forever but at least they're finishing.


eyenalxai
PROOP

25 days ago

That still looks very scary

Attachments


eyenalxai

Fixed on my end by adding the `max_parallel_workers_per_gather=0` option. I also want to say that 2 hours without a response is crazy work. Something simple like 'looking into this' would have been much appreciated. P.S. This is not a fix but a hack. Some queries take forever but at least they're finishing.

janheussner
PRO

25 days ago

Yes you are right. I am also a little bit confused that nobody is reacting and also the backup topic makes me angry. So I have to wait till the next auto backup is done (4 hours) :(


janheussner

Yes you are right. I am also a little bit confused that nobody is reacting and also the backup topic makes me angry. So I have to wait till the next auto backup is done (4 hours) :(

eyenalxai
PROOP

25 days ago

You can delete one of your older backups to create a manual one.


eyenalxai

You can delete one of your older backups to create a manual one.

janheussner
PRO

25 days ago

Yes i know but who has designed this ? 😄


janheussner

Yes i know but who has designed this ? 😄

eyenalxai
PROOP

25 days ago

Oh yeah I agree it's miserable. Just like scrolling this thread with all the images jumping around.


eyenalxai

Oh yeah I agree it's miserable. Just like scrolling this thread with all the images jumping around.

janheussner
PRO

25 days ago

I was talking about the backup design of max 10 backups and the scheduling is already using 10. 😄 The UI here is fine for me^^


janheussner
PRO

25 days ago

@Brody sorry for mention you but we really need some help here.


eyenalxai
PROOP

25 days ago

My database stopped working altogether


janheussner
PRO

25 days ago

image.png

Attachments


eyenalxai
PROOP

25 days ago

I vibecoded a script that makes backups every 5 minutes because I knew something like this would happen. I'll try to create a new Postgres deploy and restore to it.


eyenalxai
PROOP

25 days ago

It worked thank god.

btw the only happy letter i received was about redis in our staging environment


Welcome!

Sign in to your Railway account to join the conversation.

Loading...