Unable to redeploy postgres service

jclaveau

PROOP

6 months ago

Hello,

All this occurs in the environment dev of the project 65aff0db-6586-4be0-8420-b2e67ae4378d

I use the builtin Postgres 15 service https://github.com/railwayapp-templates/timescale-postgis-ssl/blob/main/Dockerfile.pg15-ts2.12

I made a typo running ALTER SYSTEM SET shared_preload_libraries = 'timescaledb, pg_stat_statements';

As result PG wasn't able to start anymore.

I tried to start PG skipping this config by setting the custom start command su -c "/usr/lib/postgresql/15/bin/postgres -c shared_preload_libraries=''" postgres

PG started but I couldn't connect with psql.
Happily ssh connection worked so I patched the content of /var/lib/postgresql/data/pgdata/postgresql.auto.conf

I then tried to redeploy without start command and it never succeeded, even after waiting 35 minutes.
The only log I can get now being

Stopping Container
Stopping Container

Contrary to my original case where logs told me sharedpreloadlibraries was not loadable

And now, I can't ssh anymore:

Warning: Received unknown message type: ready
Error: Your application is not running or in a unexpected state.

I don't know what else to do but delete and recreate this service but i'd prefer to keep the config.

Thanks in advance

Project ID: 65aff0db-6586-4be0-8420-b2e67ae4378d
environment id: 81dbf431-5eec-4a53-b82e-10555b59e0c4 (dev)

$20 Bounty

23 Replies

medim

MODERATOR

6 months ago

Hi! Was there any data in the database?

jclaveau

PROOP

6 months ago

yes

jclaveau

PROOP

6 months ago

but it can be deleted

jclaveau

PROOP

6 months ago

But plz touch only the dev env or Ill die !

medim

MODERATOR

6 months ago

You can wipe that volume data and redeploy the service, just make sure to do a backup before that.
You will also lose the 'ALTER SYSTEM' settings that you've made

jclaveau

PROOP

6 months ago

can we have a volume for one file only like in docker ?

medim

MODERATOR

6 months ago

If I understand what you're saying… I don't think you can

jclaveau

PROOP

6 months ago

Anyway I just realized we can only add one volume per service

medim

MODERATOR

6 months ago

yes

jclaveau

PROOP

6 months ago

After wiping the volum and started a redeploy, I got an error:

First it remigrated my legacy data as when I migrated to the bare metal offer
Then it failed with the message

Pre-deploy › Migrate volumes

(03:51)

Volume migration failed.

I'm now trying deleting the volume

jclaveau

PROOP

6 months ago

Ok once the volume is deleted it starts fast BUT :

When a recreate the volume and redeploy it remigrates my old data <:mildpanic:804271964587819059>

jclaveau

PROOP

6 months ago

1417227909860430000

medim

MODERATOR

6 months ago

What do you mean?

jclaveau

PROOP

6 months ago

And finally fails again

1417228228623335400

medim

MODERATOR

6 months ago

I will escalate this to the team

jclaveau

PROOP

6 months ago

It seems to be an issue related to old migration to the bare metal offer

medim

MODERATOR

6 months ago

You delete the volume, create a new one and there's the old data in it?

jclaveau

PROOP

6 months ago

For now I need to go forward so I will simply use my service with no volume

jclaveau

PROOP

6 months ago

but I will set it back once I'm not in a hurry anymore

jclaveau

PROOP

6 months ago

Almost :

I delete the volume
redeploy works
a recreate the volume
when redeploying it remigrates 1Gb of data

6 months ago

6 months ago

This thread has been escalated to the Railway team.

Status changed to Awaiting Railway Response medim • 6 months ago

jake

EMPLOYEE

6 months ago

Looks solid now. However, did you remove the volume? There's no volume attached in dev. You should defs re-add it if you care about the data

Status changed to Awaiting User Response Railway • 6 months ago

jclaveau

PROOP

6 months ago

Effectively, I deleted the volume as the only workaround to unlock my CI process but I'd like to get it back to save money (ram vs volume for our DB of several Gb and serverless mode for PRs and dev env).

I just recreated it:

(I wonder what is the 3rd change btw)

And... oO it works !

And the DB is empty, thank you very much !

Just a question. Will I have the same issue if I must redeploy my production db or is it fixed in any env ?

And another question : I let the fresh service with no data for more than an hour and it never got to sleep mode (serverless mode is enabled obviously). Is it normal or is there any configuration I miss?

Thanks in advance

Attachments

image.png

Status changed to Awaiting Railway Response Railway • 6 months ago