Postgres DB Crashed Due to Storage
Anonymous
TRIALOP

2 years ago

Hello!

I'm using a js package called ponder (https://ponder.sh/) to index the ethereum blockchain. My deployment crashed from what looks like a lack of storage. When I look at the memory chart of my postgres database on railway though, I had around 3.5GB of memory remaining when it stopped.

Is there another reason why this could have broken? I'm not sure what final storage size will end up being, but I expect it should fall under 8GB. Any help is appreciated!

Project ID: 72e571de-3aaf-4521-bfcb-d36891096f9c

27 Replies

brody
EMPLOYEE

2 years ago

You would be looking at your memory, postgres does not store persistent data in ram.

Hobby users get 5gb disks and that would dictate how much you are able to store in the database, you can check how much disk space postgres has used by clicking the volume that is attached to the bottom of the database service within your project canvas.

If you need more than 5gb you would need to upgrade to Pro for 50gb volumes, then you can move the project over to your Pro workspace and grow the volume to 50gb from within the volumes settings.


Anonymous
TRIALOP

2 years ago

Understood - looks like it failed at around 4.5GB, so that would make sense

1227639046499602400


brody
EMPLOYEE

2 years ago

You would still be looking at the memory, postgres does not store it's persistent data in memory.

In my message above I included instructions on how you would go about viewing the disk usage.


Anonymous
TRIALOP

2 years ago

thanks for pointing that out again


Anonymous
TRIALOP

2 years ago

1227639762639130600


Anonymous
TRIALOP

2 years ago

I see it's tapped out - is there anything I need to do after upgrading to pro to increase the storage limit, or will it increase automatically?


brody
EMPLOYEE

2 years ago

I also included instructions for that in my initial message.


Anonymous
TRIALOP

2 years ago

🤦‍♂️ sorry. Thanks for everything


brody
EMPLOYEE

2 years ago

No worries, let me know if you run into any difficulties.


Anonymous
TRIALOP

2 years ago

I have a follow up question: I moved the project over and grew the storage, but the db deployment failed again because it says 'invalid length of startup packet'

1227646289219682300


Anonymous
TRIALOP

2 years ago

I'm not very experienced with deploying services like this, so apologies for any dumb questions - I'm assuming there is some corrupted data because the db was interrupted last time.

Does this mean the data in the db is corrupted, or just some files that are needed for startup/other tasks?
If the db data is corrupted, is there a way I can easily roll back? No worries if not, I can run the indexing script again if needed


brody
EMPLOYEE

2 years ago

Can you access the database still? just wanna make sure even though I've seen these errors on my own database and have not faced any issues.


Anonymous
TRIALOP

2 years ago

no, the deployment failed during build


brody
EMPLOYEE

2 years ago

First off can you make sure you can connect to the database with a database client like dbgate?


brody
EMPLOYEE

2 years ago

And second, please send your build logs using this -


Anonymous
TRIALOP

2 years ago

Sure - I think I'm unable to connect with dbgate. I get: "Error: Connection terminated unexpectedly"


brody
EMPLOYEE

2 years ago

What is the status of your postgres deployment?


Anonymous
TRIALOP

2 years ago

failed


Anonymous
TRIALOP

2 years ago

Build logs of the most recent deployment of the postgres deployment that failed during the build

1227652779401805896


brody
EMPLOYEE

2 years ago

Did you set a healthcheck on the postgres service?


Anonymous
TRIALOP

2 years ago

I did. /health


Anonymous
TRIALOP

2 years ago

I did because the indexing package said to add a healthcheck path for their deployment. I assumed it wouldn't hurt. I can remove it for the db though


brody
EMPLOYEE

2 years ago

You would have wanted to add that healthcheck to the indexing service, not postgres.


Anonymous
TRIALOP

2 years ago

okay, understood. I set it for both. I'll remove it for postgres and try again


brody
EMPLOYEE

2 years ago

Sounds good


Anonymous
TRIALOP

2 years ago

confirming the build was successful!


brody
EMPLOYEE

2 years ago

Awesome


Loading...