Postgres DB Crashed Due to Storage

AnonymousTRIAL

a year ago

Hello!

I'm using a js package called ponder (https://ponder.sh/) to index the ethereum blockchain. My deployment crashed from what looks like a lack of storage. When I look at the memory chart of my postgres database on railway though, I had around 3.5GB of memory remaining when it stopped.

Is there another reason why this could have broken? I'm not sure what final storage size will end up being, but I expect it should fall under 8GB. Any help is appreciated!

Project ID: 72e571de-3aaf-4521-bfcb-d36891096f9c

0 Replies

a year ago

You would be looking at your memory, postgres does not store persistent data in ram.

Hobby users get 5gb disks and that would dictate how much you are able to store in the database, you can check how much disk space postgres has used by clicking the volume that is attached to the bottom of the database service within your project canvas.

If you need more than 5gb you would need to upgrade to Pro for 50gb volumes, then you can move the project over to your Pro workspace and grow the volume to 50gb from within the volumes settings.


AnonymousTRIAL

a year ago

Understood - looks like it failed at around 4.5GB, so that would make sense

1227639046499602400


a year ago

You would still be looking at the memory, postgres does not store it's persistent data in memory.

In my message above I included instructions on how you would go about viewing the disk usage.


AnonymousTRIAL

a year ago

thanks for pointing that out again


AnonymousTRIAL

a year ago

1227639762639130600


AnonymousTRIAL

a year ago

I see it's tapped out - is there anything I need to do after upgrading to pro to increase the storage limit, or will it increase automatically?


a year ago

I also included instructions for that in my initial message.


AnonymousTRIAL

a year ago

🤦‍♂️ sorry. Thanks for everything


a year ago

No worries, let me know if you run into any difficulties.


AnonymousTRIAL

a year ago

I have a follow up question: I moved the project over and grew the storage, but the db deployment failed again because it says 'invalid length of startup packet'

1227646289219682300


AnonymousTRIAL

a year ago

I'm not very experienced with deploying services like this, so apologies for any dumb questions - I'm assuming there is some corrupted data because the db was interrupted last time.

Does this mean the data in the db is corrupted, or just some files that are needed for startup/other tasks?
If the db data is corrupted, is there a way I can easily roll back? No worries if not, I can run the indexing script again if needed


a year ago

Can you access the database still? just wanna make sure even though I've seen these errors on my own database and have not faced any issues.


AnonymousTRIAL

a year ago

no, the deployment failed during build


a year ago

First off can you make sure you can connect to the database with a database client like dbgate?


a year ago

And second, please send your build logs using this -


AnonymousTRIAL

a year ago

Sure - I think I'm unable to connect with dbgate. I get: "Error: Connection terminated unexpectedly"


a year ago

What is the status of your postgres deployment?


AnonymousTRIAL

a year ago

failed


AnonymousTRIAL

a year ago

Build logs of the most recent deployment of the postgres deployment that failed during the build

1227652779401805896


a year ago

Did you set a healthcheck on the postgres service?


AnonymousTRIAL

a year ago

I did. /health


AnonymousTRIAL

a year ago

I did because the indexing package said to add a healthcheck path for their deployment. I assumed it wouldn't hurt. I can remove it for the db though


a year ago

You would have wanted to add that healthcheck to the indexing service, not postgres.


AnonymousTRIAL

a year ago

okay, understood. I set it for both. I'll remove it for postgres and try again


a year ago

Sounds good


AnonymousTRIAL

a year ago

confirming the build was successful!


a year ago

Awesome


Postgres DB Crashed Due to Storage - Railway Help Station