How to migrate data volume from gcp?
krupskis
HOBBYOP

a year ago

Hi, I want to migrate weaviate data volume from gcp. Any resources?

28 Replies

krupskis
HOBBYOP

a year ago

34be2320-1881-4630-b704-7afdf6e17feb


hi @Martynas


I have you on deck from Slack on your migration


  1. Get your schema

    curl -X GET "https://your-source-instance/v1/schema"

    -H "Content-Type: application/json" > schema.json

  2. Start a backup

    https://weaviate.io/developers/weaviate/configuration/backups

  3. Restore a backup assuming your name and number of nodes between source and target are identical.


krupskis
HOBBYOP

a year ago

hi @angelo how would we store the backup file on a railway weaviate instance?


You would have the backup file on your machine and then use the API to send it to your target instance


krupskis
HOBBYOP

a year ago

I see - would you think nodes refer to kubernetes nodes?


Nodes are a generic concept in the eyes of Weaviate, so if you are running a non-HA instance, you can assume you are running 1


krupskis
HOBBYOP

a year ago

we have 3 instances on gcp


krupskis
HOBBYOP

a year ago

and as far as I can see we can't scale replicas on volume services on railway


krupskis
HOBBYOP

a year ago

we have a concern for migration, as we are scalling we might need replicas for weaviate


Yep- I will need some time to see how we can mirror this


(almost 2 AM my time, but I can get to this first thing)


!remind me to figure out replicated weaviate in 8 hours


krupskis
HOBBYOP

a year ago

we have a concern for migration, as we are scalling we might need replicas for weaviate

This is a bigger concern for us right now, I bet we can do migration anyway.


my gut feeling is that we can do multiple instances and set up a reverse proxy


but we would need to sync those instances


krupskis
HOBBYOP

a year ago

What's the biggest weaviate cluster your customers have right now?


I am not sure, will need to test and read up on it


krupskis
HOBBYOP

a year ago

or if you had solved issues with replicated volumes for any customer


krupskis
HOBBYOP

a year ago

for any db service


We just ask users to set up follower DBs


and then pool connections


krupskis
HOBBYOP

a year ago

I think weaviate has leaderless architecture


we'd still need a way to tell weaviate to coordinate


krupskis
HOBBYOP

a year ago

If we end up with scalling issues, would railway in any way support us to scale weaviate?



We'd just give your instance generous limits


Welcome!

Sign in to your Railway account to join the conversation.

Loading...