How to migrate data volume from gcp?
krupskis
HOBBYOP

a year ago

Hi, I want to migrate weaviate data volume from gcp. Any resources?

28 Replies

krupskis
HOBBYOP

a year ago

34be2320-1881-4630-b704-7afdf6e17feb


hi @Martynas


I have you on deck from Slack on your migration


  1. Get your schema
    curl -X GET "https://your-source-instance/v1/schema" \
    -H "Content-Type: application/json" > schema.json

  2. Start a backup
    https://weaviate.io/developers/weaviate/configuration/backups

  3. Restore a backup assuming your name and number of nodes between source and target are identical.


krupskis
HOBBYOP

a year ago

hi @angelo how would we store the backup file on a railway weaviate instance?


You would have the backup file on your machine and then use the API to send it to your target instance


krupskis
HOBBYOP

a year ago

I see - would you think nodes refer to kubernetes nodes?


Nodes are a generic concept in the eyes of Weaviate, so if you are running a non-HA instance, you can assume you are running 1


krupskis
HOBBYOP

a year ago

we have 3 instances on gcp


krupskis
HOBBYOP

a year ago

and as far as I can see we can't scale replicas on volume services on railway


krupskis
HOBBYOP

a year ago

we have a concern for migration, as we are scalling we might need replicas for weaviate


Yep- I will need some time to see how we can mirror this


(almost 2 AM my time, but I can get to this first thing)


!remind me to figure out replicated weaviate in 8 hours


krupskis
HOBBYOP

a year ago

we have a concern for migration, as we are scalling we might need replicas for weaviate
This is a bigger concern for us right now, I bet we can do migration anyway.


my gut feeling is that we can do multiple instances and set up a reverse proxy


but we would need to sync those instances


krupskis
HOBBYOP

a year ago

What's the biggest weaviate cluster your customers have right now?


I am not sure, will need to test and read up on it


krupskis
HOBBYOP

a year ago

or if you had solved issues with replicated volumes for any customer


krupskis
HOBBYOP

a year ago

for any db service


We just ask users to set up follower DBs


and then pool connections


krupskis
HOBBYOP

a year ago

I think weaviate has leaderless architecture


we'd still need a way to tell weaviate to coordinate


krupskis
HOBBYOP

a year ago

If we end up with scalling issues, would railway in any way support us to scale weaviate?



We'd just give your instance generous limits


Loading...