How to migrate data volume from gcp?

krupskis
HOBBY

4 months ago

Hi, I want to migrate weaviate data volume from gcp. Any resources?

0 Replies

krupskis
HOBBY

4 months ago

34be2320-1881-4630-b704-7afdf6e17feb


4 months ago

hi @Martynas


4 months ago

I have you on deck from Slack on your migration


4 months ago

  1. Get your schema
    curl -X GET "https://your-source-instance/v1/schema" \
    -H "Content-Type: application/json" > schema.json

  2. Start a backup
    https://weaviate.io/developers/weaviate/configuration/backups

  3. Restore a backup assuming your name and number of nodes between source and target are identical.


krupskis
HOBBY

4 months ago

hi @angelo how would we store the backup file on a railway weaviate instance?


4 months ago

You would have the backup file on your machine and then use the API to send it to your target instance


krupskis
HOBBY

4 months ago

I see - would you think nodes refer to kubernetes nodes?


4 months ago

Nodes are a generic concept in the eyes of Weaviate, so if you are running a non-HA instance, you can assume you are running 1


krupskis
HOBBY

4 months ago

we have 3 instances on gcp


krupskis
HOBBY

4 months ago

and as far as I can see we can't scale replicas on volume services on railway


krupskis
HOBBY

4 months ago

we have a concern for migration, as we are scalling we might need replicas for weaviate


4 months ago

Yep- I will need some time to see how we can mirror this


4 months ago

(almost 2 AM my time, but I can get to this first thing)


4 months ago

!remind me to figure out replicated weaviate in 8 hours


krupskis
HOBBY

4 months ago

we have a concern for migration, as we are scalling we might need replicas for weaviate
This is a bigger concern for us right now, I bet we can do migration anyway.


4 months ago

my gut feeling is that we can do multiple instances and set up a reverse proxy


4 months ago

but we would need to sync those instances


krupskis
HOBBY

4 months ago

What's the biggest weaviate cluster your customers have right now?


4 months ago

I am not sure, will need to test and read up on it


krupskis
HOBBY

4 months ago

or if you had solved issues with replicated volumes for any customer


krupskis
HOBBY

4 months ago

for any db service


4 months ago

We just ask users to set up follower DBs


4 months ago

and then pool connections


krupskis
HOBBY

4 months ago

I think weaviate has leaderless architecture


4 months ago

we'd still need a way to tell weaviate to coordinate


krupskis
HOBBY

4 months ago

If we end up with scalling issues, would railway in any way support us to scale weaviate?


4 months ago

Yes


4 months ago

We'd just give your instance generous limits


How to migrate data volume from gcp? - Railway Help Station