How to set up Multi-Region Replica with MongoDB Atlas
Anonymous
PROOP

4 months ago

I'm trying to set up a multi-region replica with MongoDB Atlas using multi-region electable nodes.

I'm using Golang with MongoDB.

I am trying to use Railway with 3 replicas:

  1. California

  2. Netherlands

  3. Singapore

With MongoDB Atlas on GCP in the same locations as above, I expected that each region would read from its nearest database instance.

However, the issue is that on the Singapore Railway server, most of the time it reads from the MongoDB California server.
This makes my test API, which has x10 read queries, much slower — from 300ms to 3s.

My question is: Do I need to do any additional setup to make this work, or are there some service limitations I should be aware of?

func mapRegionToTags(railwayRegion string) []tag.Set {
    if strings.HasPrefix(railwayRegion, "asia-southeast1") {
		return []tag.Set{
			{{Name: "region", Value: "SOUTHEASTERN_ASIA_PACIFIC"}}, // 1st
			{{Name: "region", Value: "EUROPE_WEST_4"}},             // 2nd
			{},                                                     // 3rd (nearest)
		}
    }

    if strings.HasPrefix(railwayRegion, "europe-west4") {
		return []tag.Set{
			{{Name: "region", Value: "EUROPE_WEST_4"}},
			{{Name: "region", Value: "SOUTHEASTERN_ASIA_PACIFIC"}},
			{},
		}
    }

    if strings.HasPrefix(railwayRegion, "us-west") { 
		return []tag.Set{
			{{Name: "region", Value: "US_WEST_2"}},
			{{Name: "region", Value: "EUROPE_WEST_4"}},
			{},
		}
    }

    // Fallback
    return []tag.Set{{}}
}

func SetUp() {
	//TODO: may have to find tune the keep alive, timeout duration to make the best one
	dialer := &net.Dialer{
		KeepAlive: 300 * time.Second,
	}
	
	rawRegion := os.Getenv("RAILWAY_REPLICA_REGION")
    tagSets := mapRegionToTags(rawRegion)
	fmt.Println("Region map: ", rawRegion, tagSets)

	rp, err := readpref.New(
        readpref.NearestMode, 
        readpref.WithTagSets(tagSets...),
    )
    if err != nil {
        // If we can't create the preference, we shouldn't start the server.
        log.Fatalf("Failed to create MongoDB ReadPreference: %v", err)
    }

	wc := writeconcern.New(writeconcern.W(1))
	clientOptions := options.Client().ApplyURI(os.Getenv("DATABASE_URL")).SetMinPoolSize(5).SetDialer(dialer).SetWriteConcern(wc).SetReadPreference(rp)
	client, err := mongo.Connect(context.TODO(), clientOptions)
	if err != nil {
		log.Fatal(err)
	}
	Client = client
}
$40 Bounty

8 Replies

Railway
BOT

4 months ago

Hey there! We've found the following might help you get unblocked faster:

If you find the answer from one of these, please let us know by solving the thread!

brody
EMPLOYEE

4 months ago

This thread has been marked as public for community involvement, as it does not contain any sensitive or personal information. Any further activity in this thread will be visible to everyone.

Status changed to Open brody • 4 months ago

dev
MODERATOR

4 months ago

If I understand your setup correctly, you have 3 MongoDB Atlas regions and you have 3 Railway regions and you want each railway replica to the appropriate mongo replica but the issue is that the singapore railway replica most often uses the california mongo replica, is my understanding correct? If so, does that behaviour happen with your current code?

My current thinking is you may have maxStalenessSeconds configured somewhere which might result in the first 2 regions being avoided if its lagging behind the primary which is especially possible if the cali region holds your primary replica

yeeet
PRO

4 months ago

what it seems like is the singapore process is hitting the fallback branch, and the nearest ends up selecting California. you can wind up debugging it by logigng the rawRegion and tagSets, and hardcode a singapore preference to confirm routing.

could you also check your DATABASE_URL? a read preference in the URL will override the Go option and can get it to read from the primary in California

dev

If I understand your setup correctly, you have 3 MongoDB Atlas regions and you have 3 Railway regions and you want each railway replica to the appropriate mongo replica but the issue is that the singapore railway replica most often uses the california mongo replica, is my understanding correct? If so, does that behaviour happen with your current code?My current thinking is you may have maxStalenessSeconds configured somewhere which might result in the first 2 regions being avoided if its lagging behind the primary which is especially possible if the cali region holds your primary replica

Anonymous
PROOP

4 months ago

FYI:
in mongoDB atlas I used
M10 with Multi-Cloud, Multi-Region

The primary node is California > Netherlands > Singapore

Yes, from my inspection singapore railway replica most often uses the california mongo replica.

But I'm also newbie in the horizontal scalling as well,

yeeet

what it seems like is the singapore process is hitting the fallback branch, and the nearest ends up selecting California. you can wind up debugging it by logigng the rawRegion and tagSets, and hardcode a singapore preference to confirm routing.could you also check your DATABASE_URL? a read preference in the URL will override the Go option and can get it to read from the primary in California

Anonymous
PROOP

4 months ago

in my DATABASE_URL it have
/?retryWrites=true&w=majority
here

But how to hardcode a singapore preference to confirm routing ?

in my DATABASE_URL it have /?retryWrites=true&w=majorityhere But how to hardcode a singapore preference to confirm routing ?

yeeet
PRO

4 months ago

I'd use this for testing purposes, but it would look something like this:

rawRegion := os.Getenv("RAILWAY_REPLICA_REGION")
log.Printf("Hardcoding Singapore read preference, RAILWAY_REPLICA_REGION=%s", rawRegion)

rp, err := readpref.New(
    readpref.NearestMode,
    readpref.WithTagSets(
        tag.Set{{Name: "region", Value: "SOUTHEASTERN_ASIA_PACIFIC"}},
    ),
)
if err != nil {
    log.Fatalf("Singapore read preference error: %v", err)
}

clientOptions := options.Client().
    ApplyURI(os.Getenv("DATABASE_URL")).
    SetReadPreference(rp)
ilyassbreth
FREE

2 months ago

the behavior you’re seeing is expected with your current setup. nearest does not mean “same region”, it means “lowest ping among eligible nodes”. since your primary is in california, it’s often within the latency window and gets picked, especially if the singapore secondary is slightly lagging

key points to fix / verify:

  • atlas already auto-tags nodes with region, you don’t need custom tags

  • your code is fine only ifRAILWAY_REPLICA_REGION really matches asia-southeast1, otherwise you silently hit the {} fallback and nearest = california

  • log rawRegion + tagSets on startup to confirm you’re not falling back

  • temporarily hardcode {region:SOUTHEASTERN_ASIA_PACIFIC} to confirm routing (good test)

  • make sure you are not setting maxStalenessSeconds anywhere — that will disqualify the singapore secondary and force reads to california

  • if you want to avoid the primary completely, use secondaryPreferred instead of nearest

dev

If I understand your setup correctly, you have 3 MongoDB Atlas regions and you have 3 Railway regions and you want each railway replica to the appropriate mongo replica but the issue is that the singapore railway replica most often uses the california mongo replica, is my understanding correct? If so, does that behaviour happen with your current code?My current thinking is you may have maxStalenessSeconds configured somewhere which might result in the first 2 regions being avoided if its lagging behind the primary which is especially possible if the cali region holds your primary replica

ilyassbreth
FREE

2 months ago

no railway limitation here , this is mongo read preference behavior + latency + possible staleness filtering

Loading...