Stale Data from Replicas
need-help-scaling
PROOP

a month ago

Hey, running into a weird issue with our staging service and hoping you can help figure out what's going on.

We have multiple replicas running and they're all connected to the same external MySQL database (verified the IP). Here's what's happening:

When we update the database directly, the API keeps returning old data mixed with new data. Like, the database has 1 row, but the API returns 2 rows - one old, one new.

The weird part is that restarting the Railway service immediately fixes it and everything returns the correct data. Our code doesn't have any caching implemented, so I'm wondering if there's something at the Railway infrastructure level that could be caching database connections or responses?

We've verified:

- All replicas are connecting to the same database IP

- The database itself has the correct data

- Our code doesn't cache anything

- Restarting fixes it instantly

Is there some kind of connection pooling or caching happening on Railway's side that could cause this? And if so, is there a way to configure it so replicas always pull fresh data without needing manual restarts?

Solved$20 Bounty

5 Replies


brody
EMPLOYEE

a month ago

Hello,

We do not do any caching on our side at any point in any network path.

I will open this thread up to the community so they can help you debug this.

Best,

Brody


Status changed to Awaiting User Response Railway about 1 month ago


brody
EMPLOYEE

a month ago

This thread has been marked as public for community involvement, as it does not contain any sensitive or personal information. Any further activity in this thread will be visible to everyone.

Status changed to Open brody about 1 month ago


need-help-scaling
PROOP

a month ago

Confirmed this behavior is due to railway replicas. Scaling down to single machine completely solved this issue. I know while railway does not cache anything internally, could it be due to the default configs of AWS (or whatever cloud providers Railways is using)? @brody


brody
EMPLOYEE

a month ago

We use our own bare metal hardware, this would be your application doing something non-standard.

I'll leave this thread to the community so they can help you!


need-help-scaling
PROOP

a month ago

Looks like this is caused by bad SQL config in our backend. Thanks for the help Brody!


Status changed to Solved samgordon 28 days ago


Loading...