Replica set: data lost

harundarat
PRO

3 months ago

Hello! yesterday I was late paying my bill so my services were paused. After I completed the payment, I redeployed the paused services to be active again. But I ran into a problem when redeploying my mongo replica set. When the service was active and I accessed it from datagrip, the database that I created before was lost. I have tried restore backup, but the result is the same. when I look at the log, there's error: "mongo: Aggregate command executor error"

{ "severity": "info", "timestamp": "2025-04-17T11:27:03.004129345Z", "message": "Aggregate command executor error", "tags": { "deploymentId": "e805da81-f078-4d3f-8e8f-dd3771f404c5", "deploymentInstanceId": "34b3e400-e97a-478a-9dc8-ad521d6c801d", "environmentId": "4f49ac12-c31f-4434-bc3a-d28f67540bbc", "pluginId": null, "projectId": "b869082f-795e-4354-aefc-1a74fa52d5a8", "serviceId": "18f8d146-9aa4-487d-bee1-df8529ca0974", "snapshotId": null }, "attributes": { "attr": { "cmd": { "$db": "local", "aggregate": "oplog.rs", "cursor": {}, "pipeline": [ { "$collStats": { "storageStats": { "numericOnly": true, "waitForLock": false } } } ] }, "error": { "code": 26, "codeName": "NamespaceNotFound", "errmsg": "Unable to retrieve storageStats in $collStats stage :: caused by :: Collection [local.oplog.rs] not found." }, "stats": {} }, "c": "QUERY", "ctx": "ftdc", "id": 23799, "level": "info", "s": "W", "t": { "$date": "2025-04-17T11:27:03.001+00:00" } } }

please help me. Btw, I use Mongodb replica set template from railway. Thank you!

Solved

8 Replies

3 months ago

How long was it until you resolved your bill?


Status changed to Awaiting User Response railway[bot] 3 months ago


angelo

How long was it until you resolved your bill?

harundarat
PRO

2 months ago

I think it's about 1 day or less


Status changed to Awaiting Railway Response railway[bot] 3 months ago


2 months ago

That shouldn't have caused data loss. I wouldn't know what steps you took to restore so I don't know if it was error on your part that caused it.


Status changed to Awaiting User Response railway[bot] 2 months ago


angelo

That shouldn't have caused data loss. I wouldn't know what steps you took to restore so I don't know if it was error on your part that caused it.

harundarat
PRO

2 months ago

I mean, maybe the data is not lost. but it's not visible when I access from datagrip/compass. In the screenshot, there should be one more database with the name 'ispyn', but it's not there.

Attachments


Status changed to Awaiting Railway Response railway[bot] 2 months ago


angelo

That shouldn't have caused data loss. I wouldn't know what steps you took to restore so I don't know if it was error on your part that caused it.

harundarat
PRO

2 months ago

I do restore from 'backups' tab

Attachments


2 months ago

Yea, but you were running a replica set, so the set went out of sync... that corrupts data.


Status changed to Awaiting User Response railway[bot] 2 months ago


angelo

Yea, but you were running a replica set, so the set went out of sync... that corrupts data.

harundarat
PRO

2 months ago

Understood.. actually, I also have csv backup locally, although it's slightly outdated. Seems like there's no other choice. Thanks Angelo!


Status changed to Awaiting Railway Response railway[bot] 2 months ago


echohack
EMPLOYEE

2 months ago

Leaving this here for the future. If you haven't thought about it already, run a mongodump and test with a mongorestore

So you can avoid issues like this in the future (especially if you don't have an easy solution like this time). And if you really want to get fancy, you can set up a cron on Railway to do this periodically.

Here's some info about that process: https://www.mongodb.com/docs/manual/tutorial/backup-and-restore-tools/


Status changed to Awaiting User Response railway[bot] 2 months ago


Status changed to Solved echohack 2 months ago