Replica set: data lost

harundaratPRO

a month ago

Hello! yesterday I was late paying my bill so my services were paused. After I completed the payment, I redeployed the paused services to be active again. But I ran into a problem when redeploying my mongo replica set. When the service was active and I accessed it from datagrip, the database that I created before was lost. I have tried restore backup, but the result is the same. when I look at the log, there's error: "mongo: Aggregate command executor error"

{ "severity": "info", "timestamp": "2025-04-17T11:27:03.004129345Z", "message": "Aggregate command executor error", "tags": { "deploymentId": "e805da81-f078-4d3f-8e8f-dd3771f404c5", "deploymentInstanceId": "34b3e400-e97a-478a-9dc8-ad521d6c801d", "environmentId": "4f49ac12-c31f-4434-bc3a-d28f67540bbc", "pluginId": null, "projectId": "b869082f-795e-4354-aefc-1a74fa52d5a8", "serviceId": "18f8d146-9aa4-487d-bee1-df8529ca0974", "snapshotId": null }, "attributes": { "attr": { "cmd": { "$db": "local", "aggregate": "oplog.rs", "cursor": {}, "pipeline": [ { "$collStats": { "storageStats": { "numericOnly": true, "waitForLock": false } } } ] }, "error": { "code": 26, "codeName": "NamespaceNotFound", "errmsg": "Unable to retrieve storageStats in $collStats stage :: caused by :: Collection [local.oplog.rs] not found." }, "stats": {} }, "c": "QUERY", "ctx": "ftdc", "id": 23799, "level": "info", "s": "W", "t": { "$date": "2025-04-17T11:27:03.001+00:00" } } }

please help me. Btw, I use Mongodb replica set template from railway. Thank you!

Solved

8 Replies

a month ago

How long was it until you resolved your bill?


Status changed to Awaiting User Response railway[bot] 28 days ago


angelo

How long was it until you resolved your bill?

harundaratPRO

a month ago

I think it's about 1 day or less


Status changed to Awaiting Railway Response railway[bot] 28 days ago


a month ago

That shouldn't have caused data loss. I wouldn't know what steps you took to restore so I don't know if it was error on your part that caused it.


Status changed to Awaiting User Response railway[bot] 27 days ago


angelo

That shouldn't have caused data loss. I wouldn't know what steps you took to restore so I don't know if it was error on your part that caused it.

harundaratPRO

a month ago

I mean, maybe the data is not lost. but it's not visible when I access from datagrip/compass. In the screenshot, there should be one more database with the name 'ispyn', but it's not there.

Attachments


Status changed to Awaiting Railway Response railway[bot] 26 days ago


angelo

That shouldn't have caused data loss. I wouldn't know what steps you took to restore so I don't know if it was error on your part that caused it.

harundaratPRO

a month ago

I do restore from 'backups' tab

Attachments


a month ago

Yea, but you were running a replica set, so the set went out of sync... that corrupts data.


Status changed to Awaiting User Response railway[bot] 26 days ago


angelo

Yea, but you were running a replica set, so the set went out of sync... that corrupts data.

harundaratPRO

24 days ago

Understood.. actually, I also have csv backup locally, although it's slightly outdated. Seems like there's no other choice. Thanks Angelo!


Status changed to Awaiting Railway Response railway[bot] 24 days ago


echohackEMPLOYEE

23 days ago

Leaving this here for the future. If you haven't thought about it already, run a mongodump and test with a mongorestore

So you can avoid issues like this in the future (especially if you don't have an easy solution like this time). And if you really want to get fancy, you can set up a cron on Railway to do this periodically.

Here's some info about that process: https://www.mongodb.com/docs/manual/tutorial/backup-and-restore-tools/


Status changed to Awaiting User Response railway[bot] 23 days ago


Status changed to Solved echohack 23 days ago


Replica set: data lost - Railway Help Station