It seems that restarting several times solved the problem, but another one appeared..
One of my servers started returning this error: ReplyError: MISCONF Errors writing to the AOF file: Read-only file system

Hey, can you point me to the correct service? thats not a project id, it's a service id and i cant see anything wrong with it?

Oh sorry! Project ID: e60124ce-5bb4-41cb-8517-9606dbe5b9f7

Service: Guardian

what env?

production

looking

the only errors im seeing from Guardian are from july 18th

you mentioned mongo, the guardain service is not a mongo database, im a little confused here

if you could send a deep link to the service in question that would be great

The MongoDB server that had gone down was able to get back up and running
Guardian is running, however, it is giving an error to all users, and in the logs the error is related to disk space

https://railway.app/project/e60124ce-5bb4-41cb-8517-9606dbe5b9f7/service/84702838-10fb-4f5c-b66f-2646a3de3f4a?id=16c56465-59e6-4bc1-baea-1d3665573d83

Search for: ReplyError: MISCONF Errors writing to the AOF file: Read-only file system

please link me to the mongo database in question

the database is working fine now, link: https://railway.app/project/e60124ce-5bb4-41cb-8517-9606dbe5b9f7/service/5269510c-7793-4bb9-a4fe-9d71f7a39225?id=db90b316-1910-4972-8cd1-0d6ae62c8e8a

i'm sorry but im not seeing any errors here

i'm getting errors in another service now

https://railway.app/project/e60124ce-5bb4-41cb-8517-9606dbe5b9f7/service/29206e59-4524-44ce-8d45-9c014d5b192b?id=6858ff19-1e6d-4223-ac82-e8a55b519a9b

this looks like someone is spamming your server with an invalid request

is it a DDoS attack?

unfortunately i have no way of knowing that, you have the custom domain setup through Cloudflare so this is something you would want to check on over there

you can ignore the /socket.io requests, they're wrong requests that we're fixing

Oh, i found the problem

i restarted the guardian redis server and it's working now

Thank you @Brody, we are online again

so for clarity, nothing wrong with mongo?

it just crashed on its own without anyone touching it.. it was strange

but for now is all okay, my heart is just racing 😂

you are the second person to raise this issue about redis, ill ask around about it

oh i appreciate it

The more robust the railway becomes, the better for everyone

ill keep you in the loop if i get any answers

perfect!! thank you so much Brody 🚀

no problem!

hey, unfortunately we where not able to find any correlations to any issues, but please let me know if you see this again and we can have another look.

aadditionally, if you dont need AOF you can outright disable it, is that something you would be interested in?