Redeployment of minio service is failing. New minio service setup works (tested in staging project)
Anonymous
FREEOP

a month ago

My current minio service volume usage is around 80%. Could that be a reason for the failing redeployments?

also came across the change in stance of open-source by minio. I don't think that should be an issue as I am able to setup a new minio service

Solved

87 Replies

brody
EMPLOYEE

a month ago

Please provide a link to the deployment that is failing


thealish
PRO

a month ago

Could you please share the full logs?



Anonymous
FREEOP

a month ago

@Brody @Alisher please help


thealish
PRO

a month ago

This is your private deployment, I cant view it


Anonymous
FREEOP

a month ago

yes, what can I share?


Anonymous
FREEOP

a month ago

1459686134065139700
1459686134438429000


Anonymous
FREEOP

a month ago

1459686299899269000


Anonymous
FREEOP

a month ago

@Alisher does these help?


Anonymous
FREEOP

a month ago

@Alisher can you allocate more volume storage? I am approaching near to 250GB already on this minio service


Anonymous
FREEOP

a month ago

Also the serivce restart command gives a success feedback, but not able to see the reset in memory/cpu usage like it used to until a couple of days back


Anonymous
FREEOP

a month ago

could it be like the running service is not able to stop to yield to the new containers? 🤔


Anonymous
FREEOP

a month ago

this is a production blocker - would appreciate your quick action!



Anonymous
FREEOP

a month ago

I usually do a hourly restart of the minio bucket service to bring down the memory usage and thus my cost. but due to this issue, I am not able to bring down my memory usage and it is resulting into huge cost for me!


Anonymous
FREEOP

a month ago

@pierre can you help here?


Anonymous
FREEOP

a month ago

@angelo ?
Tagging online employees as this is resulting in unwanted cost for us!


You need the volume resized?


Above 250GB?


Anonymous
FREEOP

a month ago

Yes, there are two issues. Refer to the support threads shared above


Anonymous
FREEOP

a month ago

My service is facing some issue - not able to restart or redeploy either


Railway
BOT

a month ago

Hello!

We're acknowledging your issue and attaching a ticket to this thread.

We don't have an ETA for it, but, our engineering team will take a look and you will be updated as we update the ticket.

Please reply to this thread if you have any questions!


Anonymous
FREEOP

a month ago

@angelo why are my two threads closed? I don't see the issues resolved right?


Anonymous
FREEOP

a month ago

@Brody ?


brody
EMPLOYEE

a month ago

Please stop tagging team members - #🛂|readme #5

They were closed as they were duplicates.


Anonymous
FREEOP

a month ago

Sure. Will not tag further


Anonymous
FREEOP

a month ago

Can you please help me with my issues?


brody
EMPLOYEE

a month ago

Is your MinIO bucket currently offline?


Anonymous
FREEOP

a month ago

No, it is running currently


Anonymous
FREEOP

a month ago

From last 2 weeks


Anonymous
FREEOP

a month ago

But using a lot of memory. I usually have a cron based restart every 1hr


Anonymous
FREEOP

a month ago

Any update?


Anonymous
FREEOP

a month ago

Waiting on this!


Anonymous
FREEOP

a month ago

Hey, I am not sure how to proceed here, even the threads are closed on the support platform


Anonymous
FREEOP

a month ago

Hey @Brody can you please suggest how to proceed here - you have closed both the threads and nobody is replying to this discord thread too! I am totally blocked - Monday would result in peak traffic in production!


Anonymous
FREEOP

a month ago

hey, duplicates of which issue?


Anonymous
FREEOP

a month ago

how do I pinpoint if this is any issue on my end?


Anonymous
FREEOP

a month ago

Hey @Brody


brody
EMPLOYEE

a month ago

Please stop tagging team members.


Anonymous
FREEOP

a month ago

sure, how to proceed here?


brody
EMPLOYEE

a month ago

Your MinIO is online from my understanding, so I don't believe the urgency is super high here.


Anonymous
FREEOP

a month ago

but the cost is up and atleast extend the volume limit above 250GB



Anonymous
FREEOP

a month ago

Can the volume be extended now?


brody
EMPLOYEE

a month ago


Railway
BOT

a month ago

🛠️ The ticket Unable to deploy has been marked as in progress.


Anonymous
PRO

a month ago

what does this mean? marking it as in-progress?


Anonymous
FREEOP

a month ago

can someone help extend the 250GB volume limit?


Anonymous
FREEOP

a month ago

@Brody I tried two things

  1. Tried to create a duplicate of the Bucket service with a new volume - it works

  2. Tried to disconnect the existing Bucket service volume and attach it to the new Bucket Copy service - it gets stuck in the same deployment step Creating containers


Anonymous
FREEOP

a month ago

does this mean there is some issue with the volume? Can you please extend the limit of my volume?


brody
EMPLOYEE

a month ago

Please stop pinging team members. Your MinIO service is online, and this thread is within its SLA.


Anonymous
FREEOP

a month ago

How can I get my volume extended without your support?


brody
EMPLOYEE

a month ago

We will extend the volume's space when we are able to come back to this thread.


Anonymous
FREEOP

a month ago

what is the SLA for my thread?


Anonymous
FREEOP

a month ago

I am on the PRO plan and it says Priority Support is available


Anonymous
PRO

a month ago

Replying from the account associated with the plan - please prioritise this thread!


brody
EMPLOYEE

a month ago

The volume has been resized to 500GB.


Anonymous
FREEOP

a month ago

Thanks. Would I now be able to initiate redeployment of the service successfully?


brody
EMPLOYEE

a month ago

I would recommend doing restarts instead of redeploys, since a caveat with volumes is that no two deployments can be online at the same time, so you would experience more downtime with redeploys than you would with instant restarts.


Anonymous
PRO

a month ago

I generally do restarts every 1 hour to keep the memory usage in check. But redeploy when required


brody
EMPLOYEE

a month ago

When is a redeploy required? A restart would restart Minio and then free the memory it used.


Anonymous
PRO

a month ago

but restarts are also not happening right now. That is why I was trying redeploy


Anonymous
PRO

a month ago

even now, restart is not working - I don't see the memory dropping


Anonymous
PRO

a month ago

can you check?
like I mentioned earlier. I noticed that service deployments get stuck when I attach this particular volume to them.


Anonymous
PRO

a month ago

what can be done here? This is costing me like atleast 5x of the actual requirement


Anonymous
PRO

a month ago

are you still available? checking on this?


Anonymous
PRO

a month ago

should I open another thread for this?


brody
EMPLOYEE

a month ago

Please avoid creating duplicate threads.


Anonymous
PRO

a month ago

understood, what should be my next step to resolve the service restarts?


Anonymous
PRO

25 days ago

bumping this up for reply


Anonymous
PRO

25 days ago

bumping this - as the cost is not under control!!


Anonymous
PRO

25 days ago

I just noticed the service was redeployed automatically (atleast I did not initiate it) and was successful, bringing the memory usage down.


Anonymous
PRO

25 days ago

was there any issue identified or action taken from the Railway team?


brody
EMPLOYEE

25 days ago

Should be fixed now


Anonymous
PRO

25 days ago

Thanks for the update. May I know the issue identified? Also, is there any way to waive of the extra cost incurred?


brody
EMPLOYEE

25 days ago

It was a state issue on our end. We have already fixed this issue going forward.


Anonymous
PRO

25 days ago

Would like to understand why was there a delay of 3days for the resolution? What is the support SLA for my account? Is this justified to charge me for your side of mistake?


brody
EMPLOYEE

25 days ago

Since there was no downtime reported, the SLA is 3 days from the last user response.


Anonymous
PRO

25 days ago

Just asking - could this have been resolved on the same day I opened thread if my thread had priority?


brody
EMPLOYEE

25 days ago

Yes, for enterprise users, the response time is 1 hour.


brody
EMPLOYEE

25 days ago

We would love to give all users the same level of support, but unfortunately, that is not feasible given the volume we operate at.


Anonymous
PRO

25 days ago

Thanks for responding.

Would it be possible to share the issue in brief? I am not sure how the cost incurred is justified for me!

Also, I would like to use Railway Storage Buckets (as they are s3 api compatible), but looking forward for the object-versioning feature as Grist service depends on that feature!


Anonymous
PRO

25 days ago

Meanwhile while Storage Buckets lack the object-versioning feature, how much can you extend my existing volume to?


brody
EMPLOYEE

25 days ago

I had shared a brief on the issue above; that would be all I am able to say at this time.

At the current spend, 500GB is the ceiling of what we can offer.


Anonymous
PRO

25 days ago

Any idea by when can I expect the object-versioning feature in Storage Buckets?

What should be the spend if I want to extend beyond 500GB?


brody
EMPLOYEE

25 days ago

With only 3 upvotes on that feature request post, we unfortunately would not be able to prioritize it anytime soon.

The spend that would come along with 80% use of a 500GB would likely be sufficient.


Status changed to Solved brody 25 days ago


Railway
BOT

23 days ago

✅ The ticket Unable to deploy has been marked as completed.


Loading...