Redeployment of minio service is failing. New minio service setup works (tested in staging project)
Anonymous
FREEOP

2 months ago

My current minio service volume usage is around 80%. Could that be a reason for the failing redeployments?

also came across the change in stance of open-source by minio. I don't think that should be an issue as I am able to setup a new minio service

Solved

87 Replies

2 months ago

Please provide a link to the deployment that is failing


thealish
PRO

2 months ago

Could you please share the full logs?



Anonymous
FREEOP

2 months ago

@Brody @Alisher please help


thealish
PRO

2 months ago

This is your private deployment, I cant view it


Anonymous
FREEOP

2 months ago

yes, what can I share?


Anonymous
FREEOP

2 months ago

1459686134065139700
1459686134438429000


Anonymous
FREEOP

2 months ago

1459686299899269000


Anonymous
FREEOP

2 months ago

@Alisher does these help?


Anonymous
FREEOP

2 months ago

@Alisher can you allocate more volume storage? I am approaching near to 250GB already on this minio service


Anonymous
FREEOP

2 months ago

Also the serivce restart command gives a success feedback, but not able to see the reset in memory/cpu usage like it used to until a couple of days back


Anonymous
FREEOP

2 months ago

could it be like the running service is not able to stop to yield to the new containers? 🤔


Anonymous
FREEOP

2 months ago

this is a production blocker - would appreciate your quick action!



Anonymous
FREEOP

2 months ago

I usually do a hourly restart of the minio bucket service to bring down the memory usage and thus my cost. but due to this issue, I am not able to bring down my memory usage and it is resulting into huge cost for me!


Anonymous
FREEOP

2 months ago

@pierre can you help here?


Anonymous
FREEOP

2 months ago

@angelo ?
Tagging online employees as this is resulting in unwanted cost for us!


You need the volume resized?


Above 250GB?


Anonymous
FREEOP

2 months ago

Yes, there are two issues. Refer to the support threads shared above


Anonymous
FREEOP

2 months ago

My service is facing some issue - not able to restart or redeploy either


Railway
BOT

2 months ago

Hello!

We're acknowledging your issue and attaching a ticket to this thread.

We don't have an ETA for it, but, our engineering team will take a look and you will be updated as we update the ticket.

Please reply to this thread if you have any questions!


Anonymous
FREEOP

2 months ago

@angelo why are my two threads closed? I don't see the issues resolved right?


Anonymous
FREEOP

2 months ago

@Brody ?


2 months ago

Please stop tagging team members - #🛂|readme #5

They were closed as they were duplicates.


Anonymous
FREEOP

2 months ago

Sure. Will not tag further


Anonymous
FREEOP

2 months ago

Can you please help me with my issues?


2 months ago

Is your MinIO bucket currently offline?


Anonymous
FREEOP

2 months ago

No, it is running currently


Anonymous
FREEOP

2 months ago

From last 2 weeks


Anonymous
FREEOP

2 months ago

But using a lot of memory. I usually have a cron based restart every 1hr


Anonymous
FREEOP

2 months ago

Any update?


Anonymous
FREEOP

2 months ago

Waiting on this!


Anonymous
FREEOP

2 months ago

Hey, I am not sure how to proceed here, even the threads are closed on the support platform


Anonymous
FREEOP

2 months ago

Hey @Brody can you please suggest how to proceed here - you have closed both the threads and nobody is replying to this discord thread too! I am totally blocked - Monday would result in peak traffic in production!


Anonymous
FREEOP

2 months ago

hey, duplicates of which issue?


Anonymous
FREEOP

2 months ago

how do I pinpoint if this is any issue on my end?


Anonymous
FREEOP

2 months ago

Hey @Brody


2 months ago

Please stop tagging team members.


Anonymous
FREEOP

2 months ago

sure, how to proceed here?


2 months ago

Your MinIO is online from my understanding, so I don't believe the urgency is super high here.


Anonymous
FREEOP

2 months ago

but the cost is up and atleast extend the volume limit above 250GB



Anonymous
FREEOP

2 months ago

Can the volume be extended now?


2 months ago


Railway
BOT

2 months ago

🛠️ The ticket Unable to deploy has been marked as in progress.


Anonymous
PRO

2 months ago

what does this mean? marking it as in-progress?


Anonymous
FREEOP

2 months ago

can someone help extend the 250GB volume limit?


Anonymous
FREEOP

2 months ago

@Brody I tried two things

  1. Tried to create a duplicate of the Bucket service with a new volume - it works

  2. Tried to disconnect the existing Bucket service volume and attach it to the new Bucket Copy service - it gets stuck in the same deployment step Creating containers


Anonymous
FREEOP

2 months ago

does this mean there is some issue with the volume? Can you please extend the limit of my volume?


2 months ago

Please stop pinging team members. Your MinIO service is online, and this thread is within its SLA.


Anonymous
FREEOP

2 months ago

How can I get my volume extended without your support?


2 months ago

We will extend the volume's space when we are able to come back to this thread.


Anonymous
FREEOP

2 months ago

what is the SLA for my thread?


Anonymous
FREEOP

2 months ago

I am on the PRO plan and it says Priority Support is available


Anonymous
PRO

2 months ago

Replying from the account associated with the plan - please prioritise this thread!


2 months ago

The volume has been resized to 500GB.


Anonymous
FREEOP

2 months ago

Thanks. Would I now be able to initiate redeployment of the service successfully?


2 months ago

I would recommend doing restarts instead of redeploys, since a caveat with volumes is that no two deployments can be online at the same time, so you would experience more downtime with redeploys than you would with instant restarts.


Anonymous
PRO

2 months ago

I generally do restarts every 1 hour to keep the memory usage in check. But redeploy when required


2 months ago

When is a redeploy required? A restart would restart Minio and then free the memory it used.


Anonymous
PRO

2 months ago

but restarts are also not happening right now. That is why I was trying redeploy


Anonymous
PRO

2 months ago

even now, restart is not working - I don't see the memory dropping


Anonymous
PRO

2 months ago

can you check?
like I mentioned earlier. I noticed that service deployments get stuck when I attach this particular volume to them.


Anonymous
PRO

2 months ago

what can be done here? This is costing me like atleast 5x of the actual requirement


Anonymous
PRO

2 months ago

are you still available? checking on this?


Anonymous
PRO

2 months ago

should I open another thread for this?


2 months ago

Please avoid creating duplicate threads.


Anonymous
PRO

2 months ago

understood, what should be my next step to resolve the service restarts?


Anonymous
PRO

2 months ago

bumping this up for reply


Anonymous
PRO

2 months ago

bumping this - as the cost is not under control!!


Anonymous
PRO

2 months ago

I just noticed the service was redeployed automatically (atleast I did not initiate it) and was successful, bringing the memory usage down.


Anonymous
PRO

2 months ago

was there any issue identified or action taken from the Railway team?


2 months ago

Should be fixed now


Anonymous
PRO

2 months ago

Thanks for the update. May I know the issue identified? Also, is there any way to waive of the extra cost incurred?


2 months ago

It was a state issue on our end. We have already fixed this issue going forward.


Anonymous
PRO

2 months ago

Would like to understand why was there a delay of 3days for the resolution? What is the support SLA for my account? Is this justified to charge me for your side of mistake?


2 months ago

Since there was no downtime reported, the SLA is 3 days from the last user response.


Anonymous
PRO

2 months ago

Just asking - could this have been resolved on the same day I opened thread if my thread had priority?


2 months ago

Yes, for enterprise users, the response time is 1 hour.


2 months ago

We would love to give all users the same level of support, but unfortunately, that is not feasible given the volume we operate at.


Anonymous
PRO

2 months ago

Thanks for responding.

Would it be possible to share the issue in brief? I am not sure how the cost incurred is justified for me!

Also, I would like to use Railway Storage Buckets (as they are s3 api compatible), but looking forward for the object-versioning feature as Grist service depends on that feature!


Anonymous
PRO

2 months ago

Meanwhile while Storage Buckets lack the object-versioning feature, how much can you extend my existing volume to?


2 months ago

I had shared a brief on the issue above; that would be all I am able to say at this time.

At the current spend, 500GB is the ceiling of what we can offer.


Anonymous
PRO

2 months ago

Any idea by when can I expect the object-versioning feature in Storage Buckets?

What should be the spend if I want to extend beyond 500GB?


2 months ago

With only 3 upvotes on that feature request post, we unfortunately would not be able to prioritize it anytime soon.

The spend that would come along with 80% use of a 500GB would likely be sufficient.


Status changed to Solved brody 2 months ago


Railway
BOT

2 months ago

✅ The ticket Unable to deploy has been marked as completed.


Loading...