1fc9ac5f-29f8-4b23-ab28-1eebdaeb8c06/volume/6d410a84-ccf6-4064-b5dc-b53bb746fc15/metrics?environmentId=87adcc32-cf45-44c4-bd23-7de68500165a

kbit/s <:patrickwow:901282270806175775>

!t

Do you recommend that we spin up a new instance a database, and do a data import? Or should we try to wait it out?

This is a substantial outage for us, we've never been down for over an hour. I need to make a decision, please advise.

Hello,

Your database is not offline from the looks of it?

It says 1%, am I not understanding this?

We're waiting for the migration to complete to bring our servers back online.

1% of the migration, your database is still online.

You do not need to stop anything that connects to it.

IS our database online in California or in Virginia? It was in california to start, but we migrated to Virginia.

Its currently online in US-West2

I don't understand how that is possible, as the migration was to virginia, as it shows in the picture (from us west to east).

Its hasn't finished the migration yet, so its still running in US-West

Are you saying that we can continue to function normally? Our cron jobs immediately started having issues, and we experienced some errors; which I assumed were related.

The database is fully online.

It never went offline.

At least for the last 21 hours (since thats when the latest deploy happened)

No, we did this 1.5 hours ago.

Yes, the database has not been offline for the last 21 hours

I believe what you are saying is that my database is still running on US West, but when the migration completes, it will automatically be booted on the East. Is this correct?

If so, in the meantime, I should keep our servers running in the west, but I will need to immediately switch them to the east when the database migration completes, as they cannot be that far apart. Does that sound correct?

Correct to everything but -

they cannot be that far apart

They can be that far apart, it will work, but you will see additional latency.

I realize it will work from a technical perspective, but everything will break due to the slowdown. 90% of the functionality of our platform is handled by cron jobs which will be fired off faster than they complete.

Then you will see a temporary increase in latency.

Obviously, the kbps speed is an issue, I'm just trying to help you get yourself unblocked in the meantime.

Yes, thank you Brody. I appreciate your guidance.

Ill report the issue, but yeah, please bring your services back online.

We are handling that now. When the migration completes, will we be able to know? So long as we can get it done and it doesn't happen this Saturday or Sunday, we should be okay. However, if it happens in the middle of our live events, it puts our company in a bit of danger, as we have hundreds of people at these events depending on our service. Going down for a few minutes would be ok, but if we were to go down for a few hours, it would be disasterous for us. I will hope for the best.

I will ask platform for next steps here, we may have you stop the migration all together and restart it.

Thank you.

@Jason Layton - We stopped the migration, please start a new migration again.

Before I start the migration, can you give me an estimate of how long you think it will take? I'm trying to avoid an outage occuring tomorrow or Sunday. If you think it would complete tonight, I'm happy to do it, if not I would prefer to wait until Monday. Please advise.

It will give you an accurate ETA when you start it, +250 hours was wrong.

Ok, I will try to do it as soon as we are back on our feet. We are still experiencing outages in other areas, I'm putting out fires right now. Thank you.

No problem!

Any chance you can give me some guidance on the problem, my ai assistant thinks it's related to the migration.

We were using a link with a JWT embeded in the URL (I know this isn't great practice, but fixing it hasn't been a priority).

https://wrestlingtournaments.com/dualsWrestlerRegistrationMaster/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9/eyJ0ZWFtSWQiOjgxNCwiZHVhbERpdmlzaW9uSWQiOjEwLCJpYXQiOjE3NTEwNjAwNDUsImV4cCI6MTc1OTcwMDA0NX0/f2uJJr-ZI1p5EPVyzRiTl6Fj097t8vuwISmFdJq7Gzc

The page is blank, and the console shows:
Failed to load module script: Expected a JavaScript-or-Wasm module script but the server responded with a MIME type of "text/html". Strict MIME type checking is enforced for module scripts per HTML spec.

there is a lot of white, but how blank should the page be?

I think I fixed it by forcing a redirect to www on cloudflare. I'm not sure where the issue came from, but the subdomain seems to fix the issue.

Sounds good, but it is safe to say it wasnt due to the migration, your database's availability and operation did not change

Well, we moved the front end servers as well. It was working until we took the site down to do maintenance; once it came back online, anyone who had tried to load the site couldn't load it at all, unless they did a hard refresh.

Once they did a hard refresh, it would work; but not that link. That link required the subdomain.

Glad you got it solved!

Computers are hard. Thank you Brody. I'm a bit terrified to try the migration right now, we're in too precarious of a position; but I will do it first thing Monday morning. Thanks again for your help.

Remember, the database will not go offline until the migration starts its last step.

Sunday is the debut of our new product which will have about 700 people on location at the event (for hopefully only a few hours, assuming we don't go down at all [the product is definitely not as stable as I would like it to be]); that alone is enough stress. We're stable now (I think), so I'm going to spend the rest of my effort preparing for the event. Thank you.

No problem, good luck with the event!

@Jason Layton How did it go?

If you don't mind me asking

Any issue with your migration?

I just checked, done 3 days ago without issues, completed in 12 minutes

!s

Btw, the bot didn't mark it as solved, you should prob try it again

!s

I just checked the help station and it seems like that this thread has been already marked as solved.

I'll go ahead and lock this thread.