10 months ago
I landed some changes which included a Dango migration, these normally run flawlessly, but today it just hangs, never finishes. I tried redeploying, also tried running the migration manually from the CLI and still hangs.
The migration ran locally without issue.
The dependency listed in the migration does exsit.
Any advice on how to troubleshoot this? Not seeing any errors either.
TIA
0 Replies
Is it possible that the migration is just something that takes a long time on live data?
I guess to that end, do you mind sending the migration you are trying to run?
It is adding a new table with no data, it should not take more than a couple of seconds. My app is pretty new, so db is not very big
This is the code, let me know if I should send it as a file?
Generated by Django 4.2.4 on 2024-07-26 10:58
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
('api', '0087_modify_planlimits_data'),
]
operations = [
migrations.CreateModel(
name='StripeReLinkActivity',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('created_at', models.DateTimeField(auto_now_add=True)),
('linked_to_stripe', models.BooleanField(default=False)),
('stripe_email', models.EmailField(blank=True, max_length=100)),
('stripe_customer_id', models.CharField(blank=True, max_length=100)),
('user', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.CASCADE, to='api.userprofile')),
],
),
]To send code, you can put it in a code block:
```py
CODE HERE
```
That would look like
import django
print("hi")(I escaped the character for you. But it is the `. The character on the ~ key.
# Generated by Django 4.2.4 on 2024-07-26 10:58
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
('api', '0087_modify_planlimits_data'),
]
operations = [
migrations.CreateModel(
name='StripeReLinkActivity',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('created_at', models.DateTimeField(auto_now_add=True)),
('linked_to_stripe', models.BooleanField(default=False)),
('stripe_email', models.EmailField(blank=True, max_length=100)),
('stripe_customer_id', models.CharField(blank=True, max_length=100)),
('user', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.CASCADE, to='api.userprofile')),
],
),
]Ok, so the next thing I wonder, is if you have any open transactions during the migration. If you have running transactions, the migration won't happen. So,,, you would have to kill the web interface to the db and then let the migrations go through. Otherwise, yea, it hangs.
But if you have a small server,,, I wonder if this is actually the issue <:shrug:747680403778699304>
How are you running the migration?
The migration first ran as part of a deployment that was kicked off my merging to master on Github. When that didn't work, I used the CLI to connect to the container and kick it off. It is still running…
Operations to perform:
Apply all migrations: admin, api, auth, authtoken, contenttypes, sessions
Running migrations:
Applying api.0088_striperelinkactivity...Let me try restarting the front end
Actually, the backend is showing as up and running, but the front end is getting a 503 error, service unavailable
No, and I rebooted the railway container but it seems to try to apply the migration and just hangs, so app is not working at all
Do you not see any other logs of what the error is? If you are able to reboot the db, you can try that. But I would first look at what the logs say about the errors
Restarted the postgres DB, that caused the dango container to crash, after it restarted, it responds, but getting these errors on every request:
Error getting new users over time: connection to server at "roundhouse.proxy.rlwy.net" (35.212.138.205), port 11654 failed: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.Ok. Well we found the cause of the issue then. So, DBs don't let you do migrations if there are open connections. I am sure there are some enterprise way of getting around that. But <:shrug:747680403778699304>
Anyways, railway had an open connection through that roundhouse endpoint. And so, you couldn't migrate. By reseting the db, you forced the connection to close. There are other ways you can force all connections to be closed. Not the point. I just wonder why this proxy is hanging like that?
Are you using TCP? (I assume since you got this error)
Backend is 2 containers, Django and Postgres, frontend is hosted on Netlify
Ya maybe that's what was going on. Well phew! Thanks for your help! Will try to see if I can check for open connections before migrating or maybe upgrade if that would solve the problem.
I am going to ping someone who might have a better idea about this than me. @Brody (tldr solved for now, but something is still up)