is your postgres database in the same project as your django service?

Hi Brody, thanks for answering. I think that's maybe what I'm trying to set up. I haven't hosted my Django project anywhere, it's just on my local machine. I am hoping to have it be hosted by Railway and use the Postgres DB, if that makes sense. I thought I would connect my local django project to the Railway Postgres DB first and then continue on. Does that make sense? Maybe I'm not expressing myself well of I have a misunderstanding in how this should work.

yeah it makes sense, you only have the database on railway right now

show me the database thing in your settings.py please

Ah, I see…I think I can see how I can make a Django deployment that includes an already linked Postgres database now…

In any case, this is my database setup in settings.py:

DATABASES = {
"default": {
"ENGINE": "django.db.backends.postgresql_psycopg2",
"NAME": os.environ.get("PGDATABASE"),
"USER": os.environ.get("PGUSER"),
"PASSWORD": os.environ.get("PGPASSWORD"),
"HOST": os.environ.get("PGHOST"),
"PORT": os.environ.get("PORT"),
}
}

And in my .env file, I have the actual values, copied over from the Variables section of the database

Thank you so much for your help, by the way! I really appreciate it

lets try to do this in a way that doesnt involve putting the login to your database in a plaintext file

in the same project as your postgres database, add an empty service

OK, thank you. I added it

screenshot please

give it a better name haha

ok, I just went for the default

Will this empty service host the environment files? Or would it be the django project? What makes sense as a name?

this empty service for now will hold the variables, and it will also eventually be where you deploy your app to

so lets see a new screenshot after the name change

Ok, I see

Here you go

for the sake of simplicity, just name it "bibblio"
no need for complex names here

Got it

open up the bibblio service variables and its raw editor, paste this in -

PGDATABASE=${{Postgres.PGDATABASE}}
PGUSER=${{Postgres.PGUSER}}
PGPASSWORD=${{Postgres.PGPASSWORD}}
PGHOST=${{Postgres.PGHOST}}
PGPORT=${{Postgres.PGPORT}}

save and deploy

Got it, I was reading about shared variables, this is setting the variables from the Postgres DB to the local environment of what will be the django project, right?

It saved and deployed correctly

click their eye icons, do they all show the variables correctly?

yup!

is there any other variables your app needs? if so, go ahead and add them there

the end goal of this is so that you are able to delete your local .env file

I see. Yes, I have a few others, a Secret Key and a variable for whether the Django project is in Debug mode or not. I'll add them

ah you have debug set by a variable, okay you can keep the local .env file with only that debug variable set inside of it

in the service variables youd want to have that debug variable set so your app runs in production mode when on railway

Ok, yes. So in my local env, I'll set DEBUG to True but when I add it to the service variable I set it to false.

correct

so show me your local .env file now, it should only contain the debug line

OK, I added and deployed the changes and here's the .env file. Crucially, the DJANGO_DEBUG on the service is False

Ah sorry for the bad screenshot of the env file

looks good to me

do you have the railway cli installed?

Yes, I just installed it

And I logged in and all

then in your project directory, run railway link and link to the bibblio service

done

do you have migrations ready to apply?

Yes, I believe so

okay, run railway run python [manage.py](manage.py) migrate

this runs the given command locally, but with the linked service variables available, much better than keeping the database credentials in plaintext

Got it. That makes sense. It's running

Or at least, I haven't gotten a completion nor error message

awsome

Thank you so much again, by the way. I've really been struggling with trying to deploy this app for months (on and off) and it's been frustrating. I really appreciate you helping me out

you sure waited a long time to ask for help haha

Haha, yea, I think my mistake was jumping from one deployment environment to another (Google Cloud, Vercel, Azure) and not finding ways to ask for helpthere

yeah i dont think any of those services have a good community

Yea, it was a lot of forum hunting and being frustrated, even trying to reach out to people who wrote tutorials on LinkedIn to no avail

but anyway, I unfortunately have to run to a meeting for my day job soon (within 10 minutes)

Do you think I might be able to ask for some help later? Or generally, what next steps do i take? Should I simply run railway up hopefully once the migrations run?

do you have your code on github?

Yea, I do here: https://github.com/dalbanhi/bibbl.io

Do you think I might be able to ask for some help later?
of course!

then youd want to go the github deployment route and not railway up, we can get that hooked up when you are free again

Ok, sounds good! Thanks so much, Brody!

Another quick question: why does the migration take so long? Is it simply because it's running this on the Railway server?

as mentioned, its not running this on the railway server, its ran locally

Ah ok. I think I'm understanding a bit better. However, I'm a little concerned as the migrations never ended/failed

screenshot please

So this is the terminal where I ran the railway command (first screenshot). However, the second screenshot is where I was running my project locally ( I forgot to kill it beforehand). I got the attached error. I'm thinking I should stop my local server and then re-try the railway command?

yes

Ok

I'm running it again. Approximately how long do you usually take? The migrations for my project usually take very little time when I ran it locally before so I'm a little confused as to why it's taking significantly longer

nah theres for sure something wrong here

what verison of the cli are you on

railwayapp 3.5.2

just for fun, what happens if you run python [manage.py](manage.py) migrate

Ah. There, I get this error:

okay thats normal

what do you get if you run railway --help

it seems to be working well

Hey Brody, again, I really appreciate your help. I had an idea since I'm not really sure how to proceed from here. Would it make sense to try to set up a django with postgres template and then try to add my github repo to the django service? Or would that possibly interfere with what we're trying to do here

nah something is going on here with your project and the cli, issue isnt with the railway side of things

what does railway variables print

It seems to print out my variables

well dont send that here, this is public

Oh, sorry--yea I should have realized

check the database on railway, does it have tables?

Yes it does

im kinda stumped right now

Hmm, ok, no worries Brody. I appreciate you trying to help me out anyway

Do you think it would at all make sense to delete the services on my project and re-try it? I'm really not sure what else I could do

no, theres no issues with railway here

OK. Well, I think I will try to uninstall the CLI since you mentioned the issue seems to be there

and re-install it

Actually, one question. I should be connecting to the 'bibblio' service, right? not the "Postgres" service?

correct

Should I have a Procfile or anything in my root directory?

yes, but that has nothing to do with the cli

Hm ok, yea. Thanks for your help! I'll reach out again if some of my tinkering does anything

try railway shell and then run the migration command (without railway run)

When I run railway shell, it seems to immediately boot me off of it

or like, I'm not able to type in a command before getting back to my regular command line

you are in the shell

what terminal is this?

Its zsh

mac?

yea

okay so what happens if you run the migration command in the railway shell

I see… I didn't understand that I was in the shell. I thought it would be more like when you run a python repl or like when connecting to postgres with psql. I ran the command in the railway shell and it's still taking a long time, just like with the railway run option

something funky is happening

can you temporarily remove your local .env file

ok

I removed it and ran it again with railway shell . It seems to still be stuck

this is super odd

okay desperate attempt, copy the raw variables from railway into a local .env file. then run the migration command without railway run or railway shell

ok

So, wait, you mean to copy the actual values of the Postgres service, not the related variables like ${{Postgres.PGDATABASE}}, right?

yeah you'd need to replace the references with the values

Yea, it still didn't work. This is what my .env file looks like and the migrations are still hanging

you aren't in a railway shell right?

no, I don't think so-- when I ran railway shell it removed my virtual environment (venv), so I'm pretty sur e I'm in my regular shell

okay so then there is something wrong with your project

or your network

go download dbgate and use it to try to connect to the database

Hmm… ok. I have another meeting soon that I have to run to. So I'll try to do that and get back to you

Brody, I really deeply do appreciate your help

It's possible that its my network-- I'm a teacher so I work at a school and there might be some network restrictions

oh yes from personal experience, schools do have very restrictive firewalls

Ok. I'll try this back home and see if it works on my home network. If not, I'll try to work through DB Gate. I do have a bit of a busy schedule the next few days due to the holiday, but if I encounter any more issues, I'll be sure to ping you

thanks so much again!

happy to help, and yep feel free to ping

Hey Brody! Guess what? Trying it out at home not on the school's firewall got me different (and fast) results! I tried to run both the migration command with/without railway run and in both cases, I got the same error (see attached).

I ran into something similar when switching databases from local sqlite to postgres. What I did there is that I dumped my database into a json (it wasn't very large), then deleted the migrations, and then ran my migrations, and then finally reloaded my data using loaddata in django.
I feel like that might work, but something in me thinks that there might be something obvious I am missing that would not involve me doing this, but somehow 'syncing' or merging my databases instead.

What do you think, do you have any insights onto how to solve this new issue? This SO post (https://stackoverflow.com/questions/44651760/django-db-migrations-exceptions-inconsistentmigrationhistory) gave me some ideas last time but I'm wary of dropping my database since I know I wouldn't actually be able to do that with a team/larger database.

do you have any data in the database? like besides the tables existing?

I do have some data, like 'testing' data from when I was testing out my app. I wouldn't be sad if I had to drop those tables completely, but generally, I guess I've never really gone from dev to production and so I'm not really sure what the process for your data is supposed to go. If it were possible to keep that testing data, that would be nice but I'm not like wedded to it or anything.

Also, Brody, thanks so much for your help. Do you have something like a Ko-Fi and/or like Venmo or something? You've really helped me out a bunch and honestly just made me feel like this was possible. I'd love to like buy you a cup of coffee in gratitude or something

I'm assuming your data in the database got messed up somehow, I don't know your project so I wouldn't really know on how to advice fixing it without wiping the database and re-running the migration again.

i do have a bmac in my bio, but please don't feel any need to send me anything

got it, yea, I don't feel the need to, I just appreciate it! So, just as a follow up question, does just wiping my database mean deleting my migrations folder and trying to run teh migrations again?

wiping your database would mean you need to go into the volume's settings and wipe it, then running the migrations again with railway run

Hey Brody, thanks for getting back to me. I'm not sure what you exactly mean by 'wipe the volume's settings'. Would that be the Postgres database I just made on Railway? If I go into my Railway project, I have some tables there (see attached) but I don't actually have like any tables that correspond to any data that I might have created, like by creating users or anything. So I'm a little confused as to what/why I would wipe this volume

so to recap you where having issues with the database and i recommended wiping it so that you can start with a fresh migration

Yes, exactly

sorry, I should have mentioned that

so is that something you are going to do or are you going to try to resolve the errors manually?

I'm going to try to do that because I have a back up of my data/this is a testing project. I want to learn how to deploy this project and I'm not too fussed about the data

okay then go ahead and wipe the database's volume

So, what do you mean by that? Do I go into the Railway project and just delete all of these tables or is that something I need to do on my like local environment. I don't know if that makes sense

you simply press the wipe volume button in the databases's volume settings

OK. I'm having some trouble finding that but I will look for it. But afterwards, I should be able to run railway run python [manage.py](manage.py) migrate and then the same with runserver and if I have no errors, I can push to my github repo, then connect my repo to the service and it should work?

click on the volume of the postgres service

and yes you would want to run railway run python [manage.py](manage.py) migrate and then you can redeploy the railway service

OK. I think I did both of those things. In order to redeploy the Railway service, you mean the Postgres database, right?

Hey Brody, so I'm able to create a new user with my Postgres database service from my local machine, which is exciting! However, I then tried to deploy by connecting to my repo and it said my deploy finished. In my deploy logs for that service, all I see is this (attached), which doesn't seem good. Do you have any tips as to what to explore? I'm not entirely sure why python would be a command that is not found on Railway's python

please send your build logs

Ok. I think it's possible my issue is this: https://www.answeroverflow.com/m/1113611990238240778 (someone very knowledgeable giving some good tips there 😉 ), so I'm going to try what you are suggesting there.

I too have a package.json too because I guess I tried to make a hybrid react/django app and I'm using webpack to bundle my react files into a index-bundle.js files. In the future, I'm probably sticking with more of a cleanly split react front end / django cors framework app, but I'm digressing. Below is my build log:

Pushing [==================================================>] 94.63MB 3bea1f1ca704
Pushing [==================================================>] 96.77MB 3bea1f1ca704
Pushing [==================================================>] 98.95MB 3bea1f1ca704
Pushing [==================================================>] 101.1MB 3bea1f1ca704
Pushing [==================================================>] 103.2MB 3bea1f1ca704
Pushing [==================================================>] 105.9MB 3bea1f1ca704
Pushing [==================================================>] 106.5MB 3bea1f1ca704
Pushed 3bea1f1ca704
750024d5-965f-44e2-9fd6-d06bda6f036d: digest: sha256:9efe2f346744dc0f2552ce1a4c817d49b2c63d92a3f9f902a8ede9444b948049 size: 2421

Publish time: 9.84 seconds

(or at least the ending of it)

Hmm, so I tried this railway.json file (the one you suggested) and my build failed this time

{
    "$schema": "https://railway.app/railway.schema.json",
    "build": {
      "builder": "NIXPACKS",
      "nixpacksPlan": {
        "providers": ["python"],
        "phases": {
            "setup": {
                "nixPkgs": ["...", "nodejs"]
            }
        }
    },
    "buildCommand": "npm ci"
    },
    "deploy": {
        "startCommand": "python manage.py migrate && python manage.py collectstatic --noinput && gunicorn webapp.wsgi",
      "restartPolicyType": "ON_FAILURE",
      "restartPolicyMaxRetries": 10
    }
}

It doesn't seem to have found a version of Django on my requirements.txt but I just did pip freeze > requirements.txt to save my package requirements (this is my build log):

6.026 ERROR: No matching distribution found for Django==5.0.3
6.251
6.251 [notice] A new release of pip is available: 23.0.1 -> 24.0
6.251 [notice] To update, run: pip install --upgrade pip
-----

Dockerfile:20
-------------------
18 |     ENV NIXPACKS_PATH /opt/venv/bin:$NIXPACKS_PATH
19 |     COPY . /app/.
20 | >>> RUN --mount=type=cache,id=s/bf770e00-35cd-438c-88ec-c9385e1bc086-/root/cache/pip,target=/root/.cache/pip python -m venv --copies /opt/venv && . /opt/venv/bin/activate && pip install -r requirements.txt
21 |
22 |     # build phase
-------------------
ERROR: failed to solve: process "/bin/bash -ol pipefail -c python -m venv --copies /opt/venv && . /opt/venv/bin/activate && pip install -r requirements.txt" did not complete successfully: exit code: 1

Error: Docker build failed

Any help or insights at all you can point me to are super appreciated

full build logs please

im pretty sure i have sent you this, but you havent been using it so ill send it again -

Ah, no I dont think I'd seen that. A useful tool

1224877984272351264

what version of python do you use locally?

Python 3.11.8

then you would want to add a runtime.txt to your repo with just 3.11 in it

hm ok. Gotcha, thanks for the tip -- I'll let you know how it goes!

So the deployment worked but I'm getting a TCP/IP connection error (log attached). I'm trying to debug the issue and any help would be appreciated!

1224881113202425986

you need to use PGPORT here

omg, wow, I don't know how I would have caught that. Thanks for the second set of eyes

you arent the first to make that mistake

haha thanks

I'll let you know how it goes, but I'm about to have dinner. Thank you sooo much again for all of your help

It's truly amazing

happy to help

Thanks so much! After fixing some more typos, I'm getting no build nor deployment errors (I think… is it normal for a deploy log to look like this?):

1224896661092962406

yeah that looks good to me

Awesome! Yea, I think I'm not serving my static files for my django project correctly, but I think that's more of a django thing than a railway thing. I really appreciate all of the help, again!

yeah read up on the whitenoise docs

will do! Have a good night!

you too!

Hey Brody, I hope you're doing well. I had a quick question I was hoping you might be able to point me in the right direction of. I'm having some csrf troubles on my django app still (it's showing up now but I can't post anything). So, I'm trying to write to my own debugging logging file. I'm able to write to my own file on my local machine, but whenever I'm not sure where that logging file would exist on the Railway server. Would you happen to know anything about that? Any help/insight would be greatly appreciated

you would want to log to stdout/stderr not a file