a year ago
Project ID: 675c7104-52e7-4685-9806-2b3cbd0a9af7
Service ID: 84dc4d1a-0cea-44e8-893c-eeacbe4342fe
My cron job writes a commit hash to a volume in order to resume work on the next run. However, on every run it executes the "empty file" branch of my logic, and prints that it's "Saving current HEAD for the first time..."
I am using a WORKRDIR environment var, so I can configure where to write the "current head" txt file. In local I want it in the project root, in the Railway deployment I want it written in the volume. As far as I can tell everything is configured correctly, the WORKDIR var matches the volume mount path. Would appreciate another set of eyes on this, thank you!
130 Replies
a year ago
Hey, the only options I can think of here are:
- Is your
WORKDIRenvironment variable actually being set? Perhaps a print/console.log could confirm that. - Your volume might be mounted at the wrong path.
Maybe you're also able to attach a file browser template to confirm that there's nothing on the volume itself.
a year ago
Also, are you using a Dockerfile?
I suppose could check if the var is actually being set, seems farfetched that it wouldn't but stranger things have happened
a year ago
Providing that script would help the community help you.
#!/bin/bash
set -eu
if [ -f "./.env" ]; then
source "./.env"
fi
if [ -z "$WORKDIR" ]; then
echo "No WORKIR found in environment. Operation cancelled."
exit 1
fi
if [ -z "$SMTP_USER" ]; then
echo "No SMTP_USER found in environment. Operation cancelled."
exit 1
fi
if [ -z "$SMTP_PASS" ]; then
echo "No SMTP_PASS found in environment. Operation cancelled."
exit 1
fi
if [ ! -d "$WORKDIR" ]; then
mkdir -p "$WORKDIR"
fi
cd "$WORKDIR"
if [ ! -f "./previoushead.txt" ]; then
touch "./previoushead.txt"
fi
previous_head=$(< "./previoushead.txt")
current_head=$(git ls-remote https://github.com/supabase/supabase.git refs/heads/master | cut -f1)
if [ -z "$previous_head" ]; then
echo "Saving current HEAD for the first time..."
printf %s "$current_head" > "./previoushead.txt"
echo "Success!"
exit 0
fi
if [ "$current_head" = "$previous_head" ]; then
echo "Nothing has changed!"
exit 0
fi
echo "Supabase remote HEAD has updated! Checking for relevant changes locally..."
if [ -d "supabase" ]; then
cd supabase
git pull
else
git clone https://github.com/supabase/supabase
cd supabase
fi
if [ -z "$(git diff --name-only "$previous_head" HEAD -- docker)" ]; then
echo "Nothing has changed in /docker!"
else
echo "New changes in /docker! Notifying via email..."
curl --url 'smtps://smtp.gmail.com:465' \
--mail-rcpt 'ben.isenstein@gmail.com' \
--mail-from "$SMTP_USER" \
--user "$SMTP_USER:$SMTP_PASS" \
-T <(echo -e "From: \"PG On Rails\" <$SMTP_USER>\nSubject: New changes in \"/supabase/docker\"\n\nhttps://github.com/supabase/supabase/tree/master/docker\n\nSent securely via curl.")
fi
echo "Saving new HEAD \"$current_head\"..."
cd ..
printf %s "$current_head" > "./previoushead.txt"
echo "Success!"
exit 0
a year ago
Also, shell scripts commonly continue execution without error even when environment variables are undefined (if not used with the correct flags).
And I just confirmed that WORKDIR and the volume mount path are both /app/volume
a year ago
Try using all absolute paths.
Some more context:
This service builds from a rootDirectory within a monorepo of cron jobs. In this directory I use a railway.json to configure the cron schedule. Now, the docs say that you can't just include a railway.json in a service's rootDirectory and expect it to work, but it's working. But could that be messing up volume behaviour? highly doubt it, but thought I would share nonetheless
Is that path not absolute to the root of the container? I tried ~/volume and that was prohibited.
a year ago
Nothing is messing up the volume behavior; the railway side of things is correct, it's the script.
a year ago
Brody said to include $WORKDIR to every path instead of using cd. Here's on how it would look like:
a year ago
#!/bin/bash
set -eu
if [ -f "./.env" ]; then
source "./.env"
fi
if [ -z "$WORKDIR" ]; then
echo "No WORKIR found in environment. Operation cancelled."
exit 1
fi
if [ -z "$SMTP_USER" ]; then
echo "No SMTP_USER found in environment. Operation cancelled."
exit 1
fi
if [ -z "$SMTP_PASS" ]; then
echo "No SMTP_PASS found in environment. Operation cancelled."
exit 1
fi
if [ ! -d "$WORKDIR" ]; then
mkdir -p "$WORKDIR"
fi
cd "$WORKDIR"
if [ ! -f "$WORKDIR/previoushead.txt" ]; then
touch "$WORKDIR/previoushead.txt"
fi
previous_head=$(< "$WORKDIR/previoushead.txt")
current_head=$(git ls-remote https://github.com/supabase/supabase.git refs/heads/master | cut -f1)
if [ -z "$previous_head" ]; then
echo "Saving current HEAD for the first time..."
printf %s "$current_head" > "$WORKDIR/previoushead.txt"
echo "Success!"
exit 0
fi
if [ "$current_head" = "$previous_head" ]; then
echo "Nothing has changed!"
exit 0
fi
echo "Supabase remote HEAD has updated! Checking for relevant changes locally..."
if [ -d "$WORKDIR/supabase" ]; then
cd "$WORKDIR/supabase"
git pull
else
git clone https://github.com/supabase/supabase "$WORKDIR/supabase"
cd "$WORKDIR/supabase"
fi
if [ -z "$(git diff --name-only "$previous_head" HEAD -- docker)" ]; then
echo "Nothing has changed in /docker!"
else
echo "New changes in /docker! Notifying via email..."
curl --url 'smtps://smtp.gmail.com:465' \
--mail-rcpt 'ben.isenstein@gmail.com' \
--mail-from "$SMTP_USER" \
--user "$SMTP_USER:$SMTP_PASS" \
-T <(echo -e "From: \"PG On Rails\" <$SMTP_USER>\nSubject: New changes in \"/supabase/docker\"\n\nhttps://github.com/supabase/supabase/tree/master/docker\n\nSent securely via curl.")
fi
echo "Saving new HEAD \"$current_head\"..."
cd "$WORKDIR"
printf %s "$current_head" > "$WORKDIR/previoushead.txt"
echo "Success!"
exit 0This path style starting with / has been the common way I've mounted every volume on Railway for years. It's the same pattern that Postgres volumes are mounted. What am I missing here?
a year ago
Yep 💯
a year ago
The volume is mounted correctly, don't worry.
a year ago
That's why I kinda hate shell scripts, hear a lot of horror stories coming from them 😅
a year ago
I was also trying your script locally and it also went correctly. MacOS user here
I'm very new to this level of sophisticated shell scripting and understanding from a systems level why this doesn't work. Can either of you really explain why my script would work on my MacOS, and not in the container on Railway?
Kay well we solved my issue; still flabbergasted that my script doesn't work as-is in the deployment XD
a year ago
I really don't know the answer, but if it actually solves it I will dig up the answer, also really curious 👀
a year ago
How are you executing your script on Railway?
railway.json
{
"$schema": "https://railway.com/railway.schema.json",
"build": {
"watchPatterns": ["/notify-docker-dir/**/*"]
},
"deploy": {
"cronSchedule": "0 * * * *"
}
}
a year ago
But like the start command?
that's why I maintain crons in a monorepo. each job is a folder with a start.sh and a railway.json with the cron schedule
a year ago
I remember running into a lot of issues related to POSIX shells, confirming if it's somehow causing that issue
it should always run in bash, there's a shebang at the top of the file. I guess bash has to be included in the whatever container railway spins up to execute a start.sh...
a year ago
found it!
a year ago
had to fire up my ubuntu vm
a year ago
Railpack always uses sh as the start up command
Status changed to Solved brody • 11 months ago
a year ago
Thinking now, maybe that's a bug 🤔
In my imagination, the sh program will begin executing the file, and see the #!/bin/bash on line 1, and look for /usr/bin/bash on the system in the container to continue executing. Am I wrong about how that works?
a year ago
Pretty sure the sh doesn't do that as that's only for running it as an executable (chmod +x and then ./script.sh)
a year ago
Yep, sh doesn't do that, I changed the shebang to a python one and it still continued the execution
I think you're right. That would happen if I ran ./start.sh in my terminal, but it doesn't work that way when running it in Go
"
When you make a file executable and try to run it directly (for example, ./myscript), the operating system loader:
Reads the first two bytes of the file.
If they are #!, it reads the rest of that line as an interpreter directive.
The OS then executes the specified interpreter, passing the script file’s path as the first argument to that interpreter.
"
a year ago
Not specific to Go but Railpack itself
a year ago
I'll open an issue about it, I really think it should consider the shebang as the documentation itself mentions it
By the way, saw that you're Brazilian, LETS GOOO I was at Carnival In Rio in March
a year ago
a year ago
give a CPF to this man
lolol that's too good. I used a friend's cpf to get an eye exam in Floripa 😆
a year ago
Pretty common, everything in here requires a CPF pretty much
a year ago
the polluted one, São Paulo
a year ago
but im going to floripa next month for a conference 🎉
a year ago
got a nice Airbnb and all, pretty excited
a year ago
Yep, first one also
a year ago
Which one? 👀
a year ago
company is sponsoring the event
Will you have time to see some nice beaches? Campeche, Joaquina, Barra da Lagoa are so nice. I can't wait to surf in Campeche again, had the best wave of my life there in April
a year ago
I'll be in Barra da Lagoa actually, maybe I'll take some surf lessons and take a hike
Definitely hike to Praia da Galheta, I think it's about 45 mins and super wild and natural
a year ago
Thanks for the suggestion, will definitely do
#!/bin/bash
set -eu
if [ -f "./.env" ]; then
source "./.env"
fi
if [ -z "$WORKDIR" ]; then
echo "No WORKIR found in environment. Operation cancelled."
exit 1
fi
if [ -z "$SMTP_USER" ]; then
echo "No SMTP_USER found in environment. Operation cancelled."
exit 1
fi
if [ -z "$SMTP_PASS" ]; then
echo "No SMTP_PASS found in environment. Operation cancelled."
exit 1
fi
if [ ! -d "$WORKDIR" ]; then
mkdir -p "$WORKDIR"
fi
if [ ! -f "$WORKDIR/previoushead.txt" ]; then
touch "$WORKDIR/previoushead.txt"
fi
previous_head=$(< "$WORKDIR/previoushead.txt")
current_head=$(git ls-remote https://github.com/supabase/supabase.git refs/heads/master | cut -f1)
if [ -z "$previous_head" ]; then
echo "Saving current HEAD for the first time..."
printf %s "$current_head" > "$WORKDIR/previoushead.txt"
echo "Success!"
exit 0
fi
if [ "$current_head" = "$previous_head" ]; then
echo "Nothing has changed!"
exit 0
fi
echo "Supabase remote HEAD has updated! Checking for relevant changes locally..."
if [ -d "$WORKDIR/supabase" ]; then
cd "$WORKDIR/supabase"
git pull
else
git clone https://github.com/supabase/supabase "$WORKDIR/supabase"
cd "$WORKDIR/supabase"
fi
if [ -z "$(git diff --name-only "$previous_head" HEAD -- docker)" ]; then
echo "Nothing has changed in /docker!"
else
echo "New changes in /docker! Notifying via email..."
curl --url 'smtps://smtp.gmail.com:465' \
--mail-rcpt 'ben.isenstein@gmail.com' \
--mail-from "$SMTP_USER" \
--user "$SMTP_USER:$SMTP_PASS" \
-T <(echo -e "From: \"PG On Rails\" <$SMTP_USER>\nSubject: New changes in \"/supabase/docker\"\n\nhttps://github.com/supabase/supabase/tree/master/docker\n\nSent securely via curl.")
fi
echo "Saving new HEAD \"$current_head\"..."
printf %s "$current_head" > "$WORKDIR/previoushead.txt"
echo "Success!"
exit 0
a year ago
at what step, sorry? Thought we fixed it
if [ -z "$previous_head" ]; then
echo "Saving current HEAD for the first time..."
printf %s "$current_head" > "$WORKDIR/previoushead.txt"
echo "Success!"
exit 0
fi
a year ago
Let me see if I can see the issue on my Ubuntu VM
a year ago
Also, I don't see if Railpack supports the start command for shell scripts (as I don't see it on the code), but perhaps using ./script.sh would fix the issue (make sure you commit your file with the chmod +x permission).
Kay well I'll fix it tomorrow. Gotta go to bed. Thanks for all your help brother!
a year ago
Ok, I transformed your script to be supported by POSIX compliant shells.
a year ago
#!/bin/sh
set -eu
if [ -f "./.env" ]; then
. "./.env"
fi
if [ -z "${WORKDIR:-}" ]; then
echo "No WORKDIR found in environment. Operation cancelled."
exit 1
fi
if [ -z "${SMTP_USER:-}" ]; then
echo "No SMTP_USER found in environment. Operation cancelled."
exit 1
fi
if [ -z "${SMTP_PASS:-}" ]; then
echo "No SMTP_PASS found in environment. Operation cancelled."
exit 1
fi
if [ ! -d "$WORKDIR" ]; then
mkdir -p "$WORKDIR"
fi
cd "$WORKDIR"
if [ ! -f "$WORKDIR/previoushead.txt" ]; then
touch "$WORKDIR/previoushead.txt"
fi
previous_head=$(cat "$WORKDIR/previoushead.txt")
current_head=$(git ls-remote https://github.com/supabase/supabase.git refs/heads/master | cut -f1)
if [ -z "$previous_head" ]; then
echo "Saving current HEAD for the first time..."
printf %s "$current_head" > "$WORKDIR/previoushead.txt"
echo "Success!"
exit 0
fi
if [ "$current_head" = "$previous_head" ]; then
echo "Nothing has changed!"
exit 0
fi
echo "Supabase remote HEAD has updated! Checking for relevant changes locally..."
if [ -d "$WORKDIR/supabase" ]; then
cd "$WORKDIR/supabase"
git pull
else
git clone https://github.com/supabase/supabase "$WORKDIR/supabase"
cd "$WORKDIR/supabase"
fi
if [ -z "$(git diff --name-only "$previous_head" HEAD -- docker)" ]; then
echo "Nothing has changed in /docker!"
else
echo "New changes in /docker! Notifying via email..."
printf "From: \"PG On Rails\" <%s>\nSubject: New changes in \"/supabase/docker\"\n\nhttps://github.com/supabase/supabase/tree/master/docker\n\nSent securely via curl." "$SMTP_USER" | \
curl --url 'smtps://smtp.gmail.com:465' \
--mail-rcpt 'ben.isenstein@gmail.com' \
--mail-from "$SMTP_USER" \
--user "$SMTP_USER:$SMTP_PASS" \
-T -
fi
echo "Saving new HEAD \"$current_head\"..."
cd "$WORKDIR"
printf %s "$current_head" > "$WORKDIR/previoushead.txt"
echo "Success!"
exit 0a year ago
*I used AI for it, and from my tests, everything seems to be working.
Okay ChatGPT says it's valid. I just wonder whether the <() file syntax was also posix compliant
a year ago
removed part of the script for the initial file only, that might be it
a year ago
Also, where are you seeing the <() syntax?
a year ago
hmmm i'm able to execute the full script with fake environment variables
a year ago
Got it, nevermind then
a year ago
See ya
a year ago
Will be opening an issue on Railpack, doing it as we speak
Oh one last thing. @Brody might like this since I've been trying to do this for literally 2 years at this point - deploy Supabase on Railway with a single shell command
bash <(curl -fsSL https://raw.githubusercontent.com/BenIsenstein/pgonrails-cli/main/start.sh)
Massive shell script that uses the GraphQL API to deploy the PG On Rails template
a year ago
Very nice use of our API.
a year ago
Hey, will be closing the thread, if you've any more issues feel free to create another one
a year ago
!s
a year ago
@Railway
a year ago
!s
Spent some time to improving railpack support here! https://github.com/railwayapp/railpack/pull/328 and https://github.com/railwayapp/railpack/commit/608fada19906397a011234e3116feeb346da10b0