Mounted volume doesn't seem to persist between cron runs
benisenstein
PROOP

5 months ago

Project ID: 675c7104-52e7-4685-9806-2b3cbd0a9af7

Service ID: 84dc4d1a-0cea-44e8-893c-eeacbe4342fe

My cron job writes a commit hash to a volume in order to resume work on the next run. However, on every run it executes the "empty file" branch of my logic, and prints that it's "Saving current HEAD for the first time…"

I am using a WORKRDIR environment var, so I can configure where to write the "current head" txt file. In local I want it in the project root, in the Railway deployment I want it written in the volume. As far as I can tell everything is configured correctly, the WORKDIR var matches the volume mount path. Would appreciate another set of eyes on this, thank you!

Solved$20 Bounty

130 Replies

passos
MODERATOR

5 months ago

Hey, the only options I can think of here are:

  • Is your WORKDIR environment variable actually being set? Perhaps a print/console.log could confirm that.

  • Your volume might be mounted at the wrong path.

Maybe you're also able to attach a file browser template to confirm that there's nothing on the volume itself.

passos
MODERATOR

5 months ago

Also, are you using a Dockerfile?

benisenstein
PROOP

5 months ago

I suppose could check if the var is actually being set, seems farfetched that it wouldn't but stranger things have happened

benisenstein
PROOP

5 months ago

It's just a start.sh

benisenstein
PROOP

5 months ago

no dockerfile

brody
EMPLOYEE

5 months ago

Providing that script would help the community help you.

benisenstein
PROOP

5 months ago

on it

benisenstein
PROOP

5 months ago

#!/bin/bash
set -eu

if [ -f "./.env" ]; then
    source "./.env"
fi

if [ -z "$WORKDIR" ]; then
    echo "No WORKIR found in environment. Operation cancelled."
    exit 1
fi

if [ -z "$SMTP_USER" ]; then
    echo "No SMTP_USER found in environment. Operation cancelled."
    exit 1
fi

if [ -z "$SMTP_PASS" ]; then
    echo "No SMTP_PASS found in environment. Operation cancelled."
    exit 1
fi

if [ ! -d "$WORKDIR" ]; then
    mkdir -p "$WORKDIR"
fi

cd "$WORKDIR"

if [ ! -f "./previoushead.txt" ]; then
    touch "./previoushead.txt"
fi

previous_head=$(< "./previoushead.txt")
current_head=$(git ls-remote https://github.com/supabase/supabase.git refs/heads/master | cut -f1)

if [ -z "$previous_head" ]; then
    echo "Saving current HEAD for the first time..."
    printf %s "$current_head" > "./previoushead.txt"
    echo "Success!"
    exit 0
fi

if [ "$current_head" = "$previous_head" ]; then
    echo "Nothing has changed!"
    exit 0
fi

echo "Supabase remote HEAD has updated! Checking for relevant changes locally..." 

if [ -d "supabase" ]; then
    cd supabase
    git pull
else
    git clone https://github.com/supabase/supabase
    cd supabase
fi

if [ -z "$(git diff --name-only "$previous_head" HEAD -- docker)" ]; then
    echo "Nothing has changed in /docker!"
else
    echo "New changes in /docker! Notifying via email..."

    curl --url 'smtps://smtp.gmail.com:465' \
        --mail-rcpt 'ben.isenstein@gmail.com' \
        --mail-from "$SMTP_USER" \
        --user "$SMTP_USER:$SMTP_PASS" \
        -T <(echo -e "From: \"PG On Rails\" <$SMTP_USER>\nSubject: New changes in \"/supabase/docker\"\n\nhttps://github.com/supabase/supabase/tree/master/docker\n\nSent securely via curl.")
fi

echo "Saving new HEAD \"$current_head\"..."
cd ..
printf %s "$current_head" > "./previoushead.txt"
echo "Success!"
exit 0
passos
MODERATOR

5 months ago

Also, shell scripts commonly continue execution without error even when environment variables are undefined (if not used with the correct flags).

benisenstein
PROOP

5 months ago

Ahh it IS definitely being set. it exits if there is no $WORKDIR var

benisenstein
PROOP

5 months ago

And I just confirmed that WORKDIR and the volume mount path are both /app/volume

brody
EMPLOYEE

5 months ago

Try using all absolute paths.

benisenstein
PROOP

5 months ago

Some more context:

This service builds from a rootDirectory within a monorepo of cron jobs. In this directory I use a railway.json to configure the cron schedule. Now, the docs say that you can't just include a railway.json in a service's rootDirectory and expect it to work, but it's working. But could that be messing up volume behaviour? highly doubt it, but thought I would share nonetheless

benisenstein
PROOP

5 months ago

Is that path not absolute to the root of the container? I tried ~/volume and that was prohibited.

brody
EMPLOYEE

5 months ago

Nothing is messing up the volume behavior; the railway side of things is correct, it's the script.

passos
MODERATOR

5 months ago

Brody said to include $WORKDIR to every path instead of using cd. Here's on how it would look like:

passos
MODERATOR

5 months ago

#!/bin/bash

set -eu

if [ -f "./.env" ]; then
    source "./.env"
fi

if [ -z "$WORKDIR" ]; then
    echo "No WORKIR found in environment. Operation cancelled."
    exit 1
fi

if [ -z "$SMTP_USER" ]; then
    echo "No SMTP_USER found in environment. Operation cancelled."
    exit 1
fi

if [ -z "$SMTP_PASS" ]; then
    echo "No SMTP_PASS found in environment. Operation cancelled."
    exit 1
fi

if [ ! -d "$WORKDIR" ]; then
    mkdir -p "$WORKDIR"
fi

cd "$WORKDIR"

if [ ! -f "$WORKDIR/previoushead.txt" ]; then
    touch "$WORKDIR/previoushead.txt"
fi

previous_head=$(< "$WORKDIR/previoushead.txt")

current_head=$(git ls-remote https://github.com/supabase/supabase.git refs/heads/master | cut -f1)

if [ -z "$previous_head" ]; then
    echo "Saving current HEAD for the first time..."
    printf %s "$current_head" > "$WORKDIR/previoushead.txt"
    echo "Success!"
    exit 0
fi

if [ "$current_head" = "$previous_head" ]; then
    echo "Nothing has changed!"
    exit 0
fi

echo "Supabase remote HEAD has updated! Checking for relevant changes locally..." 

if [ -d "$WORKDIR/supabase" ]; then
    cd "$WORKDIR/supabase"
    git pull
else
    git clone https://github.com/supabase/supabase "$WORKDIR/supabase"
    cd "$WORKDIR/supabase"
fi

if [ -z "$(git diff --name-only "$previous_head" HEAD -- docker)" ]; then
    echo "Nothing has changed in /docker!"
else
    echo "New changes in /docker! Notifying via email..."
    curl --url 'smtps://smtp.gmail.com:465' \
        --mail-rcpt 'ben.isenstein@gmail.com' \
        --mail-from "$SMTP_USER" \
        --user "$SMTP_USER:$SMTP_PASS" \
        -T <(echo -e "From: \"PG On Rails\" <$SMTP_USER>\nSubject: New changes in \"/supabase/docker\"\n\nhttps://github.com/supabase/supabase/tree/master/docker\n\nSent securely via curl.")
fi

echo "Saving new HEAD \"$current_head\"..."

cd "$WORKDIR"

printf %s "$current_head" > "$WORKDIR/previoushead.txt"

echo "Success!"

exit 0
benisenstein
PROOP

5 months ago

This path style starting with / has been the common way I've mounted every volume on Railway for years. It's the same pattern that Postgres volumes are mounted. What am I missing here?

brody
EMPLOYEE

5 months ago

Yep 💯

brody
EMPLOYEE

5 months ago

The volume is mounted correctly, don't worry.

benisenstein
PROOP

5 months ago

Ya I got you, roger on the explicit path building!

benisenstein
PROOP

5 months ago

I really figured that wouldn't be necessary…

benisenstein
PROOP

5 months ago

"works fine on my machine" lollll

benisenstein
PROOP

5 months ago

Because the program cd into the WORKDIR…

passos
MODERATOR

5 months ago

That's why I kinda hate shell scripts, hear a lot of horror stories coming from them 😅

passos
MODERATOR

5 months ago

I was also trying your script locally and it also went correctly. MacOS user here

benisenstein
PROOP

5 months ago

I'm very new to this level of sophisticated shell scripting and understanding from a systems level why this doesn't work. Can either of you really explain why my script would work on my MacOS, and not in the container on Railway?

benisenstein
PROOP

5 months ago

Kay well we solved my issue; still flabbergasted that my script doesn't work as-is in the deployment XD

passos
MODERATOR

5 months ago

I really don't know the answer, but if it actually solves it I will dig up the answer, also really curious 👀

benisenstein
PROOP

5 months ago

Same.

benisenstein
PROOP

5 months ago

Kay let me commit this one minute

passos
MODERATOR

5 months ago

How are you executing your script on Railway?

benisenstein
PROOP

5 months ago

cron job

benisenstein
PROOP

5 months ago

railway.json

{
"$schema": "https://railway.com/railway.schema.json",
"build": {
"watchPatterns": ["/notify-docker-dir/*/"]
},
"deploy": {
"cronSchedule": "0 * * * *"
}
}

passos
MODERATOR

5 months ago

But like the start command?

benisenstein
PROOP

5 months ago

there's none. Railway automatically executes the start.sh

benisenstein
PROOP

5 months ago

that's why I maintain crons in a monorepo. each job is a folder with a start.sh and a railway.json with the cron schedule

passos
MODERATOR

5 months ago

I remember running into a lot of issues related to POSIX shells, confirming if it's somehow causing that issue

benisenstein
PROOP

5 months ago

it should always run in bash, there's a shebang at the top of the file. I guess bash has to be included in the whatever container railway spins up to execute a start.sh…

passos
MODERATOR

5 months ago

found it!

1433665758982049800

passos
MODERATOR

5 months ago

had to fire up my ubuntu vm

passos
MODERATOR

5 months ago

Railpack always uses sh as the start up command

Status changed to Solved brody • 5 months ago

passos
MODERATOR

5 months ago

Thinking now, maybe that's a bug 🤔

benisenstein
PROOP

5 months ago

In my imagination, the sh program will begin executing the file, and see the #!/bin/bash on line 1, and look for /usr/bin/bash on the system in the container to continue executing. Am I wrong about how that works?

passos
MODERATOR

5 months ago

Pretty sure the sh doesn't do that as that's only for running it as an executable (chmod +x and then ./script.sh)

passos
MODERATOR

5 months ago

Yep, sh doesn't do that, I changed the shebang to a python one and it still continued the execution

benisenstein
PROOP

5 months ago

I think you're right. That would happen if I ran ./start.sh in my terminal, but it doesn't work that way when running it in Go

benisenstein
PROOP

5 months ago

"
When you make a file executable and try to run it directly (for example, ./myscript), the operating system loader:

Reads the first two bytes of the file.

If they are #!, it reads the rest of that line as an interpreter directive.

The OS then executes the specified interpreter, passing the script file’s path as the first argument to that interpreter.
"

passos
MODERATOR

5 months ago

Not specific to Go but Railpack itself

passos
MODERATOR

5 months ago

I'll open an issue about it, I really think it should consider the shebang as the documentation itself mentions it

1433667139684012000

benisenstein
PROOP

5 months ago

By the way, saw that you're Brazilian, LETS GOOO I was at Carnival In Rio in March

benisenstein
PROOP

5 months ago

VAMOS LA BRODE

benisenstein
PROOP

5 months ago

PE NA AREIA, CAIPIRINHA, AGUA DE COCO, CERVEZINHA

passos
MODERATOR

5 months ago

give a CPF to this man

benisenstein
PROOP

5 months ago

hahahahha

benisenstein
PROOP

5 months ago

DA UM CPF

benisenstein
PROOP

5 months ago

lolol that's too good. I used a friend's cpf to get an eye exam in Floripa 😆

passos
MODERATOR

5 months ago

Pretty common, everything in here requires a CPF pretty much

benisenstein
PROOP

5 months ago

So where in Brazil are you living?

passos
MODERATOR

5 months ago

the polluted one, São Paulo

passos
MODERATOR

5 months ago

but im going to floripa next month for a conference 🎉

benisenstein
PROOP

5 months ago

Amazing

passos
MODERATOR

5 months ago

got a nice Airbnb and all, pretty excited

benisenstein
PROOP

5 months ago

The conference part

passos
MODERATOR

5 months ago

Yep, first one also

medim
MODERATOR

5 months ago

Which one? 👀

passos
MODERATOR

5 months ago

company is sponsoring the event

benisenstein
PROOP

5 months ago

Will you have time to see some nice beaches? Campeche, Joaquina, Barra da Lagoa are so nice. I can't wait to surf in Campeche again, had the best wave of my life there in April

passos
MODERATOR

5 months ago

I'll be in Barra da Lagoa actually, maybe I'll take some surf lessons and take a hike

benisenstein
PROOP

5 months ago

Definitely hike to Praia da Galheta, I think it's about 45 mins and super wild and natural

passos
MODERATOR

5 months ago

Thanks for the suggestion, will definitely do

benisenstein
PROOP

5 months ago

Okay my script is still stopping at the first step man…

benisenstein
PROOP

5 months ago

#!/bin/bash
set -eu

if [ -f "./.env" ]; then
    source "./.env"
fi

if [ -z "$WORKDIR" ]; then
    echo "No WORKIR found in environment. Operation cancelled."
    exit 1
fi

if [ -z "$SMTP_USER" ]; then
    echo "No SMTP_USER found in environment. Operation cancelled."
    exit 1
fi

if [ -z "$SMTP_PASS" ]; then
    echo "No SMTP_PASS found in environment. Operation cancelled."
    exit 1
fi

if [ ! -d "$WORKDIR" ]; then
    mkdir -p "$WORKDIR"
fi

if [ ! -f "$WORKDIR/previoushead.txt" ]; then
    touch "$WORKDIR/previoushead.txt"
fi

previous_head=$(< "$WORKDIR/previoushead.txt")
current_head=$(git ls-remote https://github.com/supabase/supabase.git refs/heads/master | cut -f1)

if [ -z "$previous_head" ]; then
    echo "Saving current HEAD for the first time..."
    printf %s "$current_head" > "$WORKDIR/previoushead.txt"
    echo "Success!"
    exit 0
fi

if [ "$current_head" = "$previous_head" ]; then
    echo "Nothing has changed!"
    exit 0
fi

echo "Supabase remote HEAD has updated! Checking for relevant changes locally..." 

if [ -d "$WORKDIR/supabase" ]; then
    cd "$WORKDIR/supabase"
    git pull
else
    git clone https://github.com/supabase/supabase "$WORKDIR/supabase"
    cd "$WORKDIR/supabase"
fi

if [ -z "$(git diff --name-only "$previous_head" HEAD -- docker)" ]; then
    echo "Nothing has changed in /docker!"
else
    echo "New changes in /docker! Notifying via email..."

    curl --url 'smtps://smtp.gmail.com:465' \
        --mail-rcpt 'ben.isenstein@gmail.com' \
        --mail-from "$SMTP_USER" \
        --user "$SMTP_USER:$SMTP_PASS" \
        -T <(echo -e "From: \"PG On Rails\" <$SMTP_USER>\nSubject: New changes in \"/supabase/docker\"\n\nhttps://github.com/supabase/supabase/tree/master/docker\n\nSent securely via curl.")
fi

echo "Saving new HEAD \"$current_head\"..."
printf %s "$current_head" > "$WORKDIR/previoushead.txt"
echo "Success!"
exit 0
passos
MODERATOR

5 months ago

at what step, sorry? Thought we fixed it

benisenstein
PROOP

5 months ago

if [ -z "$previoushead" ]; then echo "Saving current HEAD for the first time…" printf %s "$currenthead" > "$WORKDIR/previoushead.txt"
echo "Success!"
exit 0
fi

benisenstein
PROOP

5 months ago

1433669932436029700

benisenstein
PROOP

5 months ago

The past few manual runs

passos
MODERATOR

5 months ago

Let me see if I can see the issue on my Ubuntu VM

benisenstein
PROOP

5 months ago

cool, thanks man

passos
MODERATOR

5 months ago

Also, I don't see if Railpack supports the start command for shell scripts (as I don't see it on the code), but perhaps using ./script.sh would fix the issue (make sure you commit your file with the chmod +x permission).

benisenstein
PROOP

5 months ago

Kay I'll try it

benisenstein
PROOP

5 months ago

It's executable for sure

benisenstein
PROOP

5 months ago

1433672166817398800

benisenstein
PROOP

5 months ago

Nope. It expects a start.sh

benisenstein
PROOP

5 months ago

Kay well I'll fix it tomorrow. Gotta go to bed. Thanks for all your help brother!

passos
MODERATOR

5 months ago

Ok, I transformed your script to be supported by POSIX compliant shells.

passos
MODERATOR

5 months ago

#!/bin/sh

set -eu

if [ -f "./.env" ]; then
    . "./.env"
fi

if [ -z "${WORKDIR:-}" ]; then
    echo "No WORKDIR found in environment. Operation cancelled."
    exit 1
fi

if [ -z "${SMTP_USER:-}" ]; then
    echo "No SMTP_USER found in environment. Operation cancelled."
    exit 1
fi

if [ -z "${SMTP_PASS:-}" ]; then
    echo "No SMTP_PASS found in environment. Operation cancelled."
    exit 1
fi

if [ ! -d "$WORKDIR" ]; then
    mkdir -p "$WORKDIR"
fi

cd "$WORKDIR"

if [ ! -f "$WORKDIR/previoushead.txt" ]; then
    touch "$WORKDIR/previoushead.txt"
fi

previous_head=$(cat "$WORKDIR/previoushead.txt")

current_head=$(git ls-remote https://github.com/supabase/supabase.git refs/heads/master | cut -f1)

if [ -z "$previous_head" ]; then
    echo "Saving current HEAD for the first time..."
    printf %s "$current_head" > "$WORKDIR/previoushead.txt"
    echo "Success!"
    exit 0
fi

if [ "$current_head" = "$previous_head" ]; then
    echo "Nothing has changed!"
    exit 0
fi

echo "Supabase remote HEAD has updated! Checking for relevant changes locally..." 

if [ -d "$WORKDIR/supabase" ]; then
    cd "$WORKDIR/supabase"
    git pull
else
    git clone https://github.com/supabase/supabase "$WORKDIR/supabase"
    cd "$WORKDIR/supabase"
fi

if [ -z "$(git diff --name-only "$previous_head" HEAD -- docker)" ]; then
    echo "Nothing has changed in /docker!"
else
    echo "New changes in /docker! Notifying via email..."
    printf "From: \"PG On Rails\" <%s>\nSubject: New changes in \"/supabase/docker\"\n\nhttps://github.com/supabase/supabase/tree/master/docker\n\nSent securely via curl." "$SMTP_USER" | \
    curl --url 'smtps://smtp.gmail.com:465' \
        --mail-rcpt 'ben.isenstein@gmail.com' \
        --mail-from "$SMTP_USER" \
        --user "$SMTP_USER:$SMTP_PASS" \
        -T -
fi

echo "Saving new HEAD \"$current_head\"..."

cd "$WORKDIR"

printf %s "$current_head" > "$WORKDIR/previoushead.txt"

echo "Success!"

exit 0
benisenstein
PROOP

5 months ago

woah really?

benisenstein
PROOP

5 months ago

let's see what changed

benisenstein
PROOP

5 months ago

amazing

passos
MODERATOR

5 months ago

*I used AI for it, and from my tests, everything seems to be working.

benisenstein
PROOP

5 months ago

is cat better than printf? hmm…

benisenstein
PROOP

5 months ago

I'm gonna deploy it and see

benisenstein
PROOP

5 months ago

fuck sleep

benisenstein
PROOP

5 months ago

the format of the SMTP curl looks weird… have you tested it?

benisenstein
PROOP

5 months ago

Okay ChatGPT says it's valid. I just wonder whether the <() file syntax was also posix compliant

benisenstein
PROOP

5 months ago

it was not.

benisenstein
PROOP

5 months ago

shell substitution, expansions etc all Bash

passos
MODERATOR

5 months ago

removed part of the script for the initial file only, that might be it

benisenstein
PROOP

5 months ago

Cron services don't always execute when you try to manually run it…

passos
MODERATOR

5 months ago

Also, where are you seeing the &lt;() syntax?

benisenstein
PROOP

5 months ago

Yo we got like 5 guys in this thread. This is fucking baller

benisenstein
PROOP

5 months ago

That's how I was feeding input to the SMTP request before

benisenstein
PROOP

5 months ago

But I literally can't get the service to execute right now

benisenstein
PROOP

5 months ago

The cron stuff could use a glow-up honestly

passos
MODERATOR

5 months ago

hmmm i'm able to execute the full script with fake environment variables

1433675950213824500

benisenstein
PROOP

5 months ago

Okay finally it ran

benisenstein
PROOP

5 months ago

Yep we got "nothing has changed!"

benisenstein
PROOP

5 months ago

amazing

benisenstein
PROOP

5 months ago

Hopefully the other logic branches work fine too

benisenstein
PROOP

5 months ago

We'll see tomorrow

passos
MODERATOR

5 months ago

Got it, nevermind then

passos
MODERATOR

5 months ago

See ya

benisenstein
PROOP

5 months ago

And I guess railpack has some work to do supporting Bash? pretty please?

passos
MODERATOR

5 months ago

Will be opening an issue on Railpack, doing it as we speak

benisenstein
PROOP

5 months ago

Thanks for your help, and for hanging out in my thread ✊

benisenstein
PROOP

5 months ago

Oh one last thing. @Brody might like this since I've been trying to do this for literally 2 years at this point - deploy Supabase on Railway with a single shell command

benisenstein
PROOP

5 months ago

bash &lt;(curl -fsSL https://raw.githubusercontent.com/BenIsenstein/pgonrails-cli/main/start.sh)

benisenstein
PROOP

5 months ago

Massive shell script that uses the GraphQL API to deploy the PG On Rails template

benisenstein
PROOP

5 months ago

and eject it, and configure the ejected services

brody
EMPLOYEE

5 months ago

Very nice use of our API.

benisenstein
PROOP

5 months ago

and wait for service health as needed

benisenstein
PROOP

5 months ago

Thank you sir

passos
MODERATOR

4 months ago

Hey, will be closing the thread, if you've any more issues feel free to create another one

passos
MODERATOR

4 months ago

!s

passos
MODERATOR

4 months ago

@Railway

passos
MODERATOR

4 months ago

!s

benisenstein
PROOP

4 months ago

Close it!

Loading...