Mounted volume doesn't seem to persist between cron runs - Railway Help Station

Mounted volume doesn't seem to persist between cron runs

benisenstein

PROOP

16 days ago

Project ID: 675c7104-52e7-4685-9806-2b3cbd0a9af7

Service ID: 84dc4d1a-0cea-44e8-893c-eeacbe4342fe

My cron job writes a commit hash to a volume in order to resume work on the next run. However, on every run it executes the "empty file" branch of my logic, and prints that it's "Saving current HEAD for the first time…"

I am using a WORKRDIR environment var, so I can configure where to write the "current head" txt file. In local I want it in the project root, in the Railway deployment I want it written in the volume. As far as I can tell everything is configured correctly, the WORKDIR var matches the volume mount path. Would appreciate another set of eyes on this, thank you!

Solved$20 Bounty

0 Replies

16 days ago

Hey, the only options I can think of here are:

Is your WORKDIR environment variable actually being set? Perhaps a print/console.log could confirm that.
Your volume might be mounted at the wrong path.

Maybe you're also able to attach a file browser template to confirm that there's nothing on the volume itself.

16 days ago

Also, are you using a Dockerfile?

benisenstein

PROOP

16 days ago

I suppose could check if the var is actually being set, seems farfetched that it wouldn't but stranger things have happened

benisenstein

PROOP

16 days ago

It's just a [start.sh](start.sh)

benisenstein

PROOP

16 days ago

no dockerfile

16 days ago

Providing that script would help the community help you.

benisenstein

PROOP

16 days ago

on it

benisenstein

PROOP

16 days ago

#!/bin/bash
set -eu

if [ -f "./.env" ]; then
    source "./.env"
fi

if [ -z "$WORKDIR" ]; then
    echo "No WORKIR found in environment. Operation cancelled."
    exit 1
fi

if [ -z "$SMTP_USER" ]; then
    echo "No SMTP_USER found in environment. Operation cancelled."
    exit 1
fi

if [ -z "$SMTP_PASS" ]; then
    echo "No SMTP_PASS found in environment. Operation cancelled."
    exit 1
fi

if [ ! -d "$WORKDIR" ]; then
    mkdir -p "$WORKDIR"
fi

cd "$WORKDIR"

if [ ! -f "./previoushead.txt" ]; then
    touch "./previoushead.txt"
fi

previous_head=$(&lt; "./previoushead.txt")
current_head=$(git ls-remote https://github.com/supabase/supabase.git refs/heads/master | cut -f1)

if [ -z "$previous_head" ]; then
    echo "Saving current HEAD for the first time..."
    printf %s "$current_head" &gt; "./previoushead.txt"
    echo "Success!"
    exit 0
fi

if [ "$current_head" = "$previous_head" ]; then
    echo "Nothing has changed!"
    exit 0
fi

echo "Supabase remote HEAD has updated! Checking for relevant changes locally..." 

if [ -d "supabase" ]; then
    cd supabase
    git pull
else
    git clone https://github.com/supabase/supabase
    cd supabase
fi

if [ -z "$(git diff --name-only "$previous_head" HEAD -- docker)" ]; then
    echo "Nothing has changed in /docker!"
else
    echo "New changes in /docker! Notifying via email..."

    curl --url 'smtps://smtp.gmail.com:465' \
        --mail-rcpt 'ben.isenstein@gmail.com' \
        --mail-from "$SMTP_USER" \
        --user "$SMTP_USER:$SMTP_PASS" \
        -T &lt;(echo -e "From: \"PG On Rails\" &lt;$SMTP_USER&gt;\nSubject: New changes in \"/supabase/docker\"\n\nhttps://github.com/supabase/supabase/tree/master/docker\n\nSent securely via curl.")
fi

echo "Saving new HEAD \"$current_head\"..."
cd ..
printf %s "$current_head" &gt; "./previoushead.txt"
echo "Success!"
exit 0

16 days ago

Also, shell scripts commonly continue execution without error even when environment variables are undefined (if not used with the correct flags).

benisenstein

PROOP

16 days ago

Ahh it IS definitely being set. it exits if there is no $WORKDIR var

benisenstein

PROOP

16 days ago

And I just confirmed that WORKDIR and the volume mount path are both /app/volume

16 days ago

Try using all absolute paths.

benisenstein

PROOP

16 days ago

Some more context:

This service builds from a rootDirectory within a monorepo of cron jobs. In this directory I use a railway.json to configure the cron schedule. Now, the docs say that you can't just include a railway.json in a service's rootDirectory and expect it to work, but it's working. But could that be messing up volume behaviour? highly doubt it, but thought I would share nonetheless

benisenstein

PROOP

16 days ago

Is that path not absolute to the root of the container? I tried ~/volume and that was prohibited.

16 days ago

Nothing is messing up the volume behavior; the railway side of things is correct, it's the script.

16 days ago

Brody said to include $WORKDIR to every path instead of using cd. Here's on how it would look like:

16 days ago

#!/bin/bash

set -eu

if [ -f "./.env" ]; then
    source "./.env"
fi

if [ -z "$WORKDIR" ]; then
    echo "No WORKIR found in environment. Operation cancelled."
    exit 1
fi

if [ -z "$SMTP_USER" ]; then
    echo "No SMTP_USER found in environment. Operation cancelled."
    exit 1
fi

if [ -z "$SMTP_PASS" ]; then
    echo "No SMTP_PASS found in environment. Operation cancelled."
    exit 1
fi

if [ ! -d "$WORKDIR" ]; then
    mkdir -p "$WORKDIR"
fi

cd "$WORKDIR"

if [ ! -f "$WORKDIR/previoushead.txt" ]; then
    touch "$WORKDIR/previoushead.txt"
fi

previous_head=$(&lt; "$WORKDIR/previoushead.txt")

current_head=$(git ls-remote https://github.com/supabase/supabase.git refs/heads/master | cut -f1)

if [ -z "$previous_head" ]; then
    echo "Saving current HEAD for the first time..."
    printf %s "$current_head" &gt; "$WORKDIR/previoushead.txt"
    echo "Success!"
    exit 0
fi

if [ "$current_head" = "$previous_head" ]; then
    echo "Nothing has changed!"
    exit 0
fi

echo "Supabase remote HEAD has updated! Checking for relevant changes locally..." 

if [ -d "$WORKDIR/supabase" ]; then
    cd "$WORKDIR/supabase"
    git pull
else
    git clone https://github.com/supabase/supabase "$WORKDIR/supabase"
    cd "$WORKDIR/supabase"
fi

if [ -z "$(git diff --name-only "$previous_head" HEAD -- docker)" ]; then
    echo "Nothing has changed in /docker!"
else
    echo "New changes in /docker! Notifying via email..."
    curl --url 'smtps://smtp.gmail.com:465' \
        --mail-rcpt 'ben.isenstein@gmail.com' \
        --mail-from "$SMTP_USER" \
        --user "$SMTP_USER:$SMTP_PASS" \
        -T &lt;(echo -e "From: \"PG On Rails\" &lt;$SMTP_USER&gt;\nSubject: New changes in \"/supabase/docker\"\n\nhttps://github.com/supabase/supabase/tree/master/docker\n\nSent securely via curl.")
fi

echo "Saving new HEAD \"$current_head\"..."

cd "$WORKDIR"

printf %s "$current_head" &gt; "$WORKDIR/previoushead.txt"

echo "Success!"

exit 0

benisenstein

PROOP

16 days ago

This path style starting with / has been the common way I've mounted every volume on Railway for years. It's the same pattern that Postgres volumes are mounted. What am I missing here?

16 days ago

Yep 💯

16 days ago

The volume is mounted correctly, don't worry.

benisenstein

PROOP

16 days ago

Ya I got you, roger on the explicit path building!

benisenstein

PROOP

16 days ago

I really figured that wouldn't be necessary…

benisenstein

PROOP

16 days ago

"works fine on my machine" lollll

benisenstein

PROOP

16 days ago

Because the program cd into the WORKDIR…

16 days ago

That's why I kinda hate shell scripts, hear a lot of horror stories coming from them 😅

16 days ago

I was also trying your script locally and it also went correctly. MacOS user here

benisenstein

PROOP

16 days ago

I'm very new to this level of sophisticated shell scripting and understanding from a systems level why this doesn't work. Can either of you really explain why my script would work on my MacOS, and not in the container on Railway?

benisenstein

PROOP

16 days ago

Kay well we solved my issue; still flabbergasted that my script doesn't work as-is in the deployment XD

16 days ago

I really don't know the answer, but if it actually solves it I will dig up the answer, also really curious 👀

benisenstein

PROOP

16 days ago

Same.

benisenstein

PROOP

16 days ago

Kay let me commit this one minute

16 days ago

How are you executing your script on Railway?

benisenstein

PROOP

16 days ago

cron job

benisenstein

PROOP

16 days ago

railway.json

{
"$schema": "https://railway.com/railway.schema.json",
"build": {
"watchPatterns": ["/notify-docker-dir/*/"]
},
"deploy": {
"cronSchedule": "0 * * * *"
}
}

16 days ago

But like the start command?

benisenstein

PROOP

16 days ago

there's none. Railway automatically executes the [start.sh](start.sh)

benisenstein

PROOP

16 days ago

that's why I maintain crons in a monorepo. each job is a folder with a [start.sh](start.sh) and a railway.json with the cron schedule

16 days ago

I remember running into a lot of issues related to POSIX shells, confirming if it's somehow causing that issue

benisenstein

PROOP

16 days ago

it should always run in bash, there's a shebang at the top of the file. I guess bash has to be included in the whatever container railway spins up to execute a [start.sh](start.sh)…

16 days ago

found it!

1433665758982049800

16 days ago

had to fire up my ubuntu vm

16 days ago

Railpack always uses sh as the start up command

Status changed to Solved brody • 16 days ago

16 days ago

Thinking now, maybe that's a bug 🤔

benisenstein

PROOP

16 days ago

In my imagination, the sh program will begin executing the file, and see the #!/bin/bash on line 1, and look for /usr/bin/bash on the system in the container to continue executing. Am I wrong about how that works?

16 days ago

Pretty sure the sh doesn't do that as that's only for running it as an executable (chmod +x and then ./script.sh)

16 days ago

Yep, sh doesn't do that, I changed the shebang to a python one and it still continued the execution

benisenstein

PROOP

16 days ago

I think you're right. That would happen if I ran ./[start.sh](start.sh) in my terminal, but it doesn't work that way when running it in Go

benisenstein

PROOP

16 days ago

"
When you make a file executable and try to run it directly (for example, ./myscript), the operating system loader:

Reads the first two bytes of the file.

If they are #!, it reads the rest of that line as an interpreter directive.

The OS then executes the specified interpreter, passing the script file’s path as the first argument to that interpreter.
"

16 days ago

Not specific to Go but Railpack itself

16 days ago

I'll open an issue about it, I really think it should consider the shebang as the documentation itself mentions it

1433667139684012000

benisenstein

PROOP

16 days ago

By the way, saw that you're Brazilian, LETS GOOO I was at Carnival In Rio in March

benisenstein

PROOP

16 days ago

VAMOS LA BRODE

16 days ago

https://tenor.com/view/brasil-mentioned-brasil-mentioned-brahma-anime-gif-15357834161136461942

benisenstein

PROOP

16 days ago

PE NA AREIA, CAIPIRINHA, AGUA DE COCO, CERVEZINHA

16 days ago

give a CPF to this man

benisenstein

PROOP

16 days ago

hahahahha

benisenstein

PROOP

16 days ago

DA UM CPF

benisenstein

PROOP

16 days ago

lolol that's too good. I used a friend's cpf to get an eye exam in Floripa 😆

16 days ago

Pretty common, everything in here requires a CPF pretty much

benisenstein

PROOP

16 days ago

So where in Brazil are you living?

16 days ago

the polluted one, São Paulo

16 days ago

but im going to floripa next month for a conference 🎉

benisenstein

PROOP

16 days ago

Amazing

16 days ago

got a nice Airbnb and all, pretty excited

benisenstein

PROOP

16 days ago

The conference part

16 days ago

Yep, first one also

16 days ago

Which one? 👀

16 days ago

company is sponsoring the event

benisenstein

PROOP

16 days ago

Will you have time to see some nice beaches? Campeche, Joaquina, Barra da Lagoa are so nice. I can't wait to surf in Campeche again, had the best wave of my life there in April

16 days ago

I'll be in Barra da Lagoa actually, maybe I'll take some surf lessons and take a hike

benisenstein

PROOP

16 days ago

Definitely hike to Praia da Galheta, I think it's about 45 mins and super wild and natural

16 days ago

Thanks for the suggestion, will definitely do

benisenstein

PROOP

16 days ago

Okay my script is still stopping at the first step man…

benisenstein

PROOP

16 days ago

#!/bin/bash
set -eu

if [ -f "./.env" ]; then
    source "./.env"
fi

if [ -z "$WORKDIR" ]; then
    echo "No WORKIR found in environment. Operation cancelled."
    exit 1
fi

if [ -z "$SMTP_USER" ]; then
    echo "No SMTP_USER found in environment. Operation cancelled."
    exit 1
fi

if [ -z "$SMTP_PASS" ]; then
    echo "No SMTP_PASS found in environment. Operation cancelled."
    exit 1
fi

if [ ! -d "$WORKDIR" ]; then
    mkdir -p "$WORKDIR"
fi

if [ ! -f "$WORKDIR/previoushead.txt" ]; then
    touch "$WORKDIR/previoushead.txt"
fi

previous_head=$(&lt; "$WORKDIR/previoushead.txt")
current_head=$(git ls-remote https://github.com/supabase/supabase.git refs/heads/master | cut -f1)

if [ -z "$previous_head" ]; then
    echo "Saving current HEAD for the first time..."
    printf %s "$current_head" &gt; "$WORKDIR/previoushead.txt"
    echo "Success!"
    exit 0
fi

if [ "$current_head" = "$previous_head" ]; then
    echo "Nothing has changed!"
    exit 0
fi

echo "Supabase remote HEAD has updated! Checking for relevant changes locally..." 

if [ -d "$WORKDIR/supabase" ]; then
    cd "$WORKDIR/supabase"
    git pull
else
    git clone https://github.com/supabase/supabase "$WORKDIR/supabase"
    cd "$WORKDIR/supabase"
fi

if [ -z "$(git diff --name-only "$previous_head" HEAD -- docker)" ]; then
    echo "Nothing has changed in /docker!"
else
    echo "New changes in /docker! Notifying via email..."

    curl --url 'smtps://smtp.gmail.com:465' \
        --mail-rcpt 'ben.isenstein@gmail.com' \
        --mail-from "$SMTP_USER" \
        --user "$SMTP_USER:$SMTP_PASS" \
        -T &lt;(echo -e "From: \"PG On Rails\" &lt;$SMTP_USER&gt;\nSubject: New changes in \"/supabase/docker\"\n\nhttps://github.com/supabase/supabase/tree/master/docker\n\nSent securely via curl.")
fi

echo "Saving new HEAD \"$current_head\"..."
printf %s "$current_head" &gt; "$WORKDIR/previoushead.txt"
echo "Success!"
exit 0

16 days ago

at what step, sorry? Thought we fixed it

benisenstein

PROOP

16 days ago

if [ -z "$previoushead" ]; then echo "Saving current HEAD for the first time…" printf %s "$currenthead" > "$WORKDIR/previoushead.txt"
echo "Success!"
exit 0
fi

benisenstein

PROOP

16 days ago

1433669932436029700

benisenstein

PROOP

16 days ago

The past few manual runs

16 days ago

Let me see if I can see the issue on my Ubuntu VM

benisenstein

PROOP

16 days ago

cool, thanks man

16 days ago

Also, I don't see if Railpack supports the start command for shell scripts (as I don't see it on the code), but perhaps using ./[script.sh](script.sh) would fix the issue (make sure you commit your file with the chmod +x permission).

benisenstein

PROOP

16 days ago

Kay I'll try it

benisenstein

PROOP

16 days ago

It's executable for sure

benisenstein

PROOP

16 days ago

1433672166817398800

benisenstein

PROOP

16 days ago

Nope. It expects a [start.sh](start.sh)

benisenstein

PROOP

16 days ago

Kay well I'll fix it tomorrow. Gotta go to bed. Thanks for all your help brother!

16 days ago

Ok, I transformed your script to be supported by POSIX compliant shells.

16 days ago

#!/bin/sh

set -eu

if [ -f "./.env" ]; then
    . "./.env"
fi

if [ -z "${WORKDIR:-}" ]; then
    echo "No WORKDIR found in environment. Operation cancelled."
    exit 1
fi

if [ -z "${SMTP_USER:-}" ]; then
    echo "No SMTP_USER found in environment. Operation cancelled."
    exit 1
fi

if [ -z "${SMTP_PASS:-}" ]; then
    echo "No SMTP_PASS found in environment. Operation cancelled."
    exit 1
fi

if [ ! -d "$WORKDIR" ]; then
    mkdir -p "$WORKDIR"
fi

cd "$WORKDIR"

if [ ! -f "$WORKDIR/previoushead.txt" ]; then
    touch "$WORKDIR/previoushead.txt"
fi

previous_head=$(cat "$WORKDIR/previoushead.txt")

current_head=$(git ls-remote https://github.com/supabase/supabase.git refs/heads/master | cut -f1)

if [ -z "$previous_head" ]; then
    echo "Saving current HEAD for the first time..."
    printf %s "$current_head" &gt; "$WORKDIR/previoushead.txt"
    echo "Success!"
    exit 0
fi

if [ "$current_head" = "$previous_head" ]; then
    echo "Nothing has changed!"
    exit 0
fi

echo "Supabase remote HEAD has updated! Checking for relevant changes locally..." 

if [ -d "$WORKDIR/supabase" ]; then
    cd "$WORKDIR/supabase"
    git pull
else
    git clone https://github.com/supabase/supabase "$WORKDIR/supabase"
    cd "$WORKDIR/supabase"
fi

if [ -z "$(git diff --name-only "$previous_head" HEAD -- docker)" ]; then
    echo "Nothing has changed in /docker!"
else
    echo "New changes in /docker! Notifying via email..."
    printf "From: \"PG On Rails\" &lt;%s&gt;\nSubject: New changes in \"/supabase/docker\"\n\nhttps://github.com/supabase/supabase/tree/master/docker\n\nSent securely via curl." "$SMTP_USER" | \
    curl --url 'smtps://smtp.gmail.com:465' \
        --mail-rcpt 'ben.isenstein@gmail.com' \
        --mail-from "$SMTP_USER" \
        --user "$SMTP_USER:$SMTP_PASS" \
        -T -
fi

echo "Saving new HEAD \"$current_head\"..."

cd "$WORKDIR"

printf %s "$current_head" &gt; "$WORKDIR/previoushead.txt"

echo "Success!"

exit 0

benisenstein

PROOP

16 days ago

woah really?

benisenstein

PROOP

16 days ago

let's see what changed

benisenstein

PROOP

16 days ago

amazing

16 days ago

*I used AI for it, and from my tests, everything seems to be working.

benisenstein

PROOP

16 days ago

is cat better than printf? hmm…

benisenstein

PROOP

16 days ago

I'm gonna deploy it and see

benisenstein

PROOP

16 days ago

fuck sleep

benisenstein

PROOP

16 days ago

the format of the SMTP curl looks weird… have you tested it?

benisenstein

PROOP

16 days ago

Okay ChatGPT says it's valid. I just wonder whether the <() file syntax was also posix compliant

benisenstein

PROOP

16 days ago

it was not.

benisenstein

PROOP

16 days ago

shell substitution, expansions etc all Bash

16 days ago

removed part of the script for the initial file only, that might be it

benisenstein

PROOP

16 days ago

Cron services don't always execute when you try to manually run it…

16 days ago

Also, where are you seeing the <() syntax?

benisenstein

PROOP

16 days ago

Yo we got like 5 guys in this thread. This is fucking baller

benisenstein

PROOP

16 days ago

That's how I was feeding input to the SMTP request before

benisenstein

PROOP

16 days ago

But I literally can't get the service to execute right now

benisenstein

PROOP

16 days ago

The cron stuff could use a glow-up honestly

16 days ago

hmmm i'm able to execute the full script with fake environment variables

1433675950213824500

benisenstein

PROOP

16 days ago

Okay finally it ran

benisenstein

PROOP

16 days ago

Yep we got "nothing has changed!"

benisenstein

PROOP

16 days ago

amazing

benisenstein

PROOP

16 days ago

Hopefully the other logic branches work fine too

benisenstein

PROOP

16 days ago

We'll see tomorrow

16 days ago

Got it, nevermind then

16 days ago

See ya

benisenstein

PROOP

16 days ago

And I guess railpack has some work to do supporting Bash? pretty please?

16 days ago

Will be opening an issue on Railpack, doing it as we speak

benisenstein

PROOP

16 days ago

Thanks for your help, and for hanging out in my thread ✊

benisenstein

PROOP

16 days ago

Oh one last thing. @Brody might like this since I've been trying to do this for literally 2 years at this point - deploy Supabase on Railway with a single shell command

benisenstein

PROOP

16 days ago

bash <(curl -fsSL [https://raw.githubusercontent.com/BenIsenstein/pgonrails-cli/main/start.sh](https://raw.githubusercontent.com/BenIsenstein/pgonrails-cli/main/start.sh))

benisenstein

PROOP

16 days ago

Massive shell script that uses the GraphQL API to deploy the PG On Rails template

benisenstein

PROOP

16 days ago

and eject it, and configure the ejected services

16 days ago

Very nice use of our API.

benisenstein

PROOP

16 days ago

and wait for service health as needed

benisenstein

PROOP

16 days ago

Thank you sir

15 days ago

Hey, will be closing the thread, if you've any more issues feel free to create another one

15 days ago

!s

15 days ago

@Railway

15 days ago

!s

benisenstein

PROOP

15 days ago

Close it!

iloveitaly

HOBBY

9 days ago

Spent some time to improving railpack support here! https://github.com/railwayapp/railpack/pull/328 and https://github.com/railwayapp/railpack/commit/608fada19906397a011234e3116feeb346da10b0