18 days ago
I've noticed lately that my services tend to "ignore" the latest commits. They don't always do it - perhaps the GitHub commit webhook is being dropped at times?
For example, I just pushed a change to GitHub about 10-15 minutes ago and my API service didn't even see the change to trigger a deployment - normally it'll see it and determine "no changes made" so it won't deploy it. In this case, I had to manually remove the service and deploy fresh from my repo.
I've already tried disconnecting/reconnecting my repo as well as my GitHub account in the settings, but I'm still seeing the same issue intermittently.
It would be wonderful to have a button that allows us to deploy a new service while we have one running so that we can enable the auto failover without having to actually take the service down and bring it back up, in the case of events like this.
7 Replies
Status changed to Awaiting Railway Response Railway • 18 days ago
17 days ago
Just encountered another instance. Pushed a change at 2:25pm EST with api/** on the service watch list, but the service didn't even see it as a skipped deployment.
17 days ago
This thread has been marked as public for community involvement, as it does not contain any sensitive or personal information. Any further activity in this thread will be visible to everyone.
Status changed to Open Railway • 17 days ago
17 days ago
Hey, can you try a Cntrl + K -> Deploy latest commit, with the affected service pane open? This normally fixes this issue.
16 days ago
We are experiencing the same issue @medim.
It's not consistent but happening frequently enough become a problem.
This morning we had an incident because of this behavior. Our API deployed but a worker which needed the latest code did not deploy.
I solved it by simply disconnecting and re-connecting the same branch and the latest commit was deployed.
Is there a way for us to debug this to understand what is causing this?
Do you all have a record of the web hook events coming in so we can check if they were not delivered or not correctly processed by railway?
gtalarico
We are experiencing the same issue @medim.It's not consistent but happening frequently enough become a problem.This morning we had an incident because of this behavior. Our API deployed but a worker which needed the latest code did not deploy. I solved it by simply disconnecting and re-connecting the same branch and the latest commit was deployed.Is there a way for us to debug this to understand what is causing this? Do you all have a record of the web hook events coming in so we can check if they were not delivered or not correctly processed by railway?
15 days ago
Do you have any watch paths configured in your service settings?
darseen
Do you have any watch paths configured in your service settings?
15 days ago
Thanks for asking @darseen.
Yes we have - an it includes the worker code itself as well as the api that got correctly auto-deployed.
watch paths are set in railway.toml and both services got auto-deployed on the staging environment as well
gtalarico
Thanks for asking @darseen.Yes we have - an it includes the worker code itself as well as the api that got correctly auto-deployed.watch paths are set in railway.toml and both services got auto-deployed on the staging environment as well
15 days ago
Well, if your watch paths are configured correctly, one other thing you can check, is the branch you are making changes to (you might have main and staging). Branch selection and watch paths, are usually the most common misconfigurations causing this problem.
If you've checked both, and there isn't a problem. You can navigate to your Workspace Settings > Audit Logs. There you can filter by Deployment.created event, and see if there are any relevant information that could help you resolve this.
medim
Hey, can you try a Cntrl + K -> Deploy latest commit, with the affected service pane open? This normally fixes this issue.
11 days ago
So it was working fine for a bit after I disconnected the repo, published changes and then reconnected, but now it's back intermittently. It'll pick up on some changes, and just not see other changes. I did Cmd + K and I can get it to deploy which is great - didn't realize there was a shortcut for that. But the GitHub webhook issue is still a live one.