Thank you for reporting, the team is aware and additional fixes are incoming.

Unfortunately, they will take some time to live. We are actively trying to resolve this issue.

Thanks very much for the quick response! I’ll keep an eye out for updates

Hello. Do you have an ETA for the fix? Thanks.

Still in progress, we are getting closer. We expect to be past the issues by EoW. Deploys are still going out, just taking longer.

Still in progress, we are getting closer. We expect to be past the issues by EoW. Deploys are still going out, just taking longer.

You expect deploys to be slow till end of week? Is there anything users can do mitigate the impact? It's making rapid development impossible, needing to wait anywhere from 1-10 minutes per build.

The issue is not that builds are slow all the time, but rather, load spikes at any given moment will make it so that builds that are usually 1 minute spike up to 10. At the moment, the team is prioritizing build finality over build speed which means that we would rather have the queue drain where everyone gets their deploy live rather than hang indefinitely because the person who re-kicks their build cuts in front of everyone on a faster box.

As we speak, things are fine, but we have core fixes that unfortunately require time. If there is a production use-case that needs a feature to go out yesterday, we have been working with customers to expedite that out but I would rather be honest with you instead of saying, "No ETA."

Thanks. I did notice some deploys take <1m and some take <10m so having an explanation as to what's happening makes it feel like this isn't an accident but rather a planned -- however unfortunate -- short term reality. Anyway, thanks for the communication.

Thanks for the explanation Angelo. I wanted to add that I'm seeing delays long than 10 mins as well. One of my deploys ran overnight and was stuck on "Deployment in progress: Taking a snapshot of the code" for several hours. Is this expected on hobby user accounts?

edit:

One additional note: the build's deploy logs indicate that the app has successfully started, however the build isn't moving on to the 'post-deploy' step.

(Adding a new comment as my edit isn't updating.)

One additional note: the build's deploy logs indicate that the app has successfully started, however the build isn't moving on to the 'post-deploy' step.

Yep, thanks for raising. To peel back, we have narrowed down the cause the issue and are running testing. The reason we aren't rolling out the fix to all hosts all at once is so we don't run into an issue where we choke all in-progress builds. Hopefully I am not assuming too much but I appreciate the patience and understand the frustration.

Even our own services (like Help Station) have been nailed by this and it's massively frustrating.

Thanks
Angelo