One important notice is that we have "Always restart" policy

And when the new deployment kicks in, it seems not to shut down the previous one immediately – it feels like it hangs on for random amount of time

here's the twitter issue i mentioned at the start

although the deployment was like several ahours prior to the start of that long red-blue candle dance, it seems to me that previous failing deployment decided to boot up at 4 am

or was never off

service 0b9228e6-0528-4f63-bc97-157a8909cc2b

project id, service id, and environment please

project id 3e6a2b9c-4e34-41f8-979a-b83b27f3198d

environment mainnet

it's for this one

for this one,

project id 3e6a2b9c-4e34-41f8-979a-b83b27f3198d

service 0e389c2b-a588-46d2-a58e-b4b76fce4613

environment testnet

im sorry but the issue is not clear, you opened this thread and reported issues with two services?

since it has occured two times with different services and different environments (exactly where i pushed new code and caused new deployments to roll out), it made me think something is wrong with railway

yes, you can check any

same issue for both services?

most likely – previous deployment kept running with the new (current) one for too long before stopping

for this case this was literally hours

for this case this was 7 minutes

the twitter service literally worked without no flaws for the last two weeks, no twitter integration part was changed

i pushed a new commit to tune the ai prompt and here we are

you are giving a lot of unorganized information all at once here

ok, guide me, what do you need exactly to investigate the issue further?

lets focus on one service at a time, what service would you like me to look into first?

let's look at this one

okay, can you provide a full UTC timestamp of when you made a new deployment, and the old deployment didnt get killed?

At Dec 16 21:56 UTC I have pushed the commit

1734386160

At Dec 17 02:00 UTC our service started failing – this is quite common in case if our twitter service has a load and has to fight rate limits. there was no load.

the specific details I provided above about twitter are important since twitter api allows only one stream consumer at a time.

it was throwing "Too Many Connections" error, flagging that someone else was consuming the stream – supposingly another replica

i can see very spotty metrics during that time, indicating it crash looping

but i also see only a single deployment running during that time, at least for the given service and environment id

yes – this is something I architected on purpose so it fails occasionally to bear through rate limits

well then I'd suggest to check the second case

will do

with this details

same time stamp?

Dec 17 8:48PM - pushed first commit to fix the issue

Dec 17 8:52PM - pushed the "last fix" 🙂

the previous container kept running for at least another 4 minutes (if 8:55PM is the actual time of the "last fix" deployment container start) (i'm in gmt+2)

so the previous deployment kept running for 4 minutes until stopping, although a new one appeared

why did it take it so long to stop? – we also have "Always" as a restart policy here

utc timestamps please

1734468925 - new container started

1734469175 - old container stopped

it just looks as if there's a race condition between restarting a container and removing it

sorry that i have to say this, im not a robot, please give me human read-able timestamps

okay please refer to which kind of timestamps are you talking about, UNIX ones?

no worries i'm not a robot too beep-bop

human readable, like the ones in the screenshot logs, but UTC please

those are UTC, i deducted the local time difference which is two hours as you can see in the screenshot

8:55:25 PM - started new container

8:59:35 PM - stopped previous container

what service is this in what environment

this one

okay i think i see the issue here

you had this deploy working and online

and then you pushed bad code multiple times, the new code pushes never passed their health checks, so the working code was never taken offline.

this is by design, so the system is working properly here.

hmm gotchu ty

and yeah for the first service, I see the metrics for one deployment end, and the metrics for a new deployment start, but they do not overlap

!s