22 days ago
Hi team,
We are currently investigating a bug in our queue service where two workers are connecting simultaneously instead of one. Both connections originate from the project's private network.
We have verified our configuration multiple times and confirmed that our project is set up to use only a single connection. This was further validated by duplicating the environment and observing that the duplicate instance shows only one connection, meaning the double connections are isolated to production.
This strongly suggests that a previous instance may still be running in the background, a "ghost" worker left over from a blue-green deployment that was never properly terminated.
That's a chance of this been causing the issues on our end?
11 Replies
21 days ago
The shadow container is located at fd12:260e:d55f:0:8000:bc:d624:f590. We have attempted to drop the connections manually, but the server has auto-reconnect enabled and keeps re-establishing them. We currently have no terminal access to the container to stop the process directly.
21 days ago
Heyo, did you delete the linked service above?
21 days ago
looking at your project there is only one queue service; looks like it was made around an hour ago and there is only one running container attached
21 days ago
Hi
21 days ago
Yes, I deleted it. I thought that deleting and recreating it might solve the problem, but it didn't work.
21 days ago
I'll do some digging but I dont think there would be a shadow container still running on our end
21 days ago
We list and monitor all connections to Redis, and this address fd12:260e:d55f:0:8000:bc:d624:f590 has been there for days. When we recreated the service, we saw that the new service connected to Redis with a different IP, but this one remained. This IP appears to be a private IP from our own project's private network.
21 days ago
Alright! I did some digging and confirmed there was a ghost container on the deleted queue service.
It should be shut down now; just watched from our side and there's no more network traffic from that IP
21 days ago
Sorry about that, ideally these don't happen and this is actually only the second one I've ever seen.
21 days ago
Confirmed also that is gone on our end. Thx for the help!!
21 days ago
Happy to help!
Please reach out if you continue having issues here
Status changed to Solved mykal • 21 days ago