2 months ago
Hi team,
We are currently investigating a bug in our queue service where two workers are connecting simultaneously instead of one. Both connections originate from the project's private network.
We have verified our configuration multiple times and confirmed that our project is set up to use only a single connection. This was further validated by duplicating the environment and observing that the duplicate instance shows only one connection, meaning the double connections are isolated to production.
This strongly suggests that a previous instance may still be running in the background, a "ghost" worker left over from a blue-green deployment that was never properly terminated.
That's a chance of this been causing the issues on our end?
11 Replies
2 months ago
The shadow container is located at fd12:260e:d55f:0:8000:bc:d624:f590. We have attempted to drop the connections manually, but the server has auto-reconnect enabled and keeps re-establishing them. We currently have no terminal access to the container to stop the process directly.
2 months ago
Heyo, did you delete the linked service above?
2 months ago
looking at your project there is only one queue service; looks like it was made around an hour ago and there is only one running container attached
2 months ago
Hi
2 months ago
Yes, I deleted it. I thought that deleting and recreating it might solve the problem, but it didn't work.
2 months ago
I'll do some digging but I dont think there would be a shadow container still running on our end
2 months ago
We list and monitor all connections to Redis, and this address fd12:260e:d55f:0:8000:bc:d624:f590 has been there for days. When we recreated the service, we saw that the new service connected to Redis with a different IP, but this one remained. This IP appears to be a private IP from our own project's private network.
2 months ago
Alright! I did some digging and confirmed there was a ghost container on the deleted queue service.
It should be shut down now; just watched from our side and there's no more network traffic from that IP
2 months ago
Sorry about that, ideally these don't happen and this is actually only the second one I've ever seen.
2 months ago
Confirmed also that is gone on our end. Thx for the help!!
2 months ago
Happy to help!
Please reach out if you continue having issues here
Status changed to Solved mykal • 2 months ago