Cannot resolve private networking between two containers

charlesfr
PRO

23 days ago

I have Prometheus and a Celery Exporter service running in two seperate Docker containers in the same project, both running healthy without errors.

But Prometheus isn't able to connect the exporter.

here's the prometheus.yaml config:

global:
  scrape_interval:     15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']
  - job_name: celery-exporter
    static_configs:
      - targets: ['celery-exporter.railway.internal:9812']

What am I missing?

If I open up a public address to the exporter (https://[redacted].up.railway.app/metrics) then it works as expected and returns metrics.

Celery Exporter logs:

Starting Container
Starting celery-exporter on port 9812 with broker redis://default:[redacted]@redis.railway.internal:6379
2025-06-01 10:43:44.142 | INFO     | src.http_server:start_http_server:66 - Started celery-exporter at host='0.0.0.0' on port='9812'

0 Replies

charlesfr
PRO

23 days ago

f1b5f347-c845-4e49-a7a2-c861816cffd6

charlesfr
PRO

23 days ago

Cannot resolve private networking between two containers

erisa
HOBBYTop 10% Contributor

23 days ago

the celery exporter likely isn't listening on ipv6

erisa
HOBBYTop 10% Contributor

23 days ago

try adding the --listen-address '[::]:9812' flag to its start command

erisa
HOBBYTop 10% Contributor

23 days ago

looking further it seems you can also set that as CELERY_EXPORTER_LISTEN_ADDRESS instead of the flag

charlesfr
PRO

23 days ago

I've set the env to:

CELERY_EXPORTER_LISTEN_ADDRESS='[::]:9812'

And added the flag to the start scripts in case it wasn't getting picked up:

#!/bin/bash
set -e

# Default to redis://redis:6379/0 if not set
: "${CELERY_BROKER:=redis://redis:6379/0}"

echo "Starting celery-exporter on port 9812 with broker $CELERY_BROKER"
exec /usr/local/bin/celery-exporter --broker-url="${CELERY_BROKER}" --port=9812 --listen-address '[::]:9812'

but the target still isn't being detected:

Error scraping target: Get "http://celery-exporter:9812/metrics": dial tcp [fd12:642b:c0e2:0:9000:16:1ced:5459]:9812: connect: connection refused

lmk if there's anything else I can try - thanks!

charlesfr
PRO

23 days ago

I don't think listen address is valid either

1378731457375109260

charlesfr
PRO

23 days ago

I swapped for this and it works - thanks for the pointer!

exec /usr/local/bin/celery-exporter --broker-url="${CELERY_BROKER}" --port=9812 --host=::

erisa
HOBBYTop 10% Contributor

23 days ago

ahh okay sorry about confusion i guess i read the wrong page but I'm glad you got it working!