Problem Summary

I'm using Railway's official Grafana Stack template to monitor my Django application. Despite having all configurations correct, Prometheus cannot connect to my Django app via Railway's internal networking, even though the Django app is properly configured and accessible publicly.

Current Setup

• Django app service: Custom Django app with django-prometheus (PORT=8000)

• Grafana Stack: Deployed from Railway's official template (forked prometheus config only)

• Django prometheus: Using django-prometheus library with proper middleware and URL configuration

What Works

• Django app serves metrics correctly at /metrics endpoint

• Public domain access works: metrics endpoint returns proper Prometheus format data

• Django logs show successful internal requests: 100.64.0.x - "GET /metrics HTTP/1.1" 200

• Grafana Stack services are all running and interconnected

• DNS resolution works: myapp.railway.internal resolves to IPv6 address

What Doesn't Work

• Prometheus cannot scrape Django via internal networking

• Getting connection refused errors despite correct configuration

Error Details

Error scraping target: Get "http://myapp.railway.internal:8000/metrics": 
dial tcp [fd12:68d8:a48:0:2000:18:206b:8ce4]:8000: connect: connection refused

Key observation: DNS resolution works (resolves to IPv6), but connection is refused on port 8000.

Django Configuration (CONFIRMED WORKING)

# settings.py - ALLOWED_HOSTS includes internal domain
ALLOWED_HOSTS = [
    'localhost',
    '127.0.0.1',
    'myapp.up.railway.app',
    'myapp.railway.internal',  # Added for internal networking
    'custom-domain.com',
]

Django startup logs confirm correct binding:

[INFO] Listening at: http://0.0.0.0:8000 (1)
[INFO] Using worker: uvicorn.workers.UvicornWorker

Django is listening on all interfaces (0.0.0.0) and port 8000 as expected.

Prometheus Configuration

# prometheus.yml (in forked repository)
global:
  scrape_interval: 15s
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
  
  - job_name: 'django-app'
    static_configs:
      - targets: ['myapp.railway.internal:8000']
    metrics_path: '/metrics'
    scrape_interval: 15s

Environment variables in Prometheus service:

PORT=9090
DJANGO_METRICS_URL=myapp.railway.internal:8000/metrics
PROMETHEUS_ENABLE_SERVICE_DISCOVERY=true

Troubleshooting Steps Attempted

1. Different Internal Domain Formats Tested

• myapp.railway.internal:8000

• django-app.railway.internal:8000

• django-app:8000

• myapp.railway.internal (without port - tries port 80, also fails)

2. Public Domain (Workaround)

Using public domain works but returns 403 Forbidden due to CDN/proxy blocking Prometheus user-agent.

3. Django Configuration Verification

• ALLOWED_HOSTS includes internal domain

• django-prometheus middleware properly configured

• /metrics endpoint accessible and returns valid data

• PORT=8000 environment variable set

• Gunicorn bound to 0.0.0.0:8000 (confirmed in logs)

4. Network Connectivity Tests

• DNS resolution works (internal domain resolves to IPv6)

• Same Railway project (all services deployed together)

• Port 8000 connection refused despite Django listening on all interfaces

Technical Details

• Django service private domain: [service-name].railway.internal (anonymized for privacy)

• Django runs ASGI: Gunicorn with UvicornWorker bound to 0.0.0.0:8000

• Metrics endpoint: Properly configured with django-prometheus

• Same project: All services in same Railway project

• Template: Official Railway Grafana Stack, only prometheus config forked

Analysis

This appears to be a Railway internal networking issue rather than a configuration problem:

1. Django is correctly configured - listening on all interfaces, ALLOWED_HOSTS includes internal domain

2. DNS resolution works - Railway resolves the internal domain to IPv6

3. Public access works - the application and metrics endpoint function correctly

4. Connection refused on port 8000 - despite Django explicitly listening on that port

Questions for Railway Team

1. Are there restrictions on internal networking between Grafana Stack services and custom applications?

2. Should internal service communication work on custom ports like 8000?

3. Is there additional configuration required for internal service discovery to work with forked configurations?

4. Could this be related to IPv6 networking or firewall rules within Railway's internal network?

Any insights or suggestions would be greatly appreciated!