IPv6 Internal Networking Fails in PR Environments

Question

## **Issue Summary**

IPv6 private networking between services fails in PR environments but works correctly in persistent environments (development). Services cannot communicate using `service.railway.internal` or short hostname in PR environments, resulting in "connection refused" errors.

## **Environment Details**

* **Project ID**: `aa71fa2c-3f89-4ec1-ad45-2bfc9eaf1e41`
* **Project Name**: railwayapp
* **Working Environment**: development (ID: `1c5ada25-2d6a-441b-8d76-d52a8025909e`)
* **Failing Environment**: railwayapp-pr-5 (and all PR environments)

## **Services Involved**

1. **orkes** (ID: `9c50bba2-b7c2-4f9f-9d96-6f8a05979b35`)  
   * Internal port: 5000 (Nginx) → 8080 (Spring Boot)
2. **orkes-proxy** (ID: `41b9f469-f5ee-491c-a689-4d093b41077b`)  
   * GitHub source: `Naveo-Commerce-UK/railwayapp`, branch: main  
   * Uses Caddy to proxy to Orkes service

## **Problem Description**

The `orkes-proxy` service (running Caddy) cannot connect to the `orkes` service on port 5000 via Railway's private networking in PR environments. The same configuration works perfectly in the development environment.

### **What Works**

**Development environment:**

* Connection via `orkes.railway.internal:5000` succeeds
* Connection via `orkes:5000` succeeds
* IPv6 address resolution works: `[fd12:9ec4:c036:1:1000:95:94b4:1b8d]:5000`
* HTTP requests return 200 OK after authentication

**PR environment (public access test):**

* Public domain on port 5000 works correctly
* Orkes service is running and healthy
* Nginx is listening on port 5000 and serving content

### **What Fails**

**PR environment (internal networking):**

* Connection via `orkes.railway.internal:5000` fails with "connection refused"
* Connection via `orkes:5000` fails with "connection refused"
* IPv6 address resolution succeeds but connection fails: `[fd12:ac8b:e40a:0:1000:67:fa7e:72b0]:5000`
* HTTP requests return 502 Bad Gateway

## **Error Logs**

### **PR Environment - orkes-proxy deployment logs:**

```
dial tcp [fd12:ac8b:e40a:0:1000:67:fa7e:72b0]:5000: connect: connection refused
2026/02/20 10:30:09.293 ERROR http.log.access.log0 handled request
  "status": 502
  "user_id": "admin"
  "duration": 2.098843541

```

### **PR Environment - Railway HTTP logs:**

```json
{
  "requestId": "6AXaW1dwRcyNZpcW2JZdWA",
  "timestamp": "2026-02-20T10:30:09Z",
  "httpStatus": 502,
  "upstreamAddress": "http://[fd12:ac8b:e40a:0:4000:72:f217:cd77]:8080",
  "upstreamErrors": ""
}

```

### **Development Environment - Working logs:**

```json
{
  "requestId": "s3mFTqDZQ7-ocKSJrJsmnA",
  "timestamp": "2026-02-20T09:25:18Z",
  "httpStatus": 200,
  "upstreamAddress": "http://[fd12:9ec4:c036:1:1000:95:94b4:1b8d]:8080",
  "upstreamErrors": ""
}

```

Note: Development also occasionally shows IPv4 connection refused errors, but requests succeed overall.

## **Reproduction Steps**

1. Create a PR environment from branch with orkes-proxy configuration
2. Deploy both `orkes` and `orkes-proxy` services
3. Configure `orkes-proxy` Caddyfile to reverse proxy to `http://orkes:5000`
4. Attempt to access orkes-proxy via its public domain
5. Observe 502 errors in orkes-proxy logs with "connection refused" to IPv6 address

## **Configuration Files**

### **Caddyfile (orkes-proxy service)**

```caddyfile
:{$PORT:5000}

log {
    output stdout
    format console
}

# Health check — no auth
handle /health {
    respond 200
}

# Token endpoint — return fake JWT
handle /api/token {
    header Content-Type "application/json"
    respond `{"token":"..."}` 200
}

# API requests — strip JWT before proxying
handle /api/* {
    request_header -X-Authorization
    reverse_proxy http://orkes:5000
}

# UI requests — require HTTP Basic Auth
handle {
    basicauth {
        {$ORKES_PROXY_USER:admin} {$ORKES_PROXY_PASSWORD_HASH}
    }
    reverse_proxy http://orkes:5000
}

```

## **Testing Performed**

### **Test 1: Different hostname formats**

* `http://[orkes.railway.internal]:5000` \- connection refused (PR)
* `http://orkes.railway.internal:5000` \- connection refused (PR)
* `http://orkes:5000` \- connection refused (PR)
* `http://orkes:5000` \- works (Development)

### **Test 2: Different ports**

* `http://orkes:8080` \- connection refused (PR)
* `http://orkes:5000` \- connection refused (PR)

### **Test 3: Public domain (security test)**

* `https://orkes-railwayapp-pr-5.up.railway.app` (port 5000) - works
* Confirms Orkes service is running and Nginx is listening on port 5000

### **Test 4: IPv6 address format**

* `http://[orkes.railway.internal]:5000` \- connection refused (PR)
* Caddy still resolves to IPv6 and fails

## **Network Details**

### **Development environment:**

* Orkes IPv6: `fd12:9ec4:c036:1:1000:95:94b4:1b8d`
* Private domain: `orkes.railway.internal`
* Connection: Success

### **PR environment:**

* Orkes IPv6: `fd12:ac8b:e40a:0:1000:67:fa7e:72b0`
* Private domain: `orkes.railway.internal`
* Connection: Refused

darseen · Accepted Answer

If the Nginx server inside the `orkes` container is only configured to listen on IPv4, the container's network interface refuses the IPv6 connection. To fix this, update the Nginx configuration in your `orkes` service to listen on IPv6 by adding `listen [::]:5000;`