## Summary

Internal PostgreSQL URL postgres.railway.internal:5432) was completely unreachable for ~30+ minutes, causing all API requests to timeout. Public proxy URL worked fine. After the issue self-resolved, internal URL latency is 8-10x slower than public URL.

## Environment

- Project ID:9e652f25-f137-48d5-b587-5f1057f75ac6

- Project Name:firebackup.io

- Environment:production

- Affected Services:api, backup-worker, pitr-worker

- Database: Railway-managed PostgreSQL

- Region: (Railway default)

## Issue Timeline (January 6, 2026)

### 14:21 UTC - Issue Started

- API requests started timing out after exactly 30 seconds

- Health endpoint /health) returning 503 - database health check timing out

- All endpoints requiring database access failed

- Logs showed: [PrismaService] Disconnecting from PostgreSQL database...

### 14:21 - 15:07 UTC - Investigation

- Confirmed database was reachable via public proxy URLtrolley.proxy.rlwy.net:21973)

- Local connection test to public URL: 610ms, successful

- Internal URL postgres.railway.internal:5432) was completely unreachable from the API service

### 15:07 UTC - Workaround Applied

- Changed DATABASE_URL from internal to public URL

- Service immediately recovered

- Health check: 4ms latency, healthy

### 15:28 UTC - Internal URL Recovered

- Created diagnostic endpoint to test both URLs

- Internal URL now working but significantly slower:

- Internal: 223-238ms

- Public: 2-30ms

## Diagnostic Evidence

### During Outage

```

# Health endpoint (using internal URL)

HTTP 503 - Service Unavailable

Response time: 5.3 seconds (timeout)

Database health check: FAILED - timed out after 5000ms

```

### After Switching to Public URL

```

# Health endpoint (using public URL)

HTTP 200 - OK

Response time: 0.4 seconds

Database: healthy, latency 4ms

```

### Current State (Both URLs Working)

```json

{

"internalUrl": {

"masked": "postgresql://postgres.railway.internal:5432/railway",

"status": "healthy",

"latencyMs": 238

},

"publicUrl": {

"masked": "postgresql://trolley.proxy.rlwy.net:21973/railway",

"status": "healthy",

"latencyMs": 30

}

}

```

## Expected vs Actual Behavior

| Aspect | Expected | Actual |

|--------|----------|--------|

| Internal URL availability | Always available | Was completely down for 30+ minutes |

| Internal URL latency | Lower than public (same network) | 8-10x slower than public URL |

| Service recovery | Automatic | Required manual switch to public URL |

## Questions

1. What caused the internal networking outage? Was there maintenance or an infrastructure issue around 14:21 UTC?

2. Why is internal URL latency higher than public? Internal networking should be faster, not slower. Is there a DNS resolution or routing issue?

3. How can we be notified of internal networking issues? We had no visibility into this problem until users reported timeouts.

4. Is the public proxy URL recommended for production? Given the reliability issues with internal URLs, should we continue using the public proxy?

## Diagnostic Endpoint

We've created a diagnostic endpoint for ongoing monitoring:

```

GET https://api.firebackup.io/health-internal

```

This tests both internal and public database URLs and reports latency/status for each.

## Current Workaround

We've switched all services to use the public proxy URL:

```

DATABASE_URL=postgresql://postgres:***@trolley.proxy.rlwy.net:21973/railway

```

The internal URL is configured separately for monitoring:

```

DATABASE_INTERNAL_URL=postgresql://postgres:***@postgres.railway.internal:5432/railway

```

## Impact

- Duration: ~45 minutes of degraded service

- User Impact: All authenticated API requests failed

- Services Affected: API, backup-worker, pitr-worker

- Data Loss: None (database itself was healthy)

---

Submitted by:Firebackup.io Team

Date: January 6, 2026

Priority: High (production outage)