20 days ago
Dear Railway Support,
I'm experiencing critical issues with the Railway PostgreSQL TCP proxy that are preventing large-scale data ingestion operations from completing successfully.
Issue Summary:
- PostgreSQL proxy:
caboose.proxy.rlwy.net:30164 - Service: PostgreSQL in project
rdbenefits-dev(production environment) - Problem: TCP proxy is causing data corruption and SSL connection drops during data ingestion
Symptoms:
-
Data Corruption:
- Downloaded files are corrupted during transfer through the proxy
- Error: "Compressed file ended before the end-of-stream marker was reached"
- Files download successfully but fail gzip validation
- Corruption occurs consistently across multiple files (hundreds of MB to GB size)
-
SSL Connection Drops:
- Frequent SSL connection drops during database operations
- Error: "SSL connection has been closed unexpectedly"
- Connection drops occur even with reduced concurrency (4 workers)
- Retry mechanism is overwhelmed by connection failures
-
Performance Impact:
- Job processing time increased from expected 12-18 hours to 24-36 hours
- Only ~2.7% progress after 27 minutes due to corruption and retries
- Frequent transaction rollbacks due to connection failures
Technical Details:
- Workload: data ingestion for my apps
- Data downloaded locally to SDD
- Data imported to my Postgress service running on Railway
- Local data purged
- File sizes: 100 MB to 11 GB compressed JSON files
- Concurrency: Reduced from 16 → 12 → 8 → 4 workers to mitigate issues
- Connection pooling: SQLAlchemy with pool_recycle=280s to prevent idle timeouts
- Retry mechanism: Exponential backoff with transaction rollback on errors
Current Configuration:
- Proxy URL:
postgresql://postgres:***@caboose.proxy.rlwy.net:30164/railway - Private hostname:
postgres-8ndh.railway.internal:5432(not accessible from local) - Proxy variables:
RAILWAY_TCP_PROXY_DOMAIN,RAILWAY_TCP_PROXY_PORT,RAILWAY_TCP_APPLICATION_PORT
Attempted Mitigations:
- Reduced worker concurrency (16 → 4)
- Added connection recycling (pool_recycle=280s)
- Increased PostgreSQL max_wal_size (attempted, but service recreation required)
- Retry mechanism with exponential backoff
Current Hardware:
- Mac Studio with a Apple M2 Ultra chipset
- 196GB of shared RAM
- 76 GPU cores
- 8Tb external SDD with fast USB C connection.
Questions:
- Can the Railway TCP proxy be configured to handle large file transfers without corruption?
- Are there proxy timeout or SSL settings that can be adjusted for this workload?
- Is there a way to bypass the proxy for external access (VPN, direct connection, etc.)?
- Are there known issues with the TCP proxy for high-throughput data ingestion workloads?
- Would deploying the ingestion job inside Railway (using private network) resolve these issues?
Request:
Please investigate the TCP proxy configuration for this PostgreSQL service and provide guidance on either:
- Fixing the proxy corruption/SSL issues, or
- Providing an alternative connection method that bypasses the unstable proxy
This is blocking critical data ingestion operations for some data sets I need for my applications. The proxy appears to be struggling with this workload type.
I can provide detailed logs showing the corruption patterns and SSL connection drops if needed.
Thank you and best regards,
Roger
1 Replies
20 days ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • 20 days ago
14 days ago
Hi Roger,
It sounds like you are hitting a known bottleneck with high-throughput ingestion over public TCP proxies. When moving multi-gigabyte files, public-facing TCP proxies are prone to packet loss, timeout variations, and MTU (Maximum Transmission Unit) fragmentation, which explains the "unexpectedly closed" SSL connections and gzip stream corruption you're seeing.
To answer your specific questions:
Can the proxy be configured? Generally, no. The TCP proxy is a shared infrastructure component meant for intermittent connections, not sustained, high-throughput ETL (Extract, Transform, Load) tasks. There are no user-accessible "knobs" to adjust its internal timeout or SSL buffer behavior.
Is there a way to bypass it? Yes, this is the most critical part of your solution. Because the proxy is the source of the instability, you want to eliminate the "middleman."
Would moving the job inside Railway help? Yes, absolutely. This is the recommended architectural pattern for your workload. By running your ingestion job inside the same Railway project (as a separate Service or a Cron Job), you can communicate with your PostgreSQL service via the Private Networking hostname (postgres-8ndh.railway.internal:5432).
Recommended Path Forward:
Move the compute to the data: Stop attempting to stream 11GB files over the public internet. Instead, package your ingestion script into a simple container (Docker-based) and deploy it as a "Service" or "Cron Job" within your Railway project.
Utilize Internal Networking: By using the .railway.internal hostname, your traffic never touches the public TCP proxy. It stays entirely within Railway’s internal software-defined network. This eliminates the proxy as a point of failure and provides a much higher, stable throughput with zero SSL/TCP fragmentation issues.
Data Staging: If you need to get the files into that environment, upload them to an S3-compatible bucket (like R2, AWS S3, or a Railway Volume if persistent storage is required) and have your internal service pull from there.