2 months ago
I've been trying to set up a prometheus collector for a Redpanda cluster that can be accessed via public HTTPS, but I keep getting TCP_INVALID_SYN errors. The Railway agent told me this:
This is a Railway infrastructure issue, not a config problem. The pattern is clear:
srcAddr: 3.136.56.55 — Railway's egress IP (consistent across all errors)
dstAddr: 10.x.x.x — Internal Railway network address (different each time)
dropCause: TCP_INVALID_SYN — Railway's edge is rejecting the outbound connection
The problem: Your collector's outbound traffic is being routed through Railway's internal network instead of the public internet, even though you're trying to reach a public Redpanda endpoint.
This is a routing/NAT misconfiguration at Railway's edge. The connection is being intercepted and dropped before it reaches the actual Redpanda server.
Immediate actions:
Contact Railway support with these details:
Service ID: 12064de1-2476-40bb-b1d2-3b0810560790
Target endpoint: console-blah.byoc.prd.cloud.redpanda.com:443
Error: TCP_INVALID_SYN on outbound HTTPS to public endpoint
This appears to be a routing issue where egress traffic is being looped back through internal Railway IPs
Pinned Solution
2 months ago
the giveaway in your trace is src 3.136.56.55 (railway egress) going to a 10.x.x.x destination. youre trying to hit a public redpanda endpoint but the traffic is getting looped back through railways internal IPs instead of out to the internet, so the SYN gets dropped (TCP_INVALID_SYN). thats a routing/NAT thing on railways edge, not your config, so it's worth escalating to them with that exact src/dst pair. on your side, first rule out the boring stuff: nslookup the redpanda host from inside the container to confirm it resolves to a PUBLIC ip (not 10.x), and curl -v https://console-blah...:443. if dns is handing you an internal address thats your loop. also make sure your CA certs/schema registry TLS are set so it's not masking a cert failure as a connection one.
3 Replies
2 months ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • 2 months ago
2 months ago
Looking at the code structure, when Console tries to connect to the Schema Registry endpoint at port 443, these are the likely errors:
Error 1: Connection Refused (Endpoint Doesn't Exist)
Code:
dial tcp console-blah.byoc.prd.cloud.redpanda.com:443: connection refused
Cause: The domain doesn't resolve or the endpoint is not available
Location: backend/pkg/console/schema_registry.go:68-70 (GetSchemaRegistryClient call)
Error 2: TLS Certificate Verification Failed
Code:
x509: certificate signed by unknown authority
Cause: The CA certificate path is not configured or is invalid
Location: backend/pkg/tls/ca.go:52-65 (VerifyConnection)
Error 3: Dial Timeout
Code:
dial tcp console-blah.byoc.prd.cloud.redpanda.com:443: i/o timeout
Cause: Network unreachable or firewall blocking port 443
Location: backend/pkg/config/schema.go (Schema Registry configuration)
Solution:
Update your Console configuration with proper Schema Registry settings:
config.yaml
schemaRegistry:
enabled: true
urls:
- "https://console-blah.byoc.prd.cloud.redpanda.com:443" # or just the domainauthentication:
# If SASL/basic auth needed:
basicAuth:
username: "your-username"
password: "your-password"tls:
enabled: true
caFilepath: "/path/to/ca.crt" # Path to CA certificate
certFilepath: "/path/to/client.crt" # If mTLS needed
keyFilepath: "/path/to/client.key" # If mTLS needed
insecureSkipTlsVerify: false # Set to true only for testingor check these diagnostics:
Test DNS resolution
nslookup console-blah.byoc.prd.cloud.redpanda.com
Test connectivity
curl -v https://console-blah.byoc.prd.cloud.redpanda.com:443
Verify certificate chain
openssl s_client -connect console-blah.byoc.prd.cloud.redpanda.com:443 -showcerts
The issue is typically a network/firewall configuration problem or missing TLS CA certificate configuration in your Console deployment.
2 months ago
the giveaway in your trace is src 3.136.56.55 (railway egress) going to a 10.x.x.x destination. youre trying to hit a public redpanda endpoint but the traffic is getting looped back through railways internal IPs instead of out to the internet, so the SYN gets dropped (TCP_INVALID_SYN). thats a routing/NAT thing on railways edge, not your config, so it's worth escalating to them with that exact src/dst pair. on your side, first rule out the boring stuff: nslookup the redpanda host from inside the container to confirm it resolves to a PUBLIC ip (not 10.x), and curl -v https://console-blah...:443. if dns is handing you an internal address thats your loop. also make sure your CA certs/schema registry TLS are set so it's not masking a cert failure as a connection one.
2 months ago
I tried to make this private at first, but the bot marked it public for me. Anywho, it seems like the problem resolved itself by the time I went through nslookup steps! Thanks for the pointers!
Status changed to Solved passos • 2 months ago