dba1dc0b-0430-45f8-8120-7e2db2c58ed3

I went and tried using a simple websocket echo server in my latest commit

```// echo-server.js
const WebSocket = require("ws");

const port = 4001;

const server = new WebSocket.Server({ host: "0.0.0.0", port }, () => {
console.log(Echo server listening on port ${port});
});

server.on("connection", (ws) => {
console.log("Client connected");
ws.on("message", (message) => {
console.log(Received: ${message});
ws.send(Echo: ${message});
});
});

server.on("error", (err) => {
console.error("Server error:", err);
});

and instead I now get 

curl -v --http1.1 --header "Connection: Upgrade" --header "Upgrade: websocket" https://metro.proxy.rlwy.net:49766

• Trying 35.212.99.204:49766…

• Connected to metro.proxy.rlwy.net (35.212.99.204) port 49766

• ALPN: curl offers http/1.1

• (304) (OUT), TLS handshake, Client hello (1):

• CAfile: /etc/ssl/cert.pem

• CApath: none

• Recv failure: Connection reset by peer

• LibreSSL/3.3.6: error:02FFF036:system library:func(4095):Connection reset by peer

• Closing connection
  curl: (35) Recv failure: Connection reset by peer
  ```

at least with IPFS it was hanging and ws:// was giving a 302 redirect. ChatGPT has the following to say

I went ahead and created a TLS backend for websockets anyway

// echo-server.js
const https = require("https");
const fs = require("fs");
const WebSocket = require("ws");

const port = 4001;

// Read your TLS certificate and key
const serverOptions = {
    key: fs.readFileSync("key.pem"),
    cert: fs.readFileSync("cert.pem"),
};

// Create an HTTPS server using the certificate and key
const httpsServer = https.createServer(serverOptions);

// Create the WebSocket server, binding it to the HTTPS server
const wss = new WebSocket.Server({ server: httpsServer });

httpsServer.listen(port, "0.0.0.0", () => {
    console.log(`TLS Echo server listening on port ${port}`);
});

wss.on("connection", (ws) => {
    console.log("Client connected");
    ws.on("message", (message) => {
        console.log(`Received: ${message}`);
        ws.send(`Echo: ${message}`);
    });
});

wss.on("error", (err) => {
    console.error("Server error:", err);
});

still getting the same errors

Starting Container
Starting Echo Server for testing...
TLS Echo server listening on port 4001

then when running

wscat -c wss://[metro.proxy.rlwy.net:49766](metro.proxy.rlwy.net:49766)

i get error: read ECONNRESET

and the same for wscat -c ws://[metro.proxy.rlwy.net:49766](metro.proxy.rlwy.net:49766)

error: read ECONNRESET

lastly

nc -zv metro.proxy.rlwy.net 49766

Connection to metro.proxy.rlwy.net port 49766 [tcp/*] succeeded!

Seeing this on reddit but unsure how true

So I ended up doing the following

// echo-server.js
const http = require("http");
const WebSocket = require("ws");

// Use Railway’s assigned port or default to 80.
const port = process.env.PORT || 80;

// Create an HTTP server that can also handle upgrade requests.
const server = http.createServer((req, res) => {
    res.writeHead(200, { "Content-Type": "text/plain" });
    res.end("Hello, this is an HTTP server that supports WebSocket upgrades.");
});

server.listen(port, "0.0.0.0", () => {
    console.log(`HTTP/WebSocket server listening on port ${port}`);
});

// Bind the WebSocket server to the same HTTP server.
const wss = new WebSocket.Server({ server });

wss.on("connection", (ws) => {
    console.log("Client connected");
    ws.on("message", (message) => {
        console.log(`Received: ${message}`);
        ws.send(`Echo: ${message}`);
    });
});

wss.on("error", (err) => {
    console.error("Server error:", err);
});

and then did

wscat -c wss://jupyter-notebooks-production.up.railway.app
Connected (press CTRL+C to quit)

which worked

however I'm not exactly sure how this translates to IPFS since it needs to listen on port 4001, which is different from the default port

Is this correct? As I'm trying to figure out how would I do this for my IPFS node which is listening on 4001

So I guess is this something Railway can fix? (TCP port mapping correctly handling TLS handshakes for websocket connections)

After more research it seems that Railway automatically assigns 443 to the websocket endpoint and that there is only support for one externally assigned port via TCP proxy which is randomly assigned and one cannot pick.

the tcp proxy is just forwarding bytes, so no issues on our side there.

thanks Brody, this thread became a live debug thread haha, is more than one tcp proxy allowed?

only 1 TCP proxy per service

damn, are there any plans to add more?

there are no current plans, you would need to open a feedback post -

One more question here, do you know if the same port supports ws upgrade request?

of course it does

Does this sound right to you? (I don't know much about networking or how it's handled generally)

is this chatgpt?

yes it is, I don't know enough about networking to know if this is true but claude also affirms it

Are you able to confirm if HTTP upgrade handshakes are supported by the TCP proxy?

its a layer 4 proxy, it has no knowledge of what is done at layer 7 (http/ws)

meaning, as long as your application handles the upgrade, the behavior of the tcp proxy does not come into play

regardless, you should be using domains for this, the tcp proxy has no ssl / tls

hmm not sure what you mean w.r.t using domains for this

dont use the tcp proxy

since the tcp proxy does provide a domain no?

well the issue is that ipfs advertises its websocket on 4001

it is, im talking about https domains

so? point the domain to that port

so that's what I have been doing

you are using the tcp proxy still

ah so you're saying to use the public domain that's not the tcp proxy domain?

correct

yeah unfortunately it needs to have 8888 open on the inbound as well (hence needing to create the tcp proxy in the first place)

because that's for jupyter lab to be accessible

https://jupyter-notebooks-production.up.railway.app/lab?

a service can have multiple domains

indeed, adding a custom one, but is there any way that Railway can offer this much like the TCP proxy domain?

im not seeing why tcp proxies are needed here, you are not doing anything with tcp

ipfs advertises its websockets this way

{
  "API": "/ip4/127.0.0.1/tcp/5001",
  "Announce": [
    "/dns4/metro.proxy.rlwy.net/tcp/49766/tls/sni/metro.proxy.rlwy.net/ipfs/12D3KooWLXuf6mMsyAcHaX6gYzU74jsaBaTxpbzdjDMNKYj2Wj1A",
    "/dns4/metro.proxy.rlwy.net/tcp/49766/tls/sni/metro.proxy.rlwy.net/ws/p2p/12D3KooWLXuf6mMsyAcHaX6gYzU74jsaBaTxpbzdjDMNKYj2Wj1A"
  ],
  "AppendAnnounce": [],
  "Gateway": "/ip4/127.0.0.1/tcp/8080",
  "NoAnnounce": [],
  "Swarm": [
    "/ip4/0.0.0.0/tcp/4001",
    "/ip6/::/tcp/4001",
    "/ip4/0.0.0.0/udp/4001/webrtc-direct",
    "/ip4/0.0.0.0/udp/4001/quic-v1",
    "/ip4/0.0.0.0/udp/4001/quic-v1/webtransport",
    "/ip6/::/udp/4001/webrtc-direct",
    "/ip6/::/udp/4001/quic-v1",
    "/ip6/::/udp/4001/quic-v1/webtransport",
    "/ip4/0.0.0.0/tcp/4001/ws"
  ]
}

im not seeing why a tcp proxy is needed

What would your suggestion be? A custom domain would have to be added manually otherwise no?

to map to the port since only 1 inbound port is supported at the moment on railway

so in the case of launching this combined application on railway where 8888 and 4001 need to both be open to inbounds (the former for jupyterlab and the latter for TLS requests for the websockets) two domains need to exist

so far so good?

two ports = two domains

yep exactly

so whats stopping you?

ok so we're on the same page (and I'm following what you're saying then)

my question then was could railway autoset that up the same way it sets up the domain for the TCP proxy

(the reason I brought up the tcp proxy is because it also spins up a domain automatically for that)

just add two custom domains and point them to the correct port

it's not that easy

yes it is

for some extra background people will be launching this template

and in an ideal world there will be hundreds of these launches

getting a domain and setting up DNS records is not easy for the average person

get it working outside of the template and then i can help with the template later

so what the IPFS team did was they created an ability for autotls

so it's a template issue moreso than anything else

because AutoTLS already solves this problem

not yet, you have yet to get it working

I'm not sure what you mean?

^

so if I did this though I don't understand how other people could use this without needing to setup their own custom domains and setting up DNS records to confirm ownership?

do not worry about that right now, you need to get it working first, one step at a time

It's more I don't want to do that and then have this as an issue

lol

so it's possible where others won't need to setup their own custom domains if I do this?

as right now I will have to setup my own custom domain, I want to avoid other people deploying the template from having to do that

it might be required

for additional context the ipfs team knows people won't do this, but having websocket connectivity is critical to the network being resilient so they added something called autotls https://blog.libp2p.io/autotls/ which auto sets up a domain in the background and then relays it back to the node, I can't use the AutoTLS functionality within Railway because the inbounds are blocked

lets focus on getting it working in your own project please

I've gotten this to work on my own on a few different deployments, having done the custom domain in the past on different platforms & deployments. If the users need to have a custom domain it's dead in the water

and I rather deploy it on digitalocean because autotls will work there

So there's no real point to pursue it with a custom domain here

I'll set up a domain anyway to confirm/test it

Ahh need SSL on this to be able to accurately test it, back to the TCP stuff (since I think there's a way to get this to work still)

However I just checked out http://metro.proxy.rlwy.net:49766/ and somehow this is wrapping back to my public domain which is mapped to 8888 … aka https://jupyter-notebooks-production.up.railway.app/ but the TCP proxy is mapped to 4001, I can't see how it would be opening up the jupyterlabs page

https://github.com/dClimate/jupyter-notebooks/blob/main/scripts/start.sh#L141

we handle ssl for you though

at least when using the service and custom domains

hmm I had added the custom domain and there was no ssl it's possible maybe it didn't generate in time. I can try again, as for the TCP PROXY above any reason why it's seemingly still pointing to 8888?

have you tried to redploy after adding / updating the tcp proxy?

yeah it's been set for a while now (many deploys in between)

what makes you believe its still pointing to 8888

hitting the url metro.proxy.rlwy.net:49766 shouldn't open any webpage/that of jupyterlab and 8888 is where jupyerlab is being exposed

as in theory it should only be proxying to [localhost](localhost):4001 (where localhost is the local service) right?

try removing and re-adding

did and redeployed

when opened tramway.proxy.rlwy.net:41298 goes to my service running on 8888 (instead of 4001 which is what it's pointing to), 8888 is my default port on my public domain which points to jupyterlab but 4001 shouldn't be going there and it doesn't on a local docker setup

can we please go back to service or custom domains please? again, you are not doing anything with tcp as far as i can tell

Yes I am, I'm trying to advertise using autotls on the ipfs backend. Setting a custom domain does nothing for me because I can already do that on another PaaS

And the whole reason to use railway is so that users won't need to setup a domain

If 4001 is not reachable via the tcp proxy then autotls cannot work

Autotls automatically issues a cert and domain so that websockets will work since seemingly railway doesn't support upgrading the connection autotls via libp2p will handle that

But instead of pointing to 4001 it points to 8888 then that's its own issue

railway's tcp proxy or any tcp proxy for that matter does not have anything to do with the websocket upgrade

though im not sure why you are focused on the tcp proxy, its not like you can add a tcp proxy and a service domain to a template service together

It's one more button for the user to press, add tcp proxy which would've done it for the user if this were to work. I still don't understand why my tcp proxy when mapped to 4001 is pointing to 8888

Yes sorry I misspoke I meant to say that the tcp proxy domain doesn't have certs issued so that the domain that's provided cannot be used for the secure websockets. Using autotls with ipfs they do handle that so the alternate for me was to use this tcp proxy as a way to bridge the internal 4001 to the outside so that autotls can handle the rest (issue cert and assign a domain)

But if 4001 is actually 8888 then this plan won't work

Debugging some of this in the meantime appreciate your assistance here @Brody will follow back up sometime soon. Last question in the meantime is if there are any plans for UDP proxies?

there are no current plans for UDP

Coming back to this now after a little detour, so the alternative I think I am arriving to is to put an Nginx Reverse Proxy in a separate service, which can get a Railway generated SSL domain, which then can route the secure websocket connection to the other service. Does this sound like a correct approach to obviate the above? If it is, what's the best way to dynamically set the routing within the nginx.config for the internal service?

I know there is a private networking domain but ideally I would be able to pass service A's internal domain into Nginx (service B) config without hardcoding. Could nginx simply read the service name and map to the private networking domain? I.e

http {
    # Set internal resolver (optional but can be helpful)
    resolver 127.0.0.11 8.8.8.8 valid=30s;

    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log /var/log/nginx/access.log main;
    # Use stderr for error logs to easily see them in Railway logs
    error_log /dev/stderr warn;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;
    sendfile on;
    keepalive_timeout 65;

    server {
        # Listen on the port specified by Railway ($PORT) on both IPv4 and IPv6
        # Railway forwards external port 443 (HTTPS) traffic to this internal port (HTTP)
        listen 80;          # Hardcoded IPv4
        listen [::]:80;     # Hardcoded IPv6
        server_name _; # Catch-all

        location / {
            # Define the internal upstream using Railway's service discovery DNS. AI Generated 
            set $upstream_ipfs http://jupyter-ipfs:4001; # Service name defined in railway.json
           ...

or does it need to be dynamically constructed at buildtime? If so, is there a way to make service b (nginx) wait until service a is up in order so that service a's internal domain name can be pulled?

Looking at the answer here https://station.railway.com/questions/make-service-wait-for-another-service-be-14f84c89 there's no way to wait until a service is up to pull its domain. So I guess the question is:

what is the way to get the private internal domain of a service dynamically, does the service name suffice?

you definitely don't want to use NGINX, use caddy, NGINX caches upstream DNS and the private network's DNS on Railway is not static.

you can use a reference variable to get the private domain of another service -

Ah nice so the namespace for example being Jupyter-lab. can be referenced and then access RAILWAYPRIVATEDOMAIN ? Or would it just be .DOMAIN?

it would be ${{Jupyter-lab.RAILWAY_PRIVATE_DOMAIN}}

Ah perfect so the service name can be used that's great. Phew.

yep!

Just wanted to check in and say this all worked, thanks for the Caddy suggestion too and all the other assistance @Brody as always!

/fix

No problem, also the TCP proxy pointed to that port because you had it set as a $PORT variable, doesn't matter what you have the TCP proxy itself set to, as long as you have a PORT variable, the TCP proxy will always point to $PORT