My project service is unable to connect to SurrealDB, it appears to be a wss (websockets secure) issue with railway

Another person in SurrealDB discord server is having this issue as well. I have made no changes to my configuration and my database is down for two different services

hey Albert, can you explain the setup for me. You have SurrealDB running at https://railway.app/project/a72c350e-c977-4440-95f8-1fc4d1c5fdd6.

You are connecting to it with [surrealdb-production-fae7.up.railway.app](surrealdb-production-fae7.up.railway.app) from clients correct ?

Yes. When using wss protocol, it fails to connect

https works fine

This was previously functional with no issues. Another user is experiencing the same issue with railway and running on a different database version

Are you available to check if it's working in a few minutes ?

Yes

ok

try now

Working! @Mig

What was the issue

I will have to troubleshoot it. We are moving everyone over to a new edge network and I just put your application back on the legacy network.

I still have one service down, oddly

(same project)

what's the issue exactly ?

Exact same connection error, I'll give it a minute to see if it might be some sort of pre-existing connection issue. I'll trigger a re-deploy for the project now

I've just changed DNS settings so existing connections will continue to talk to the previous network

we have migrated 400k applications to our new network and this is the first issue I've seen :/

Websockets can be a bitch

I suspect many other SurrealDB users are experiencing the same issue

I will try to repro right now

Okay, so my other service is online now after a redeploy

Approximately when did this migration take place?

the migration process has been ongoing for the past 2 weeks.

slowly moving apps over

ok, it was using connection to previous network

Do you know roughly when our project might've been migrated?

Just would like to know how much downtime there might've been.

it must've been a few days since I started having this issue on one service before the other one started showing issues (probably the existing connection)

I tried checking but I don't have enough data to know for sure.

I might be able to check 1 more place

@Mig is it normal to have different IPs for the host in the DNS? I'm assuming they're just edge routing or something?

https://dnschecker.org/#A/surrealdb-production-fae7.up.railway.app

the IPs resolve to different regional load balancers

so geodns is picking the closest LB

ok thanks just checking

I can confirm the date

just will take time as I grep the log file

If you'd like to reproduce my setup,

I was running

surrealdb/surrealdb:v1.4.2 as the docker image

/surreal start --auth --allow-guests --allow-funcs --deny-funcs "http" --allow-scripting --deny-net --no-banner as the custom start command

Region: US West (Oregon)

Runtime: Legacy but also tried V2 with same issue

The other person having the issue was on 1.5.2

To reiterate, this issue only occurred with wss connection within Surrealist GUI

Not with https

ok, your app was migrated excatly at 2024-08-26T02:32:51Z

utc

so about 48 hours

Yeah that sounds about right

close to 24 hours

I know websockets work

I'm sus about wss:// requiring TLS but our proxy doesn't do TLS to your app.

https:// worked though so maybe not that

ty for this!

@Mig are you saying data isn't encrypted during transit?

it is encrypted all the way to your app, just not using HTTPS.

internet <-- HTTPS --> Edge Network < -- WireGuard encrypted tunnel -- > Your App

did you configure tls certs on your app ?

hmm you might be right,

surreal start has the following option

HTTP server:
      --web-crt 
          Path to the certificate file for encrypted client connections

          [env: SURREAL_WEB_CRT=]

      --web-key 
          Path to the private key file for encrypted client connections

          [env: SURREAL_WEB_KEY=]

if so the edge network could be talking over HTTPS (and also wireguard)

most people don't have TLS terminating at their application

most is read as, I've never seen it

Well, I think TLS is enabled for https most definitely, but what about Websockets? How does websockets interact with proxies?

TLS would work for ws the same since that happens before the application (websocket/http) right after tcp

Anyhow, thank you for resolving the issue as fast as you could. I would investigate this as I imagine other users besides the one I spoke with is having this issue

yup, i'm working to reproduce right now

sorry you had this issue.

No problem - these things happen, fortunately I'm not running a critical service

When you are able, I would love an update on what occurred, whether you were able to reproduce/fix the issue with the new network

that drop down is in the surreal ui right that I should be able to visit just from the same service I am deploying on railway

I don't believe so

https://surrealist.app/

You can probably use the web GUI if its easier

otherwise downloading the application

oh I see. the service is just the DB and the screenshot is a frontend to it

That's right yeah

Let me know if you have any questions regarding connection setup, or whether you need me to test with my project once a fix is deployed

I used Root authentication method via the credentials generated by railway, any namespace and database should work for the connection

The URL should just be the host, and wss for the protocol

Hello.

I cannot connect to my deployed surrealdb instance on railway: wss://[surrealdb-production-c358.up.railway.app/rpc](surrealdb-production-c358.up.railway.app/rpc), even though it was working a couple of days ago.

Here is my projectID: 320c6ccd-0556-4d5f-9f87-05d51b1043ed

do you still have the issue if you use https:// protocol in the url instead of wss:// ?

I am aware of this surrealdb issue and wss://. Working to reproduce and fix.

i see no issues with the https:// protocol, only the wss://

Hello,
I've been experiencing the same issue. Any way to resolve this?
My ProjectID: 09be9421-29f2-4e16-b051-751c7fb2b16a

Make surrealdb client use https:// protocol instead of wss://. The connection will upgrade to websocket anyways.

the library which i use specifically wants a url in the format of wss:// or ws:// though. I tried for a bit but i don't think there's a good way around this…

how was this resolved for Albert and bachargit? is there something else I can try?

The issue is with the edge network being used by the application. We are moving on from our old proxy but I can put your app on it while we try to resolve this issue for surrealdb

yeah if that could at least temporarily resolve the issue i'd appreciate if you could do that

wss:// should work for you in a few seconds.

yeah it works, thank you for the help

Hey @Mig, can you do that for my app as well?

I started needing the wss:// protocol now

.

please do not ping team members, we are currently on a day off

this domain is now on the legacy proxy, should be updated soon

Hey @Albert @DrSmrtnik @NoBypass - We have moved all domains over to the new edge proxy, including the domains attached to the surrealdb databases.

awesome @Brody thanks for the update

sorry i should have been more clear, we have not patched away whatever surrealdb is doing incorrectly with websocket connections

It appears to be working fine for me @Brody ?

well you are on the v2 proxy, so thats awsome, are you using wss or https?

wss I believe but I could double check soon

actually it may be https but I am not too sure

I guess I'll find out when I migrate my new update using wss