well one way to make sure nothing is connecting to it publicly would be to remove its tcp proxy, then before long something will crash

😆 true!

i can do this for our stg and dev servers

lets see how it goes

oh wait, i think i know why

i am using the internal ip but using the public port

ie

**.rlwy.net:56258 -> :6379

Private Networking
redis.railway.internal

and I am connecting to it via redis.railway.internal:6379 lol

the public port is 56258 in this case, 6379 being the private port

oh, right right

then thats not it

what you actually described redis.railway.internal:56258 would also just outright not work

yeah i didn't think it would, i was kinda suprised it was working at all with what i described!

but yeah seems like I have that the right way around

disabling the public TCP proxy now for dev/stg and observing/testing

seems fine so far… strange.. there is so much traffic on prd

prod is somehow connecting to it via the public network?

hm… i dont think so

but we will see

i disbled it for prd too and just observing again

but both stg and dev still show some traffic, minor, but still

will see what happens with prd

sounds good

Hmm, nope

its like the Redis traffic counts as egress..?

been about 10 mins

the traffic on the private network shouldn't be measured, check the cost breakdown to see if your redis service is incurring any egress

good idea

yeah it is <:blobconfused:618587926862757888>

and 'double the cost' i guess since the service talking to it is also taking the egress hit

and you have completely removed its tcp proxy?

yep!

is redis somehow sending data elsewhere? there's no tcp proxy but that doesn't disconnect it from the internet

I am not sure how i would check that. I just spun up redis using right click > database > redis, nothing bespoke

what are you using redis for?

various things - caching, temp store, used by my redis io adapter

each has their own, and they are the only ones communicating with their own redis db

lol

okay well now I have to go off on a tangent here

why service groups for your different environments instead of project environments?

hahaha i was debating whether i should put this up because i figured you would mention that haha

is there a cross-project egress cost? because my API Gateway would have to be in its own project

then the game servers in a nother, with an environment for each

but it means all comms will happen from API Gateway <- across projects -> Game Server

well you would have a single project, with multiple environments, each environment would have a gateway, game server and it's redis database

yeah but the gateway directs traffic to one of the 3 environments

so which is the 'true true' gateway, and are the other 2 just dummies that dont do anything, i guess? haha

if you where to go with the architecture I mentioned, your gateway would only route traffic to a production (or set of production) game server services in the production environment, and the gateway in the development environment would route traffic to the development game server in the same environment

it makes most sense to me if i could nest this setup inside one another.. something like

Main Project - contains Gateway                                                   | ->  Dev/Stg/Prd
   Game Server Project - contains game server/redis database. | ->  Dev/Stg/Prd   |

yeah but I need the single gateway to be able to route traffic to the Dev or Stg or Prd version of the server

no matter how you do it, you absolutely wouldn't want to be using multiple projects, and honestly with the group UI you have done it quite cleanly, was just curious is all

basically I'm stalling while trying to think of potential reasons for seeing such high outbound usage

it would be cool to be able to nest a project inside another, and then from the parent project I could have an API gateway to proxy the traffic to different environments of the sub project (dev/stg/prd)

and then I could spin up a dev version fo this WHOLE setup and test the entire stack

because right now I don't really have a 'stg' or 'dev' version of the my API gateway

-> 🙂

For that, I could use the environments feature. Have a Dev/Stg version of my current project that, inside it, has the Dev/Stg/Prd game services 'baked in'

hah! right? its a lot of traffic

we do run a multiplayer game so there is a lot of traffic going in and out, so I think the volume maybe makes sense

does the game server even have comparable traffic?

yeah, much more

i am grappling with this now, as our egress costs have made hosting on railway much more expensive than heroku hah

the gateway doesn't touch redis at all right?

yeah unfortunately there's not much railway can do about that right now since GCP has very high egress costs

so i am trying to make sure i am doing everything right and only incuring egress from what is absolutely neccessary, to get a better cost estimate. and thats how i encountered this odity with Redis

+1 yep

railway is moving to bare metal, I suspect the egress costs will drop and they will hopefully pass that down to the users too

but I am genuinely stumped here

fingers crossed, yeah it is a bit too pricing, it stings. I was excited about railway for future projects - for hosting game servers and such, but it might just cost too much in terms of egress for realtime applications

here is our prd game server

it might sting less if redis didn't have traffic for some reason <:mildpanic:804271964587819059>

exactly! hah 🙂

but good news, you're pro so I can escalate this to the team, hopefully they can dig into some of the internal network logs for these services, any last ideas before i do that though?

i just double checked my redis urls and such

but i guess with public networking disabled it doesnt mean much

so no, not really. I am also stumped! thanks 🙂

the team when available will communicate via that help station thread

I've heard they love when I escalate long debugging threads 🙂

😄 awesome thanks so much, curious to see what the outcome will be

me too!

Ok - I figured it out. While waiting for DNS changes to propagate I left our Heroku Dyno live. I had it connected to our Railway Redis service via public networking. I figured if I disable public networking it would kill any active connections but it seems like thats not the case? The Heroku Dyno was still connected to the Redis Service.

Anyway, shutting down the dyno reduced network usage as expected

how was the dyno generating so much network usage lol

Hmm, yeah, I am not sure why there is so much inbound traffic. I sort of expect the outbound. We're using the Redis Adapator with Socket IO to stream realtime multiplayer gameplay from Railways to the Heroku Dyno

anyway, I'll have to make sure I didn't write something too stupid on my end.

but atleast the egress issue is solved

tldr user error, though it is curious that the dynos connection to the Redis DB was not 'severed' when I disabled public networking

i will ask the person who wrote the tcp proxy if that would even be possible

but glad you where able to solve!

all the more reasons to shut down all your dynos 😉