c96ea2ca-1b03-415f-8931-5af0e51c87c4

what's weird is that i see an arroww from my app to postgres but not to my other deployments...is is possible that they are not on the same n etwork?

I tried this in local and it worked

> Is there a way to get a prompt on a deployed service

there is not, railway does not provide a way to ssh into a service.

> whether prometheus is accessible from my deployed app

it is as long as you are listening on ipv6 and using the correct port, as railway's private network is ipv6 only.

> what's weird is that i see an arroww from my app to postgres but not to my other deployments

the arrows and their directions are dynamically detected when you use reference variables, you are likely hard coding variables when you shouldn't be -

> is is possible that they are not on the same n etwork?

its not possible, every service within a given environment in a project is in the same private network.

> I tried this in local and it worked

local is likely ipv4, your services need to listen on an ipv6 host, or ideal they should all dual stack bind.

this is the first time i hear about this (and i've been using aws a lot)...what's an ipv6 host and how do i listen to one? what's a dual stack bind? 😅

this worked literally everywhere before (local, aws, heroku) and it only fails on railway

are there docs on this?

also why does it work this way?

it forces people to change their code so that they can facilitate the quirks of railway

there must be a good reason

i don't even understand why ipv6 is a concern when i'm using domain names eg: "http://prometheus:1234"

i checked and there is no docs on this

id mentions that private networking uses wireguard

and some sort of mesh

(whatever those might be)

but no details on what this means when i try to send metrics/traces

IPv6 is not a quirk at all, and not something specific to Railway, you just happen to always work on networks with IPv4 before.

There are docs for this -

because the domain names will resolve to an IPv6 address

so why can't i send metrics from prometheus and traces from tempo?

I get that, but the fact is that my code worked everywhere else

i only have to change it because of railway

yep, as mentioned everywhere else just happened to be IPv4

you don't get my point

and this is a tendency here

you implement something, cut corners by not implementing ipv4, and then you imply that all of this is totally OK, and your users are to blame for not thinking about the corners you cut 😅

and this is not the first case 😒

i'm not saying that you're not right, but from a user perspective it is a slap on the face

not implementing IPv4 is not corner cutting, IPv6 was introduced over two decade ago

in your opinion

see, that again

my program works everywhere but here

but all blame is assigned to me

I don't like to compare but Fly.io's private network is also IPv6 only

haven't used it

Just an example to show that we are not the only ones who choose to use a newer standard

i get that, but it is besides my point

i'm not arguing that this is a better way

or more forward-compatible

so if i understand this correctly the problem is tempo / prometheus is not configured to listen on ipv6? i'm using a metric exporter

interestingly enough grafana can connect to both

If your app is attempting to connect to temp / prometheus, they need to listen on IPv6, that is correct

i'm using a fork of a railway template

that's why i'm assuming that it would work

you'd also need to be using the correct port when connecting to them, same way you use ports even developing locally

i'm doing just that in theory

The vast majority of templates are user provided, it's possible they didn't fully test the template?

i don't know, maybe they have some other use case

can you send me over some actual error messages?

there are no errors interstingly

i'm using the official otel library

i just don't see the metrics

(nor the traces)

is there perhaps a verbose debug mode you can turn on?

i'll take a look

but i think the main problem is that prom/tempo is not configured for ipv6

i'm trying to use the setup that I was using before (metrics through prom, traces through tempo, aggregated in grafana)

once i figure this out i can share the code too if somebody is interested

I'm sure future readers would love that

👍

maybe i can post an article about it

anyway

is it easier / more secure / other if you use ipv6?

what's the rationale behind it?

I'm sorry I wouldn't know the exact reasons why it was chosen, I was not around for that discussion

ah, ok

in the docs it says that i have to use x.railway.internal in order for this to work, but in the railway app it says "you can also call me at "x". so would it make a difference if i used x.railway.internal:1234 instead of x:1234 or is it the same?

it shouldn't, I've never seen it make a difference

have you ever used prometheus or tempo on railway?

I have not

the search goes on then 😄

have you got your services to listen on ipv6?

i had to enable the otlp write receiver

now i can see metrics in prom

awsome!

looks like the ipv6 settings crash tempo

i copied the settings from this page: https://grafana.com/docs/tempo/latest/configuration/network/ipv6/

but when i push this to railway i get a wall of errors

this on repeat

link me to the specific service please

what is the name of the network interface?

i think eth0 doesn't exist

railnet0, but it shouldnt matter since you shouldnt be hardcoding any interfaces

b502be2b-037a-47c7-8744-2fe5e2c93a2d

tempo does that i think

bad practice 😬

what env?

according to this: https://community.grafana.com/t/grafana-loki-cluster-failed-to-start-if-no-network-interface-name-eth0/59157/3

staging

best to not hardcode an interface, but if you have to its railnet0 -

now that im looking at this, i think i should note that the ipv4 address is used for public traffic.

i don't hardcode stuff, but this comes from the tracing tool i've been using (Grafana Tempo)

you mean the public api?

i only added a public api so that i can check it with postman

that's how i figured out what the problem was with prom

i didnt mention a public api?

i'm not sure what you meant

😅

ok it seems the interface name is hardcoded in many places

https://github.com/grafana/tempo/issues/3590

lemme try this

look at the link i sent for context on what i said

railnet0, right?

yes, please look at the link

the stats page?

let's see if this solves the issue

ok, tempo booted up

still no traces though

thats not ideal

someting is amyss, but i see no requests in the log

so it might be in my service

when i tamper with tempo from grafana i can see it in the logs

good news is that app -> prom -> grafana is now working

awsome!

is there a way to expose a port on a deployment?

i think tempo is listening on a hardcoded port

publicly?

no, only on the private network

you dont need to do anything for that, there is no firewall or anything

oh, i'll try something then

FUCK YES

thanks for the help, everything seems to be working now!

what was the final fix?

I'm facing similar issues, @addamsson can you share your tempo.yaml?

Right now Im facing this issue:

level=error ts=2025-01-03T15:29:42.508327114Z caller=app.go:223 msg="module failed" module=distributor err="starting module distributor: invalid service state: Failed, expected: Running, failure: failed to start subservices: not healthy, 0 terminated, 1 failed: [error starting receiver: listen tcp [fd12:2222:f2cb:0:2000:8:eb5f:db2]:4317: bind: cannot assign requested address]"

changed tempo.yaml to:

distributor:
  ring:
    kvstore:
      store: memberlist
    instance_interface_names:
      - railnet0
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: "0.0.0.0:4317"
        http:
          endpoint: "0.0.0.0:4318"

And now the service starts and I'm able to connect from grafana...

But my app is failing to publish any traces... although it works locally (I'm also using effect.ts)

finally got it working for anyone looking into this in the future:

tempo.yaml:

---
server:
  grpc_listen_address: '::0'
  grpc_listen_port: 9095
  http_listen_address: '::0'
  http_listen_port: 3200

query_frontend:
  search:
    duration_slo: 5s
    throughput_bytes_slo: 1.073741824e+09
  trace_by_id:
    duration_slo: 100ms

distributor:
  ring:
    kvstore:
      store: memberlist
    instance_interface_names:
      - railnet0
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: "0.0.0.0:4317"
        http:
          endpoint: "0.0.0.0:4318"

ingester:
  lifecycler: 
    ring:
      replication_factor: 1
      kvstore:
        store: memberlist
    address: '::'
    enable_inet6: true
  max_block_duration: 5m # cut the headblock when this much time passes. this is being set for demo purposes and should probably be left alone normally

memberlist:
  bind_addr:
    - '::'
  bind_port: 7946

compactor:
  ring:
    instance_interface_names:
      - railnet0
    kvstore:
      store: memberlist
    enable_inet6: true
  compaction:
    block_retention: 1h # overall Tempo trace retention. set for demo purposes

metrics_generator:
  ring:
    instance_interface_names:
      - railnet0
    enable_inet6: true
  registry:
    external_labels:
      source: tempo
      cluster: docker-compose
  storage:
    path: /var/tempo/generator/wal
    remote_write:
      - url: http://prometheus:9090/api/v1/write
        send_exemplars: true
  traces_storage:
    path: /var/tempo/generator/traces
  processor:
    local_blocks:
      filter_server_spans: false
      flush_to_storage: true

storage:
  trace:
    backend: local # backend configuration to use
    wal:
      path: /var/tempo/wal # where to store the the wal locally
    local:
      path: /var/tempo/blocks

overrides:
  defaults:
    metrics_generator:
      processors: [service-graphs, span-metrics, local-blocks] # enables metrics generator
      generate_native_histograms: both

Make sure to add a volume to your tempo service, mount it in /var/tempo and add RAILWAY_RUN_UID="0" on tempo service

!s