Problems grafana stack
andrest3086
FREEOP

10 months ago

I have a problem trying to send metrics to grafana from my Backend service, I used the "grafana stack" template with the variables as the documentation says, but I only get a grcp error when trying to send:
  • Failed to export metrics. Server responded with gRPC status code 2. Error message: timeout

  • i.o.exporter.internal.grpc.GrpcExporter : Failed to export spans. Server responded with gRPC status code 2. Error message:

$10 Bounty

2 Replies

lofimit
HOBBY

10 months ago

Hey, this usually happens because your backend can’t reach the Grafana metrics endpoint properly or the timeout is too low. Most likely you’re using the wrong service address or port for sending the metrics, or the Grafana service is slow/down.

Check if you’re using the right internal service name and port (usually 4317 for gRPC). Also, try increasing the timeout in your exporter config. If the Grafana stack is under heavy load, it might be slow, causing the timeout error.

You can also try pinging the Grafana service from your backend container to see if it connects.

In case what I said didn't help, please try providing more information for me to help you more precisely, maybe share the Error message?


andrest3086
FREEOP

10 months ago

Hi, this is my stack, I think that my backend is correctly connected to Grafana but nothing happens when I reach an endpoint. I can´t see metrics.
These are my variables that I use to connect my backend with grafana's services:

GF_DEFAULT_INSTANCE_NAME

Hotel-Backend

GF_INSTALL_PLUGINS

grafana-piechart-panel,grafana-worldmap-panel,grafana-clock-panel,grafana-simple-json-datasource

GF_LOG_MODE

console

GF_PORT

3000

GF_VERSION

latest

INTERNAL_HTTP_INGEST

http://tempo-jx08.railway.internal:4318

LOKI_INTERNAL_URL

http://loki-tlb1.railway.internal:3100

MANAGEMENT_ENDPOINTS_WEB_EXPOSURE_INCLUDE

health,loggers,info,metrics

MANAGEMENT_METRICS_BINDERS_PROCESS_ENABLED

*******

MANAGEMENT_METRICS_BINDERS_PROCESSOR_ENABLED

*******

MANAGEMENT_METRICS_ENABLE_PROCESS

false

MANAGEMENT_METRICS_ENABLE_PROCESSOR

false

MANAGEMENT_OTLP_METRICS_EXPORT_STEP

15s
PROMETHEUS_INTERNAL_URL

http://prometheus-t5ud.railway.internal:9090

TEMPO_INTERNAL_URL

http://tempo-jx08.railway.internal:3200

I appreciate any help

Attachments


Loading...