4 months ago
Hey, I'm currently battling elevated DPM in Grafana, and pinpointed the reason to the fact that we have a few services deployed in multiple regions, each exporting metrics every 60s with the same OTEL_RESOURCE_ATTRIBUTES. We use standard OTEL libraries that are configured entirely through environment variables across 6 different services.
The challenge now is that we need to ensure metrics from multi-instance services are exported with unique instance IDs. So I went looking for that and found RAILWAY_REPLICA_ID and RAILWAY_REGION. I thought I could just set OTEL_SERVICE_ATTRIBUTES to e.g.
service.name=filecoin-app,service.namespace=app,deployment.environment=production,service.instance.id=${{RAILWAY_REPLICA_ID}},cloud.region=${{RAILWAY_REGION}}but unfortunately those substitutions are empty. I assume that those two env vars are only available at run / build-time and not available for substitution within other env vars, and this is probably expected behavior?
But then, the question remains - how can I solve this problem? 🤔
8 Replies
4 months ago
That is expected behavior, those variables are dynamically injected at runtime.
any chance someone had this problem before and you're maybe aware of some workaround? not really sure what the best approach is... with our node services i could relatively easily mess with the OTEL sdk and configure these values at runtime, but with Deno OTEL as far as I'm aware, the only way to set these attributes is through the environment... 😵💫
4 months ago
Then set them through the environment at runtime?
ended up simply setting the start command to append the runtime vars:
sh -c 'OTEL_RESOURCE_ATTRIBUTES="${OTEL_RESOURCE_ATTRIBUTES},service.instance.id=${RAILWAY_REPLICA_ID},cloud.region=${RAILWAY_REPLICA_REGION}" node --require ./build/telemetry.cjs build'works perfectly 🎉 thanks @Brody, no idea why i didn't think of that 🙂
4 months ago
No problem!
Status changed to Solved brody • 4 months ago