query-orchestration/internal/serviceconfig/observability/prometheus/echocustommetrics.md

Echo Custom Metrics integration

In March of 2025 we looked into integration the metrics that we get from the echo server prometheus middleware into the custom metrics that we get from the framework for non echo servers. (as a sizing exercise)

Attempting to do a simple integration will fail because the metrics for each appear to be collected in different repositories.

The final solution for getting this to work will probably require the following:

• Obtain a shared registry for the prometheus metrics. (see example 1 below)
• Change InitializeMetrics to use the shared registry. (see example 3 below)
• Initialize echo with the custom registry. (see example 2 below)
• Debug the metrics to see if they are being collected correctly.

Notes: Code like the examples below is only partially working. The custom metrics still fail to show up with the prometheus metrics. So it may also be necessary to expose a /metrics endpoint manually like we do in the non echo case to get past this. Either way further investigation is needed.

Example 1 of getting a custom shared registry for the prometheus metrics.

var (
	singletonCustomRegistry *prometheus.Registry
	onceRegistry            sync.Once
)

func GetSingletonCustomRegistry(namespace string) *prometheus.Registry {
	if namespace == "" {
		namespace = "doczy"
	}
	onceRegistry.Do(func() {
		singletonCustomRegistry = prometheus.NewRegistry()
		singletonCustomRegistry.MustRegister(collectors.NewGoCollector())
		singletonCustomRegistry.MustRegister(collectors.NewProcessCollector(collectors.ProcessCollectorOpts{
			Namespace: namespace,
		}))
	})

	return singletonCustomRegistry
}

Example 2. Initialize echo with the custom registry

	e := echo.New()

	registry := p2.GetSingletonCustomRegistry("")
	//// Configure Echo Prometheus middleware with the custom registry
	e.Use(echoprometheus.NewMiddlewareWithConfig(echoprometheus.MiddlewareConfig{
		Registerer: registry,
	}))
	//
	e.GET("/metrics", echo.WrapHandler(
		promhttp.HandlerFor(registry, promhttp.HandlerOpts{}),
	))
	// end experimental

	// Other middlewares
	e.Use(getSlogCustomEchoMiddleware(cfg.GetLogger()))
	e.Use(middleware.Recover())
	e.Use(middleware.CORS())

Example 3. Initialize metrics with the custom registry

func InitializeMetrics(ctx context.Context, cfg ListenerConfig) error {
	standardDefinitions := prometheus.NewStandardMetricDefinitions()

	registry := prometheus.GetSingletonCustomRegistry("")
...