query-orchestration/internal/serviceconfig/observability/prometheus

Prometheus (Doczy)

Prometheus is a simplified wrapper around the Prometheus client library for Go, making it easier to define and use Prometheus metrics in your services. It provides a straightforward way to create and record various types of Prometheus metrics with minimal boilerplate code.

Features

• Simple configuration-based metric definition
• Support for all major Prometheus metric types:
  • Counter
  • Gauge
  • Histogram
  • Summary
• Automatic HTTP metrics endpoint setup
• Thread-safe metric recording
• Label support for all metric types
• Context-based graceful shutdown
• Clean server lifecycle management

Example cmd (metricsExample_test)

See cmd/metricsExample_test for a sample usage in a working main.go. The Readme.md in that directory has more details on how to use the harness.

Running tests with prometheus server

See prometheus.md for prometheus related instructions and testing instructions.

Quick Start (with your own http server)

This example (and the metricsExample_test) shows how to use the harness to create a simple http server that exposes metrics. If you are using this with an API server you can indicate in your config that you already have a http server and it will not start a http server for you. (e.g. Use the one that we automatically setup as Echo middleware) For a more complete and up-to-date example always refer to the metricsExample_test sample which you can run.

package main

import (
    "context"
    "log"
    "os"
    "os/signal"
    "syscall"
    "time"
    "queryorchestration/internal/serviceconfig/prometheus"
)

func main() {
    // Create a context that we can cancel
    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()

    config := prometrics.Config{
        Namespace: "myapp",
        Subsystem: "subsystem",
        ServeHttp: true,
        Port:      8080,
        Definitions: []prometheus.MetricDefinition{
            {
                Type:   prometheus.Counter,
                Name:   "requests_total",
                Help:   "Total number of requests processed",
                Labels: []string{"endpoint", "status"},
            },
        },
    }

    metrics, err := prometheus.New(ctx, config)
    if err != nil {
        panic(err)
    }

    // Setup graceful shutdown
    sigChan := make(chan os.Signal, 1)
    signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)

    go func() {
        <-sigChan
        // Cancel context
        cancel()
        
        // Give the server up to 5 seconds to shutdown gracefully
        shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 5*time.Second)
        defer shutdownCancel()
        
        if err := metrics.Shutdown(shutdownCtx); err != nil {
            log.Printf("Error shutting down metrics server: %v", err)
        }
    }()

    // Record a metric
    metrics.RecordMetric("requests_total", 1, map[string]string{
        "endpoint": "/api/v1",
        "status":   "success",
    })

    // Block until signal is received
    <-ctx.Done()
}

Configuration

MetricsConfig Structure

type MetricsConfig struct {
    Definitions []MetricDefinition  // List of metrics to create
    Namespace   string             // Namespace for all metrics
    Subsystem   string             // Subsystem for all metrics
    ServeHttp   bool              // Whether to serve metrics endpoint
    Port        int               // Port for metrics endpoint
}

MetricDefinition Structure

type MetricDefinition struct {
    Type       MetricType          // Counter, Gauge, Histogram, or Summary
    Name       string              // Metric name
    Help       string              // Help text for the metric
    Labels     []string            // Label names for the metric
    Buckets    []float64           // Buckets for Histogram type
    Objectives map[float64]float64 // Objectives for Summary type
}

Metric Types

Counter

Used for cumulative metrics that only increase in value.

MetricDefinition{
    Type:   prometrics.Counter,
    Name:   "requests_total",
    Help:   "Total requests processed",
    Labels: []string{"endpoint"},
}

Gauge

Used for metrics that can increase and decrease.

MetricDefinition{
    Type:   prometrics.Gauge,
    Name:   "active_connections",
    Help:   "Current number of active connections",
    Labels: []string{"server"},
}

Histogram

Used for measurements like request duration or response size.

MetricDefinition{
    Type:    prometrics.Histogram,
    Name:    "request_duration_seconds",
    Help:    "Request duration in seconds",
    Labels:  []string{"endpoint"},
    Buckets: []float64{0.1, 0.5, 1, 2, 5},
}

Summary

Similar to histogram but calculates quantiles over time.

MetricDefinition{
    Type:       prometrics.Summary,
    Name:       "request_latency",
    Help:       "Request latency",
    Labels:     []string{"endpoint"},
    Objectives: map[float64]float64{0.5: 0.05, 0.9: 0.01, 0.99: 0.001},
}

Recording Metrics

Use the RecordMetric method to record values for any metric type:

err := metrics.RecordMetric("metric_name", value, map[string]string{
    "label1": "value1",
    "label2": "value2",
})

The method is thread-safe and will return an error if:

• The metric doesn't exist
• The label values don't match the metric's label names
• The metric type is invalid

HTTP Endpoint

When ServeHttp is set to true in the configuration, the package automatically serves metrics at /metrics on the specified port. This endpoint can be scraped by Prometheus.

Lifecycle Management

The library provides two ways to manage the lifecycle of the metrics server:

1. Context-based shutdown:

ctx, cancel := context.WithCancel(context.Background())
metrics, err := prometheus.New(ctx, config)
// Server will shut down when context is cancelled
cancel()

2. Manual shutdown:

shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 5*time.Second)
defer shutdownCancel()
err := metrics.Shutdown(shutdownCtx)

The metrics server will also automatically shut down when the main process exits, but it's recommended to use one of the above methods for graceful shutdown.

Thread Safety

All operations in the package are thread-safe, allowing for concurrent metric recording from multiple goroutines.

Error Handling

The package returns errors in the following cases:

• Invalid metric type specified in configuration
• Failed metric registration (e.g., duplicate metrics)
• Recording to non-existent metric
• Invalid label combinations
• Server shutdown failures

Best Practices

1. Always provide a context when creating new metrics:

ctx, cancel := context.WithCancel(context.Background())
defer cancel()
metrics, err := prometheus.New(ctx, config)

2. Implement graceful shutdown in your application:

shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 5*time.Second)
defer shutdownCancel()
if err := metrics.Shutdown(shutdownCtx); err != nil {
    log.Printf("Error during metrics shutdown: %v", err)
}

3. Handle server errors appropriately:

if err := metrics.Shutdown(ctx); err != nil {
    // Log the error but don't panic - the application can continue
    log.Printf("Metrics server shutdown error: %v", err)
}

4. Use appropriate timeouts for shutdown:

ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
metrics.Shutdown(ctx)