Skip to content

Scrape Based (Pull)

gnmic supports exposing gnmi updates on a prometheus server, for a prometheus client to scrape.

A Prometheus output can be defined using the below format in gnmic config file under outputs section:

outputs:
  output1:
    type: prometheus # require
    # address to listen on for incoming scrape requests
    listen: :9804 
    # path to query to get the metrics
    path: /metrics 
    # maximum lifetime of metrics in the local cache, #
    # a zero value defaults to 60s, a negative duration (e.g: -1s) disables the expiration
    expiration: 60s 
    # a string to be used as the metric namespace
    metric-prefix: "" 
    # a boolean, if true the subscription name will be appended to the metric name after the prefix
    append-subscription-name: false 
    # boolean, if true the message timestamp is changed to current time
    override-timestamps: false
    # a boolean, enables exporting timestamps received from the gNMI target as part of the metrics
    export-timestamps: false 
    # a boolean, enables setting string type values as prometheus metric labels.
    strings-as-labels: false
    # a boolean, if set to true, the received gNMI notifications are stored in a cache.
    # the prometheus metrics are generated at the time a prometheus server sends scrape request.
    # this behavior allows the processors (if defined) to be run on all the generated events at once.
    # this mode uses more resource compared to the default one, but offers more flexibility when it comes 
    # to manipulating the data to customize the returned metrics using event-processors.
    gnmi-cache: false
    # duration, scrape request timeout.
    # this timer is started when a scrape request is received, 
    # if it is reached, the metrics generation/collection is stopped.
    timeout: 10s
    # enable debug for prometheus output
    debug: false 
    # string, one of `overwrite`, `if-not-present`, ``
    # This field allows populating/changing the value of Prefix.Target in the received message.
    # if set to ``, nothing changes 
    # if set to `overwrite`, the target value is overwritten using the template configured under `target-template`
    # if set to `if-not-present`, the target value is populated only if it is empty, still using the `target-template`
    add-target: 
    # string, a GoTemplate that allow for the customization of the target field in Prefix.Target.
    # it applies only if the previous field `add-target` is not empty.
    # if left empty, it defaults to:
    # {{- if index . "subscription-target" -}}
    # {{ index . "subscription-target" }}
    # {{- else -}}
    # {{ index . "source" | host }}
    # {{- end -}}`
    # which will set the target to the value configured under `subscription.$subscription-name.target` if any,
    # otherwise it will set it to the target name stripped of the port number (if present)
    target-template:
    # list of processors to apply on the message before writing
    event-processors: 
    # Enables Consul service registration
    service-registration:
      # Consul server address, default to localhost:8500
      address:
      # Consul Data center, defaults to dc1
      datacenter: 
      # Consul username, to be used as part of HTTP basicAuth
      username:
      # Consul password, to be used as part of HTTP basicAuth
      password:
      # Consul Token, is used to provide a per-request ACL token which overrides the agent's default token
      token:
      # Prometheus service check interval, for both http and TTL Consul checks,
      # defaults to 5s
      check-interval:
      # Maximum number of failed checks before the service is deleted by Consul
      # defaults to 3
      max-fail:
      # Consul service name
      name:
      # List of tags to be added to the service registration, 
      # if available, the instance-name and cluster-name will be added as tags,
      # in the format: gnmic-instance=$instance-name and gnmic-cluster=$cluster-name
      tags:
      # bool, enables http service check on top of the TTL check
      enable-http-check:
      # string, if enable-http-check is true, this field can be used to specify the http endpoint to be used to the check
      # if provided, this filed with be prepended with 'http://' (if not already present), 
      # and appended with the value in 'path' field.
      # if not specified, it will be derived from the fields 'listen' and 'path'
      http-check-address:
      # if set to true, the gnmic instance will try to ac quire a lock before registering the prometheus output in consul.
      # this allows to register a single instance of the cluster in consul.
      # if the instance which acquired the lock fails, one of the remaining ones will take over.
      use-lock: false

gnmic creates the prometheus metric name and its labels from the subscription name, the gnmic path and the value name.

Metric Generation#

The below diagram shows an example of a prometheus metric generation from a gnmi update

Metric Naming#

The metric name starts with the string configured under metric-prefix.

Then if append-subscription-name is true, the subscription-name as specified in gnmic configuration file is appended.

The resulting string is followed by the gNMI path stripped of its keys if there are any.

All non-alphanumeric characters are replaced with an underscore "_"

The 3 strings are then joined with an underscore "_"

If further customization of the metric name is required, the processors can be used to transform the metric name.

For example, a gNMI update from subscription port-stats with path:

/interfaces/interface[name=1/1/1]/subinterfaces/subinterface[index=0]/state/counters/in-octets

is exposed as a metric named:

gnmic_port_stats_interfaces_interface_subinterfaces_subinterface_state_counters_in_octets

Metric Labels#

The metrics labels are generated from the subscription metadata (e.g: subscription-name and source) and the keys present in the gNMI path elements.

For the previous example the labels would be:

{interface_name="1/1/1",subinterface_index=0,source="$routerIP:Port",subscription_name="port-stats"}

Service Registration#

gnmic supports prometheus_output service registration via Consul.

It allows prometheus to dynamically discover new instances of gnmic exposing a prometheus server ready for scraping via its service discovery feature.

If the configuration section service-registration is present under the output definition, gnmic will register the prometheus_output service in Consul.

Configuration Example#

The below configuration, registers a service name gnmic-prom-srv with IP=10.1.1.1 and port=9804

# gnmic.yaml
outputs:
  output1:
    type: prometheus
    listen: 10.1.1.1:9804
    path: /metrics 
    service-registration:
      address: consul-agent.local:8500
      name: gnmic-prom-srv

This allows running multiple instances of gnmic with minimal configuration changes by using prometheus service discovery feature.

Simplified scrape configuration in the prometheus client:

# prometheus.yaml
scrape_configs:
  - job_name: 'gnmic'
    scrape_interval: 10s
    consul_sd_configs:
      - server: $CONSUL_ADDRESS
        services:
          - $service_name

Service Name and ID#

The $service_name to be discovered by prometheus is configured under outputs.$output_name.service-registration.name.

If the service registration name field is not present, the name prometheus-${output_name} will be used.

In both cases the service ID will be prometheus-${output_name}-${instance_name}.

Service Checks#

gnmic registers the service in Consul with a ttl check enabled by default:

  • ttl: gnmic periodically updates the service definition in Consul. The goal is to allow Consul to detect a same instance restarting with a different service name.

If service-registration.enable-http-check is true, an http check is added:

  • http: Consul periodically scrapes the prometheus server endpoint to check its availability.
# gnmic.yaml
outputs:
  output1:
    type: prometheus
    listen: 10.1.1.1:9804
    path: /metrics 
    service-registration:
      address: consul-agent.local:8500
      name: gnmic-prom-srv
      enable-http-check: true

Note that for the http check to work properly, a reachable address ( IP or name ) should be specified under listen.

Otherwise, a reachable address should be added under service-registration.http-check-address.

Caching#

When caching is enabled, the received messages are not immediately converted into metrics, they are written to the cache as gNMI updates. They are converted only when a scrape request is received.

The below diagram shows how a prometheus output works with and without cache enabled:

When caching is enabled, the received gNMI updates are not processed and converted into metrics immediately, they are rather stored as is in the configured gNMI cache.

Once a scrape request is received from Prometheus, all the cached gNMI updates are retrieved from the cache, converted to events, the configured processors, if any, are then applied to the whole list of events. Finally, The resulting event are converted into metrics and written back to Prometheus within the scrape response.