Remote Write (Push)
gnmic
supports writing metrics to Prometheus using its remote write API.
gNMIc
's prometheus remote write can be used to push metrics to a variety of monitoring systems like Prometheus, Mimir, CortexMetrics, VictoriaMetrics, Thanos...
A Prometheus write output can be defined using the below format in gnmic
config file under outputs
section:
outputs:
output1:
# required
type: prometheus_write
# url to push metrics towards, scheme is required
url: http://<grafana-mimir-addr>:9009/api/v1/push
# a map of string:string,
# custom HTTP headers to be sent along with each remote write request.
headers:
# header: value
# sets the `Authorization` header on every remote write request with the
# configured username and password.
authentication:
username:
password:
# sets the `Authorization` header with type `.authorization.type` and the token value.
authorization:
type: Bearer
credentials: <token string>
# tls config
tls:
# string, path to the CA certificate file,
# this will be used to verify the clients certificates when `skip-verify` is false
ca-file:
# string, client certificate file.
cert-file:
# string, client key file.
key-file:
# boolean, if true, the client will not verify the server
# certificate against the available certificate chain.
skip-verify: false
# duration, defaults to 10s, time interval between write requests
interval: 10s
# integer, defaults to 1000.
# Buffer size for time series to be sent to the remote system.
# metrics are sent to the remote system every `.interval` or when the buffer is full. Whichever one is reached first.
buffer-size: 1000
# integer, defaults to 500, sets the maximum number of timeSeries per write request to remote.
max-time-series-per-write: 500
# integer, defaults to 0
# number of retries per write, retries will have a back off of 100ms.
max-retries: 0
# metadata configuration
metadata:
# boolean,
# if true, metrics metadata is sent.
include: false
# duration, defaults to 60s.
# Applies if `metadata.include` is set to true
# Interval after which all metadata entries are sent to the remote write address
interval: 60s
# integer, defaults to 500
# applies if `metadata.include` is set to true
# Max number of metadata entries per write request.
max-entries-per-write: 500
# string, to be used as the metric namespace
metric-prefix: ""
# boolean, if true the subscription name will be appended to the metric name after the prefix
append-subscription-name: false
# boolean, enables setting string type values as prometheus metric labels.
strings-as-labels: false
# duration, defaults to 10s
# Push request timeout.
timeout: 10s
# boolean, defaults to false
# Enables debug for prometheus write output.
debug: false
# string, one of `overwrite`, `if-not-present`, ``
# This field allows populating/changing the value of Prefix.Target in the received message.
# if set to ``, nothing changes
# if set to `overwrite`, the target value is overwritten using the template configured under `target-template`
# if set to `if-not-present`, the target value is populated only if it is empty, still using the `target-template`
add-target:
# string, a GoTemplate that allow for the customization of the target field in Prefix.Target.
# it applies only if the previous field `add-target` is not empty.
# if left empty, it defaults to:
# {{- if index . "subscription-target" -}}
# {{ index . "subscription-target" }}
# {{- else -}}
# {{ index . "source" | host }}
# {{- end -}}`
# which will set the target to the value configured under `subscription.$subscription-name.target` if any,
# otherwise it will set it to the target name stripped of the port number (if present)
target-template:
# list of processors to apply on the message before writing
event-processors:
# an integer, sets the number of worker handling messages to be converted into Prometheus metrics
num-workers: 1
# an integer, sets the number of writers draining the buffer and writing to Prometheus
num-writers: 1
gnmic
creates the prometheus metric name and its labels from the subscription name, the gnmic path and the value name.
Metric Generation#
The below diagram shows an example of a prometheus metric generation from a gnmi update
Metric Naming#
The metric name starts with the string configured under metric-prefix.
Then if append-subscription-name is true
, the subscription-name as specified in gnmic
configuration file is appended.
The resulting string is followed by the gNMI path stripped of its keys if there are any.
All non-alphanumeric characters are replaced with an underscore "_
"
The 3 strings are then joined with an underscore "_
"
If further customization of the metric name is required, the processors can be used to transform the metric name.
For example, a gNMI update from subscription port-stats
with path:
/interfaces/interface[name=1/1/1]/subinterfaces/subinterface[index=0]/state/counters/in-octets
is exposed as a metric named:
gnmic_port_stats_interfaces_interface_subinterfaces_subinterface_state_counters_in_octets
Metric Labels#
The metrics labels are generated from the subscription metadata (e.g: subscription-name
and source
) and the keys present in the gNMI path elements.
For the previous example the labels would be:
{interface_name="1/1/1",subinterface_index=0,source="$routerIP:Port",subscription_name="port-stats"}
Prometheus Write Metrics#
When a Prometheus server (gNMI API) is enabled, gnmic
prometheus write output exposes 4 prometheus counters and 2 prometheus Gauges:
number_of_prometheus_write_msgs_sent_success_total
: Number of msgs successfully sent by gnmic prometheus_write output.number_of_prometheus_write_msgs_sent_fail_total
: Number of failed msgs sent by gnmic prometheus_write output.-
msg_send_duration_ns
: gnmic prometheus_write output send duration in ns. -
number_of_prometheus_write_metadata_msgs_sent_success_total
: Number of metadata msgs successfully sent by gnmic prometheus_write output. number_of_prometheus_write_metadata_msgs_sent_fail_total
: Number of failed metadata msgs sent by gnmic prometheus_write output.metadata_msg_send_duration_ns
: gnmic prometheus_write output metadata send duration in ns.