Categories
DevOps

How to inspect consul metrics

Inspect current consul metrics.

Get the PID of the consul process.

$ systemctl show --property MainPID --value consul
31045

Send USR1 signal to the consul process.

sudo kill -SIGUSR1 $(systemctl show --property MainPID --value consul)

Inspect consul logs for data.

$ sudo journalctl -u consul -f
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.raft.last_index': 80560.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.raft.boltdb.txstats.pageAlloc': 9900032.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.consul.members.servers.dc-lab-1': 3.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.runtime.heap_objects': 46577.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.raft.boltdb.numFreePages': 913.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.consul.state.connect_instances.dc-lab-1.connect-native': 0.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.consul.state.config_entries.dc-lab-1.ingress-gateway': 0.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.consul.state.connect_instances.dc-lab-1.ingress-gateway': 0.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.consul.state.config_entries.dc-lab-1.service-resolver': 0.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.session_ttl.active': 0.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.raft.applied_index': 65554.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.runtime.num_goroutines': 84.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.consul.state.connect_instances.dc-lab-1.connect-proxy': 0.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.consul.state.config_entries.dc-lab-1.proxy-defaults': 0.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.raft.boltdb.openReadTxn': 0.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.consul.state.config_entries.dc-lab-1.service-intentions': 0.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.consul.state.config_entries.dc-lab-1.terminating-gateway': 0.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.consul.state.config_entries.dc-lab-1.service-defaults': 0.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.consul.state.kv_entries.dc-lab-1': 62.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.autopilot.failure_tolerance': 0.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.runtime.alloc_bytes': 11396104.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.runtime.sys_bytes': 33833992.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.runtime.malloc_count': 5820461.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.consul.state.nodes.dc-lab-1': 5.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.consul.state.config_entries.dc-lab-1.exported-services': 0.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.consul.state.config_entries.dc-lab-1.service-splitter': 0.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.raft.boltdb.numPendingPages': 3.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.raft.boltdb.txstats.pageCount': 2417.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.consul.state.services.dc-lab-1': 3.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.consul.state.connect_instances.dc-lab-1.mesh-gateway': 0.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.consul.state.config_entries.dc-lab-1.service-router': 0.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.raft.boltdb.freelistBytes': 7344.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.consul.state.service_instances.dc-lab-1': 5.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.consul.state.connect_instances.dc-lab-1.terminating-gateway': 0.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.consul.state.config_entries.dc-lab-1.mesh': 0.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.consul.members.clients.dc-lab-1': 0.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.autopilot.healthy': 0.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.runtime.free_count': 5773884.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.runtime.total_gc_pause_ns': 13157574.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.runtime.total_gc_runs': 122.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][G] 'consul.raft.boltdb.freePageBytes': 3751936.000
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][C] 'consul.memberlist.udp.sent': Count: 23 Min: 104.000 Mean: 142.391 Max: 181.000 Stddev: 38.119 Sum: 3275.000 LastUpdated: 2022-07-23 22:53:19.938807097 +0000 UTC m=+1988.209919664
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][C] 'consul.memberlist.tcp.connect': Count: 1 Sum: 1.000 LastUpdated: 2022-07-23 22:53:11.63750088 +0000 UTC m=+1979.908613448
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][C] 'consul.raft.state.candidate': Count: 1 Sum: 1.000 LastUpdated: 2022-07-23 22:53:13.722373881 +0000 UTC m=+1981.993486455
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][C] 'consul.raft.boltdb.totalReadTxn': Count: 2 Sum: 0.000 LastUpdated: 2022-07-23 22:53:16.814405495 +0000 UTC m=+1985.085518062
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][C] 'consul.raft.boltdb.txstats.cursorCount': Count: 2 Sum: 18.000 LastUpdated: 2022-07-23 22:53:16.814412627 +0000 UTC m=+1985.085525194
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][C] 'consul.rpc.accept_conn': Count: 19 Sum: 19.000 LastUpdated: 2022-07-23 22:53:19.296848463 +0000 UTC m=+1987.567961030
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][C] 'consul.memberlist.tcp.sent': Count: 1 Sum: 809.000 LastUpdated: 2022-07-23 22:53:11.637645833 +0000 UTC m=+1979.908758399
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][C] 'consul.raft.boltdb.txstats.nodeDeref': Count: 2 Sum: 0.000 LastUpdated: 2022-07-23 22:53:16.814415131 +0000 UTC m=+1985.085527697
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][C] 'consul.raft.boltdb.txstats.rebalance': Count: 2 Sum: 0.000 LastUpdated: 2022-07-23 22:53:16.814416313 +0000 UTC m=+1985.085528879
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][C] 'consul.raft.boltdb.txstats.spill': Count: 2 Sum: 6.000 LastUpdated: 2022-07-23 22:53:16.814447296 +0000 UTC m=+1985.085559863
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][C] 'consul.memberlist.udp.received': Count: 23 Min: 104.000 Mean: 145.739 Max: 181.000 Stddev: 37.975 Sum: 3352.000 LastUpdated: 2022-07-23 22:53:19.938643452 +0000 UTC m=+1988.209756023
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][C] 'consul.raft.boltdb.txstats.nodeCount': Count: 2 Sum: 12.000 LastUpdated: 2022-07-23 22:53:16.814413987 +0000 UTC m=+1985.085526555
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][C] 'consul.raft.boltdb.txstats.split': Count: 2 Sum: 0.000 LastUpdated: 2022-07-23 22:53:16.814446205 +0000 UTC m=+1985.085558773
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][C] 'consul.raft.boltdb.txstats.write': Count: 2 Sum: 18.000 LastUpdated: 2022-07-23 22:53:16.814449934 +0000 UTC m=+1985.085562500
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][S] 'consul.memberlist.pushPullNode': Count: 1 Sum: 1.340 LastUpdated: 2022-07-23 22:53:11.638359857 +0000 UTC m=+1979.909472424
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][S] 'consul.raft.boltdb.txstats.rebalanceTime': Count: 2 Sum: 0.000 LastUpdated: 2022-07-23 22:53:16.814443064 +0000 UTC m=+1985.085555636
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][S] 'consul.serf.queue.Intent': Count: 2 Sum: 0.000 LastUpdated: 2022-07-23 22:53:11.95907866 +0000 UTC m=+1980.230191228
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][S] 'consul.serf.queue.Event': Count: 2 Sum: 0.000 LastUpdated: 2022-07-23 22:53:11.959109924 +0000 UTC m=+1980.230222492
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][S] 'consul.runtime.gc_pause_ns': Count: 1 Sum: 85561.000 LastUpdated: 2022-07-23 22:53:15.250622957 +0000 UTC m=+1983.521735524
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][S] 'consul.memberlist.gossip': Count: 70 Min: 0.006 Mean: 0.019 Max: 0.063 Stddev: 0.007 Sum: 1.304 LastUpdated: 2022-07-23 22:53:19.917414952 +0000 UTC m=+1988.188527519
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][S] 'consul.serf.coordinate.adjustment-ms': Count: 12 Min: 0.130 Mean: 0.333 Max: 0.402 Stddev: 0.095 Sum: 3.995 LastUpdated: 2022-07-23 22:53:19.918126991 +0000 UTC m=+1988.189239557
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][S] 'consul.memberlist.probeNode': Count: 12 Min: 0.488 Mean: 0.684 Max: 0.795 Stddev: 0.081 Sum: 8.213 LastUpdated: 2022-07-23 22:53:19.918129767 +0000 UTC m=+1988.189242334
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][S] 'consul.raft.boltdb.txstats.spillTime': Count: 2 Min: 0.016 Mean: 0.018 Max: 0.019 Stddev: 0.002 Sum: 0.036 LastUpdated: 2022-07-23 22:53:16.814448764 +0000 UTC m=+1985.085561332
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][S] 'consul.raft.boltdb.txstats.writeTime': Count: 2 Min: 6.659 Mean: 7.145 Max: 7.631 Stddev: 0.687 Sum: 14.290 LastUpdated: 2022-07-23 22:53:16.814451344 +0000 UTC m=+1985.085563910
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][S] 'consul.serf.queue.Query': Count: 2 Sum: 0.000 LastUpdated: 2022-07-23 22:53:11.959120713 +0000 UTC m=+1980.230233281
Jul 23 22:53:27 consul-2 consul[31045]: [2022-07-23 22:53:10 +0000 UTC][S] 'consul.raft.candidate.electSelf': Count: 1 Sum: 1.474 LastUpdated: 2022-07-23 22:53:13.726320218 +0000 UTC m=+1981.997432793

Use sed to extract metrics data.

$ sudo journalctl -u consul -f | sed  -n -e "s/.*\[.*\]\[[CS]\] \(.*\)/\1/p"
'consul.memberlist.udp.sent': Count: 23 Min: 104.000 Mean: 142.391 Max: 181.000 Stddev: 38.119 Sum: 3275.000 LastUpdated: 2022-07-23 22:53:19.938807097 +0000 UTC m=+1988.209919664
'consul.memberlist.tcp.connect': Count: 1 Sum: 1.000 LastUpdated: 2022-07-23 22:53:11.63750088 +0000 UTC m=+1979.908613448
'consul.raft.state.candidate': Count: 1 Sum: 1.000 LastUpdated: 2022-07-23 22:53:13.722373881 +0000 UTC m=+1981.993486455
'consul.raft.boltdb.totalReadTxn': Count: 2 Sum: 0.000 LastUpdated: 2022-07-23 22:53:16.814405495 +0000 UTC m=+1985.085518062
'consul.raft.boltdb.txstats.cursorCount': Count: 2 Sum: 18.000 LastUpdated: 2022-07-23 22:53:16.814412627 +0000 UTC m=+1985.085525194
'consul.rpc.accept_conn': Count: 19 Sum: 19.000 LastUpdated: 2022-07-23 22:53:19.296848463 +0000 UTC m=+1987.567961030
'consul.memberlist.tcp.sent': Count: 1 Sum: 809.000 LastUpdated: 2022-07-23 22:53:11.637645833 +0000 UTC m=+1979.908758399
'consul.raft.boltdb.txstats.nodeDeref': Count: 2 Sum: 0.000 LastUpdated: 2022-07-23 22:53:16.814415131 +0000 UTC m=+1985.085527697
'consul.raft.boltdb.txstats.rebalance': Count: 2 Sum: 0.000 LastUpdated: 2022-07-23 22:53:16.814416313 +0000 UTC m=+1985.085528879
'consul.raft.boltdb.txstats.spill': Count: 2 Sum: 6.000 LastUpdated: 2022-07-23 22:53:16.814447296 +0000 UTC m=+1985.085559863
'consul.memberlist.udp.received': Count: 23 Min: 104.000 Mean: 145.739 Max: 181.000 Stddev: 37.975 Sum: 3352.000 LastUpdated: 2022-07-23 22:53:19.938643452 +0000 UTC m=+1988.209756023
'consul.raft.boltdb.txstats.nodeCount': Count: 2 Sum: 12.000 LastUpdated: 2022-07-23 22:53:16.814413987 +0000 UTC m=+1985.085526555
'consul.raft.boltdb.txstats.split': Count: 2 Sum: 0.000 LastUpdated: 2022-07-23 22:53:16.814446205 +0000 UTC m=+1985.085558773
'consul.raft.boltdb.txstats.write': Count: 2 Sum: 18.000 LastUpdated: 2022-07-23 22:53:16.814449934 +0000 UTC m=+1985.085562500
'consul.memberlist.pushPullNode': Count: 1 Sum: 1.340 LastUpdated: 2022-07-23 22:53:11.638359857 +0000 UTC m=+1979.909472424
'consul.raft.boltdb.txstats.rebalanceTime': Count: 2 Sum: 0.000 LastUpdated: 2022-07-23 22:53:16.814443064 +0000 UTC m=+1985.085555636
'consul.serf.queue.Intent': Count: 2 Sum: 0.000 LastUpdated: 2022-07-23 22:53:11.95907866 +0000 UTC m=+1980.230191228
'consul.serf.queue.Event': Count: 2 Sum: 0.000 LastUpdated: 2022-07-23 22:53:11.959109924 +0000 UTC m=+1980.230222492
'consul.runtime.gc_pause_ns': Count: 1 Sum: 85561.000 LastUpdated: 2022-07-23 22:53:15.250622957 +0000 UTC m=+1983.521735524
'consul.memberlist.gossip': Count: 70 Min: 0.006 Mean: 0.019 Max: 0.063 Stddev: 0.007 Sum: 1.304 LastUpdated: 2022-07-23 22:53:19.917414952 +0000 UTC m=+1988.188527519
'consul.serf.coordinate.adjustment-ms': Count: 12 Min: 0.130 Mean: 0.333 Max: 0.402 Stddev: 0.095 Sum: 3.995 LastUpdated: 2022-07-23 22:53:19.918126991 +0000 UTC m=+1988.189239557
'consul.memberlist.probeNode': Count: 12 Min: 0.488 Mean: 0.684 Max: 0.795 Stddev: 0.081 Sum: 8.213 LastUpdated: 2022-07-23 22:53:19.918129767 +0000 UTC m=+1988.189242334
'consul.raft.boltdb.txstats.spillTime': Count: 2 Min: 0.016 Mean: 0.018 Max: 0.019 Stddev: 0.002 Sum: 0.036 LastUpdated: 2022-07-23 22:53:16.814448764 +0000 UTC m=+1985.085561332
'consul.raft.boltdb.txstats.writeTime': Count: 2 Min: 6.659 Mean: 7.145 Max: 7.631 Stddev: 0.687 Sum: 14.290 LastUpdated: 2022-07-23 22:53:16.814451344 +0000 UTC m=+1985.085563910
'consul.serf.queue.Query': Count: 2 Sum: 0.000 LastUpdated: 2022-07-23 22:53:11.959120713 +0000 UTC m=+1980.230233281
'consul.raft.candidate.electSelf': Count: 1 Sum: 1.474 LastUpdated: 2022-07-23 22:53:13.726320218 +0000 UTC m=+1981.997432793

You can also use API to access metrics data.

$ curl "http://127.0.0.1:8500/v1/agent/metrics?pretty"
{
    "Timestamp": "2022-07-23 22:57:40 +0000 UTC",
    "Gauges": [
        {
            "Name": "consul.autopilot.failure_tolerance",
            "Value": 0,
            "Labels": {}
        },
        {
            "Name": "consul.autopilot.healthy",
            "Value": 0,
            "Labels": {}
        },
        {
            "Name": "consul.consul.members.clients",
            "Value": 0,
            "Labels": {
                "datacenter": "dc-lab-1"
            }
        },
        {
            "Name": "consul.consul.members.servers",
            "Value": 3,
            "Labels": {
                "datacenter": "dc-lab-1"
            }
        },
        {
            "Name": "consul.consul.state.config_entries",
            "Value": 0,
            "Labels": {
                "datacenter": "dc-lab-1",
                "kind": "exported-services"
            }
        },
        {
            "Name": "consul.consul.state.config_entries",
            "Value": 0,
            "Labels": {
                "datacenter": "dc-lab-1",
                "kind": "ingress-gateway"
            }
        },
        {
            "Name": "consul.consul.state.config_entries",
            "Value": 0,
            "Labels": {
                "datacenter": "dc-lab-1",
                "kind": "mesh"
            }
        },
        {
            "Name": "consul.consul.state.config_entries",
            "Value": 0,
            "Labels": {
                "datacenter": "dc-lab-1",
                "kind": "proxy-defaults"
            }
        },
        {
            "Name": "consul.consul.state.config_entries",
            "Value": 0,
            "Labels": {
                "datacenter": "dc-lab-1",
                "kind": "service-defaults"
            }
        },
        {
            "Name": "consul.consul.state.config_entries",
            "Value": 0,
            "Labels": {
                "datacenter": "dc-lab-1",
                "kind": "service-intentions"
            }
        },
        {
            "Name": "consul.consul.state.config_entries",
            "Value": 0,
            "Labels": {
                "datacenter": "dc-lab-1",
                "kind": "service-resolver"
            }
        },
        {
            "Name": "consul.consul.state.config_entries",
            "Value": 0,
            "Labels": {
                "datacenter": "dc-lab-1",
                "kind": "service-router"
            }
        },
        {
            "Name": "consul.consul.state.config_entries",
            "Value": 0,
            "Labels": {
                "datacenter": "dc-lab-1",
                "kind": "service-splitter"
            }
        },
        {
            "Name": "consul.consul.state.config_entries",
            "Value": 0,
            "Labels": {
                "datacenter": "dc-lab-1",
                "kind": "terminating-gateway"
            }
        },
        {
            "Name": "consul.consul.state.connect_instances",
            "Value": 0,
            "Labels": {
                "datacenter": "dc-lab-1",
                "kind": "connect-native"
            }
        },
        {
            "Name": "consul.consul.state.connect_instances",
            "Value": 0,
            "Labels": {
                "datacenter": "dc-lab-1",
                "kind": "connect-proxy"
            }
        },
        {
            "Name": "consul.consul.state.connect_instances",
            "Value": 0,
            "Labels": {
                "datacenter": "dc-lab-1",
                "kind": "ingress-gateway"
            }
        },
        {
            "Name": "consul.consul.state.connect_instances",
            "Value": 0,
            "Labels": {
                "datacenter": "dc-lab-1",
                "kind": "mesh-gateway"
            }
        },
        {
            "Name": "consul.consul.state.connect_instances",
            "Value": 0,
            "Labels": {
                "datacenter": "dc-lab-1",
                "kind": "terminating-gateway"
            }
        },
        {
            "Name": "consul.consul.state.kv_entries",
            "Value": 62,
            "Labels": {
                "datacenter": "dc-lab-1"
            }
        },
        {
            "Name": "consul.consul.state.nodes",
            "Value": 5,
            "Labels": {
                "datacenter": "dc-lab-1"
            }
        },
        {
            "Name": "consul.consul.state.service_instances",
            "Value": 5,
            "Labels": {
                "datacenter": "dc-lab-1"
            }
        },
        {
            "Name": "consul.consul.state.services",
            "Value": 3,
            "Labels": {
                "datacenter": "dc-lab-1"
            }
        },
        {
            "Name": "consul.raft.applied_index",
            "Value": 65554,
            "Labels": {}
        },
        {
            "Name": "consul.raft.boltdb.freePageBytes",
            "Value": 3751936,
            "Labels": {}
        },
        {
            "Name": "consul.raft.boltdb.freelistBytes",
            "Value": 7344,
            "Labels": {}
        },
        {
            "Name": "consul.raft.boltdb.numFreePages",
            "Value": 913,
            "Labels": {}
        },
        {
            "Name": "consul.raft.boltdb.numPendingPages",
            "Value": 3,
            "Labels": {}
        },
        {
            "Name": "consul.raft.boltdb.openReadTxn",
            "Value": 0,
            "Labels": {}
        },
        {
            "Name": "consul.raft.boltdb.txstats.pageAlloc",
            "Value": 11300864,
            "Labels": {}
        },
        {
            "Name": "consul.raft.boltdb.txstats.pageCount",
            "Value": 2759,
            "Labels": {}
        },
        {
            "Name": "consul.raft.last_index",
            "Value": 80560,
            "Labels": {}
        },
        {
            "Name": "consul.runtime.alloc_bytes",
            "Value": 20196600,
            "Labels": {}
        },
        {
            "Name": "consul.runtime.free_count",
            "Value": 6344014,
            "Labels": {}
        },
        {
            "Name": "consul.runtime.heap_objects",
            "Value": 77448,
            "Labels": {}
        },
        {
            "Name": "consul.runtime.malloc_count",
            "Value": 6421462,
            "Labels": {}
        },
        {
            "Name": "consul.runtime.num_goroutines",
            "Value": 84,
            "Labels": {}
        },
        {
            "Name": "consul.runtime.sys_bytes",
            "Value": 34096136,
            "Labels": {}
        },
        {
            "Name": "consul.runtime.total_gc_pause_ns",
            "Value": 14784511,
            "Labels": {}
        },
        {
            "Name": "consul.runtime.total_gc_runs",
            "Value": 137,
            "Labels": {}
        },
        {
            "Name": "consul.session_ttl.active",
            "Value": 0,
            "Labels": {}
        }
    ],
    "Points": [],
    "Counters": [
        {
            "Name": "consul.client.rpc",
            "Count": 2,
            "Rate": 0.2,
            "Sum": 2,
            "Min": 1,
            "Max": 1,
            "Mean": 1,
            "Stddev": 0,
            "Labels": {}
        },
        {
            "Name": "consul.memberlist.tcp.accept",
            "Count": 1,
            "Rate": 0.1,
            "Sum": 1,
            "Min": 1,
            "Max": 1,
            "Mean": 1,
            "Stddev": 0,
            "Labels": {}
        },
        {
            "Name": "consul.memberlist.tcp.sent",
            "Count": 1,
            "Rate": 80.5,
            "Sum": 805,
            "Min": 805,
            "Max": 805,
            "Mean": 805,
            "Stddev": 0,
            "Labels": {}
        },
        {
            "Name": "consul.memberlist.udp.received",
            "Count": 24,
            "Rate": 345.6,
            "Sum": 3456,
            "Min": 104,
            "Max": 181,
            "Mean": 144,
            "Stddev": 38.1051177665153,
            "Labels": {}
        },
        {
            "Name": "consul.memberlist.udp.sent",
            "Count": 24,
            "Rate": 345.6,
            "Sum": 3456,
            "Min": 104,
            "Max": 181,
            "Mean": 144,
            "Stddev": 38.1051177665153,
            "Labels": {}
        },
        {
            "Name": "consul.raft.boltdb.totalReadTxn",
            "Count": 2,
            "Rate": 0.1,
            "Sum": 1,
            "Min": 0,
            "Max": 1,
            "Mean": 0.5,
            "Stddev": 0.7071067811865476,
            "Labels": {}
        },
        {
            "Name": "consul.raft.boltdb.txstats.cursorCount",
            "Count": 2,
            "Rate": 2,
            "Sum": 20,
            "Min": 9,
            "Max": 11,
            "Mean": 10,
            "Stddev": 1.4142135623730951,
            "Labels": {}
        },
        {
            "Name": "consul.raft.boltdb.txstats.nodeCount",
            "Count": 2,
            "Rate": 1.2,
            "Sum": 12,
            "Min": 6,
            "Max": 6,
            "Mean": 6,
            "Stddev": 0,
            "Labels": {}
        },
        {
            "Name": "consul.raft.boltdb.txstats.nodeDeref",
            "Count": 2,
            "Rate": 0,
            "Sum": 0,
            "Min": 0,
            "Max": 0,
            "Mean": 0,
            "Stddev": 0,
            "Labels": {}
        },
        {
            "Name": "consul.raft.boltdb.txstats.rebalance",
            "Count": 2,
            "Rate": 0,
            "Sum": 0,
            "Min": 0,
            "Max": 0,
            "Mean": 0,
            "Stddev": 0,
            "Labels": {}
        },
        {
            "Name": "consul.raft.boltdb.txstats.spill",
            "Count": 2,
            "Rate": 0.6,
            "Sum": 6,
            "Min": 3,
            "Max": 3,
            "Mean": 3,
            "Stddev": 0,
            "Labels": {}
        },
        {
            "Name": "consul.raft.boltdb.txstats.split",
            "Count": 2,
            "Rate": 0,
            "Sum": 0,
            "Min": 0,
            "Max": 0,
            "Mean": 0,
            "Stddev": 0,
            "Labels": {}
        },
        {
            "Name": "consul.raft.boltdb.txstats.write",
            "Count": 2,
            "Rate": 1.8,
            "Sum": 18,
            "Min": 9,
            "Max": 9,
            "Mean": 9,
            "Stddev": 0,
            "Labels": {}
        },
        {
            "Name": "consul.raft.state.candidate",
            "Count": 1,
            "Rate": 0.1,
            "Sum": 1,
            "Min": 1,
            "Max": 1,
            "Mean": 1,
            "Stddev": 0,
            "Labels": {}
        },
        {
            "Name": "consul.rpc.accept_conn",
            "Count": 19,
            "Rate": 1.9,
            "Sum": 19,
            "Min": 1,
            "Max": 1,
            "Mean": 1,
            "Stddev": 0,
            "Labels": {}
        }
    ],
    "Samples": [
        {
            "Name": "consul.acl.ResolveToken",
            "Count": 1,
            "Rate": 0.004297799989581108,
            "Sum": 0.04297799989581108,
            "Min": 0.04297799989581108,
            "Max": 0.04297799989581108,
            "Mean": 0.04297799989581108,
            "Stddev": 0,
            "Labels": {}
        },
        {
            "Name": "consul.api.http",
            "Count": 1,
            "Rate": 0.05300869941711426,
            "Sum": 0.5300869941711426,
            "Min": 0.5300869941711426,
            "Max": 0.5300869941711426,
            "Mean": 0.5300869941711426,
            "Stddev": 0,
            "Labels": {
                "method": "GET",
                "path": "v1_agent_metrics"
            }
        },
        {
            "Name": "consul.api.http",
            "Count": 1,
            "Rate": 1246.4619140625,
            "Sum": 12464.619140625,
            "Min": 12464.619140625,
            "Max": 12464.619140625,
            "Mean": 12464.619140625,
            "Stddev": 0,
            "Labels": {
                "method": "GET",
                "path": "v1_agent_monitor"
            }
        },
        {
            "Name": "consul.memberlist.gossip",
            "Count": 70,
            "Rate": 0.1340316001791507,
            "Sum": 1.340316001791507,
            "Min": 0.0066559999249875546,
            "Max": 0.03837500140070915,
            "Mean": 0.019147371454164386,
            "Stddev": 0.005150694713146948,
            "Labels": {}
        },
        {
            "Name": "consul.memberlist.probeNode",
            "Count": 12,
            "Rate": 0.8715510934591293,
            "Sum": 8.715510934591293,
            "Min": 0.4353649914264679,
            "Max": 0.8789029717445374,
            "Mean": 0.7262925778826078,
            "Stddev": 0.1086966914278882,
            "Labels": {}
        },
        {
            "Name": "consul.raft.boltdb.txstats.rebalanceTime",
            "Count": 2,
            "Rate": 0,
            "Sum": 0,
            "Min": 0,
            "Max": 0,
            "Mean": 0,
            "Stddev": 0,
            "Labels": {}
        },
        {
            "Name": "consul.raft.boltdb.txstats.spillTime",
            "Count": 2,
            "Rate": 0.005971099808812141,
            "Sum": 0.059710998088121414,
            "Min": 0.01712699979543686,
            "Max": 0.042583998292684555,
            "Mean": 0.029855499044060707,
            "Stddev": 0.018000816266059597,
            "Labels": {}
        },
        {
            "Name": "consul.raft.boltdb.txstats.writeTime",
            "Count": 2,
            "Rate": 1.4999948024749756,
            "Sum": 14.999948024749756,
            "Min": 7.315351963043213,
            "Max": 7.684596061706543,
            "Mean": 7.499974012374878,
            "Stddev": 0.2610950060779553,
            "Labels": {}
        },
        {
            "Name": "consul.raft.candidate.electSelf",
            "Count": 1,
            "Rate": 0.14611899852752686,
            "Sum": 1.4611899852752686,
            "Min": 1.4611899852752686,
            "Max": 1.4611899852752686,
            "Mean": 1.4611899852752686,
            "Stddev": 0,
            "Labels": {}
        },
        {
            "Name": "consul.serf.coordinate.adjustment-ms",
            "Count": 12,
            "Rate": 0.2115220993757248,
            "Sum": 2.115220993757248,
            "Min": 0.13530999422073364,
            "Max": 0.24645300209522247,
            "Mean": 0.17626841614643732,
            "Stddev": 0.035050204255583875,
            "Labels": {}
        },
        {
            "Name": "consul.serf.queue.Event",
            "Count": 2,
            "Rate": 0,
            "Sum": 0,
            "Min": 0,
            "Max": 0,
            "Mean": 0,
            "Stddev": 0,
            "Labels": {}
        },
        {
            "Name": "consul.serf.queue.Intent",
            "Count": 2,
            "Rate": 0,
            "Sum": 0,
            "Min": 0,
            "Max": 0,
            "Mean": 0,
            "Stddev": 0,
            "Labels": {}
        },
        {
            "Name": "consul.serf.queue.Query",
            "Count": 2,
            "Rate": 0,
            "Sum": 0,
            "Min": 0,
            "Max": 0,
            "Mean": 0,
            "Stddev": 0,
            "Labels": {}
        }
    ]
}

This format makes it easy to parse.

$ curl --silent "http://127.0.0.1:8500/v1/agent/metrics?pretty" | \
  jq '.Gauges[] | select(.Name=="consul.consul.members.servers") | .Value'
3