Merge multi-process HAProxy statistics using Lua.
Inspect operating system version.
$ lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 10 (buster) Release: 10 Codename: buster
Install HAProxy.
$ sudo apt install haproxy
Inspect the HAProxy version and Lua support.
$ /sbin/haproxy -vv HA-Proxy version 1.8.19-1 2019/02/12 Copyright 2000-2019 Willy Tarreau <willy@haproxy.org> Build options : TARGET = linux2628 CPU = generic CC = gcc CFLAGS = -O2 -g -O2 -fdebug-prefix-map=/build/haproxy-1.8.19=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-format-truncation -Wno-null-dereference -Wno-unused-label OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_SYSTEMD=1 USE_PCRE2=1 USE_PCRE2_JIT=1 USE_NS=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 Built with OpenSSL version : OpenSSL 1.1.1a 20 Nov 2018 Running on OpenSSL version : OpenSSL 1.1.1d 10 Sep 2019 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3 Built with Lua version : Lua 5.3.3 Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Encrypted password support via crypt(3): yes Built with multi-threading support. Built with PCRE2 version : 10.32 2018-09-10 PCRE2 library supports JIT : yes Built with zlib version : 1.2.11 Running on zlib version : 1.2.11 Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip") Built with network namespace support. Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. Available filters : [SPOE] spoe [COMP] compression [TRACE] trace
Create <a href="https://www.arpalert.org/haproxy-scripts.html#stats" target="_blank" rel="external noopener noreferrer">/etc/haproxy/stats.lua</a>
Lua script. I have updated it, so it works for me.
stats = {} -- -- Configuration -- -- stats.conf = {} -- Lists all known haproxy, the index is the friendly -- name of the instance. stats.update_interval = 1 -- update interval -- -- internal shared vars --- stats.agg = {} -- contains the aggregated data stats.servers = {} -- List of the last data retrieved by server stats.lookup = {} -- contains a lookup for used keys. --[[ HAProxy stats compatibility array. All the metrics are not used with all the parts (backend, frontend, servers or listener. This array permit to give a useless element, and gives an aggregation method. 1 : type 0: frontend 2 : type 1: backend 3 : type 2: server 4 : type 3: listener 5 : type of data: "string" : string. "gauge" : integer, value. "counter" : integer, always monotonic, increase with the time spent. "time" : integer, time elapsed. 6 : aggregation type: "first" : keep the first value "add" : add all values "cat" : concatenate the values, separated by a "/" "avg" : perform the average of the values ]]-- stats.values = { -- 1 2 3 4 5 6 ["pxname"] = { "1", "1", "1", "1", "string", "first"}, ["svname"] = { "1", "1", "1", "1", "string", "first"}, ["qcur"] = { " ", "1", "1", " ", "gauge", "add" }, ["qmax"] = { " ", "1", "1", " ", "gauge", "add" }, ["scur"] = { "1", "1", "1", "1", "gauge", "add" }, ["smax"] = { "1", "1", "1", "1", "gauge", "add" }, ["slim"] = { "1", "1", "1", "1", "gauge", "add" }, ["stot"] = { "1", "1", "1", "1", "counter", "add" }, ["bin"] = { "1", "1", "1", "1", "counter", "add" }, ["bout"] = { "1", "1", "1", "1", "counter", "add" }, ["dreq"] = { "1", "1", " ", "1", "counter", "add" }, ["dresp"] = { "1", "1", "1", "1", "counter", "add" }, ["ereq"] = { "1", " ", " ", "1", "counter", "add" }, ["econ"] = { " ", "1", "1", " ", "counter", "add" }, ["eresp"] = { " ", "1", "1", " ", "counter", "add" }, ["wretr"] = { " ", "1", "1", " ", "counter", "add" }, ["wredis"] = { " ", "1", "1", " ", "counter", "add" }, ["status"] = { "1", "1", "1", "1", "string", "cat" }, ["weight"] = { " ", "1", "1", " ", "gauge", "add" }, -- maybe more complex ["act"] = { " ", "1", "1", " ", "gauge", "add" }, ["bck"] = { " ", "1", "1", " ", "gauge", "add" }, ["chkfail"] = { " ", " ", "1", " ", "counter", "add" }, ["chkdown"] = { " ", "1", "1", " ", "counter", "add" }, ["lastchg"] = { " ", "1", "1", " ", "time", "avg" }, ["downtime"] = { " ", "1", "1", " ", "time", "avg" }, ["qlimit"] = { " ", " ", "1", " ", "gauge", "add" }, ["pid"] = { "1", "1", "1", "1", "gauge", "cat" }, ["iid"] = { "1", "1", "1", "1", "gauge", "cat" }, ["sid"] = { " ", " ", "1", "1", "gauge", "cat" }, ["throttle"] = { " ", " ", "1", " ", "gauge", "avg" }, ["lbtot"] = { " ", "1", "1", " ", "counter", "add" }, ["tracked"] = { " ", " ", "1", " ", "counter", "add" }, ["type"] = { "1", "1", "1", "1", "gauge", "first"}, ["rate"] = { "1", "1", "1", " ", "gauge", "add" }, ["rate_lim"] = { "1", " ", " ", " ", "gauge", "add" }, ["rate_max"] = { "1", "1", "1", " ", "gauge", "add" }, ["check_status"] = { " ", " ", "1", " ", "string", "cat" }, ["check_code"] = { " ", " ", "1", " ", "string", "cat" }, ["check_duration"]= { " ", " ", "1", " ", "time", "cat" }, ["hrsp_1xx"] = { "1", "1", "1", " ", "counter", "add" }, ["hrsp_2xx"] = { "1", "1", "1", " ", "counter", "add" }, ["hrsp_3xx"] = { "1", "1", "1", " ", "counter", "add" }, ["hrsp_4xx"] = { "1", "1", "1", " ", "counter", "add" }, ["hrsp_5xx"] = { "1", "1", "1", " ", "counter", "add" }, ["hrsp_other"] = { "1", "1", "1", " ", "counter", "add" }, ["hanafail"] = { " ", " ", "1", " ", "string", "cat" }, ["req_rate"] = { "1", " ", " ", " ", "gauge", "add" }, ["req_rate_max"] = { "1", " ", " ", " ", "gauge", "add" }, ["req_tot"] = { "1", " ", " ", " ", "counter", "add" }, ["cli_abrt"] = { " ", "1", "1", " ", "counter", "add" }, ["srv_abrt"] = { " ", "1", "1", " ", "counter", "add" }, ["comp_in"] = { "1", "1", " ", " ", "counter", "add" }, ["comp_out"] = { "1", "1", " ", " ", "counter", "add" }, ["comp_byp"] = { "1", "1", " ", " ", "counter", "add" }, ["comp_rsp"] = { "1", "1", " ", " ", "counter", "add" }, ["lastsess"] = { " ", "1", "1", " ", "time", "avg" }, ["last_chk"] = { " ", " ", "1", " ", "string", "cat" }, ["last_agt"] = { " ", " ", "1", " ", "string", "cat" }, ["qtime"] = { " ", "1", "1", " ", "gauge", "avg" }, ["ctime"] = { " ", "1", "1", " ", "gauge", "avg" }, ["rtime"] = { " ", "1", "1", " ", "gauge", "avg" }, ["ttime"] = { " ", "1", "1", " ", "gauge", "avg" }, ["agg"] = { "1", "1", "1", "1", "string", "cat" }, } -- csv header order stats.csv = { "pxname", "svname", "qcur", "qmax", "scur", "smax", "slim", "stot", "bin", "bout", "dreq", "dresp", "ereq", "econ", "eresp", "wretr", "wredis", "status", "weight", "act", "bck", "chkfail", "chkdown", "lastchg", "downtime", "qlimit", "pid", "iid", "sid", "throttle", "lbtot", "tracked", "type", "rate", "rate_lim", "rate_max", "check_status", "check_code", "check_duration", "hrsp_1xx", "hrsp_2xx", "hrsp_3xx", "hrsp_4xx", "hrsp_5xx", "hrsp_other", "hanafail", "req_rate", "req_rate_max", "req_tot", "cli_abrt", "srv_abrt", "comp_in", "comp_out", "comp_byp", "comp_rsp", "lastsess", "last_chk", "last_agt", "qtime", "ctime", "rtime", "ttime", "agg" } -- -- Code extracted from http://lua-users.org/wiki/BaseSixtyFour -- -- character table string local b='ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/' -- encoding function b64enc(data) return ((data:gsub('.', function(x) local r,b='',x:byte() for i=8,1,-1 do r=r..(b%2^i-b%2^(i-1)>0 and '1' or '0') end return r; end)..'0000'):gsub('%d%d%d?%d?%d?%d?', function(x) if (#x < 6) then return '' end local c=0 for i=1,6 do c=c+(x:sub(i,i)=='1' and 2^(6-i) or 0) end return b:sub(c+1,c+1) end)..({ '', '==', '=' })[#data%3+1]) end --[[ get_stats(uri, mode) uri is the location of the required resources. It understand the following forms: * http://[user:password@]<ip>[:<port>]/<path> * socket://<path> This function returns a hash table with the frontends ans backend hierarchically sorted. In error case, send an error using the"error" function and doesn't returns. ]]-- stats.get_stats = function(uri) local pathpos local path local creds = "" local addr local msg local s local csv = {} local line = {} local pos1 local pos2 local typeidx local typepx local res local i local t -- uri must start with http:// or socket:// if string.sub(uri, 1, 7) == "http://" then mode = 1 addr = string.sub(uri, 8) elseif string.sub(uri, 1, 9) == "socket://" then mode = 2 addr = string.sub(uri, 10) else error("'uri' must starts with 'http://', we found '" .. tostring(string.sub(uri, 1, 7)) .. "'") end -- -- mode 1: sends http request -- if mode == 1 then -- separates ip:port from the path local pathpos = string.find(addr, "/") if (pathpos == nil) then path = "/" else path = string.sub(addr, pathpos) end addr = string.sub(addr, 1, pathpos - 1) -- extract login / password local sep = string.find(addr, "@") if sep ~= nil then creds = string.sub(addr, 1, sep - 1) addr = string.sub(addr, sep + 1, -1) -- build header creds = "Authorization: Basic " .. b64enc(creds) .. "\r\n" end -- send request s = core.tcp() s:connect(addr) s:send("GET " .. path .. " HTTP/1.0\r\nHost: " .. addr .. "\r\n" .. creds .. "\r\n"); -- read response, remove headers msg = s:receive("*l") if msg == nil then -- or not string.match(msg,"HTTP/1.1 200 OK") then s:close() return nil end if not string.match(msg,"HTTP/1.1 200 OK") then s:close() return nil end while true do msg = s:receive("*l") if msg == nil then s:close() return nil end msg = string.gsub(msg, "\r", "") msg = string.gsub(msg, "\n", "") if msg == "" then break end end -- if string.len(msg) == 0 then -- s:close() -- return nil -- end -- read CSV msg = s:receive("*a") if msg == nil then s:close() return nil end -- close connection s:close() -- -- mode 2: sens stats request -- elseif mode == 2 then -- send request s = core.tcp() s:connect("unix@" .. addr) s:send("show stat -1 15 -1\n"); -- read CSV msg = s:receive("*a") if msg == nil then s:close() return nil end -- close connection s:close() end -- removes \r msg = string.gsub(msg, "\r", "") while true do -- read first value. expects if string.sub(msg, 1, 1) == '"' then -- read a simple value terminated by '"', escapes the '""'. else -- read a simple value terminated by "," or "\n" pos1 = string.find(msg, ",") pos2 = string.find(msg, "\n") if pos1 == nil then pos1 = string.len(msg) end if pos2 == nil then pos2 = string.len(msg) end if pos2 < pos1 then pos1 = pos2 end table.insert(line, string.sub(msg, 1, pos1 - 1)) -- if the separator is a newline if string.sub(msg, pos1, pos1) == "\n" then table.insert(csv, line) line = {} end msg = string.sub(msg, pos1 + 1) -- eat the cell end -- break process if the csv message is empty if msg == "" then break end end -- transform value array to object array local hdr = {} local obj = {} for i, t in pairs(csv) do if type(t[1]) == "string" and t[1] == "# pxname" then t[1] = "pxname" hdr = t -- look for the type idx for i,v in pairs(hdr) do if v == "type" then typeidx = i break end end else -- create new array that is a mix between -- column name and values. Clean the useless -- values. local h = {} if t[typeidx] ~= nil then typepx = tonumber(t[typeidx]) + 1 for i,v in pairs(t) do if hdr[i] ~= "" then if stats.values[hdr[i]] ~= nil then if stats.values[hdr[i]][typepx] == "1" then if stats.values[hdr[i]][5] == "gauge" or stats.values[hdr[i]][5] == "counter" or stats.values[hdr[i]][5] == "time" then h[hdr[i]] = tonumber(v) else h[hdr[i]] = v end end end end end end table.insert(obj, h) end end -- define the array object res = {} res['frontends'] = {} res['backends'] = {} -- process all the frontends for i, t in pairs(obj) do if t["type"] == 0 then -- frontend res['frontends'][t["pxname"]] = t res['frontends'][t["pxname"]]["listeners"] = {} elseif t["type"] == 1 then -- backend res['backends'][t["pxname"]] = t res['backends'][t["pxname"]]["servers"] = {} end end -- process all the backends for i, t in pairs(obj) do if t["type"] == 2 then -- server res['backends'][t["pxname"]]["servers"][t["svname"]] = t elseif t["type"] == 3 then -- listener res['frontends'][t["pxname"]]["listeners"][t["svname"]] = t end end return res end stats.merge_agg = function(stats, mode) local k local v local res local c -- -- first -- if mode[6] == "first" then for k, v in pairs(stats) do res = stats[k] break end -- -- add -- elseif mode[6] == "add" then res = 0 for k, v in pairs(stats) do res = res + stats[k] end -- -- concatenate -- elseif mode[6] == "cat" then res = "" for k, v in pairs(stats) do if res == "" then res = tostring(stats[k]) else res = res .. "/" .. tostring(stats[k]) end end -- -- average -- elseif mode[6] == "avg" then res = 0 c = 0 for k, v in pairs(stats) do res = res + stats[k] c = c + 1 end if c > 0 then res = res / c end end return res; end stats.merge_stats = function(stats_array) local res; local list; local k local v local val local srv local frt local bck local lis local stat -- destination struct res = {} res['frontends'] = {} res['backends'] = {} -- fill the destination for srv, stat in pairs(stats_array) do if stat['frontends'] ~= nil then for frt, val in pairs(stat['frontends']) do if res['frontends'][frt] == nil then res['frontends'][frt] = {} res['frontends'][frt]['listeners'] = {} end if stat['frontends'][frt]['listeners'] ~= nil then for lis, val in pairs(stat['frontends'][frt]['listeners']) do if res['frontends'][frt] == nil then res['frontends'][frt] = {} end if res['frontends'][frt]['listeners'] == nil then res['frontends'][frt]['listeners'] = {} end if res['frontends'][frt]['listeners'][lis] == nil then res['frontends'][frt]['listeners'][lis] = {} end end end end end if stat['backends'] ~= nil then for frt, val in pairs(stat['backends']) do if res['backends'][frt] == nil then res['backends'][frt] = {} res['backends'][frt]['servers'] = {} end if stat['backends'][frt]['servers'] ~= nil then for srv, val in pairs(stat['backends'][frt]['servers']) do if res['backends'][frt] == nil then res['backends'][frt] = {} end if res['backends'][frt]['servers'] == nil then res['backends'][frt]['servers'] = {} end if res['backends'][frt]['servers'][srv] == nil then res['backends'][frt]['servers'][srv] = {} end end end end end end -- data aggregation -- -- frontends -- for frt, val in pairs(res['frontends']) do for prop, mode in pairs(stats.values) do if mode[1] == "1" then -- build list of frontends list = {} for k, v in pairs(stats_array) do if stats_array[k]['frontends'] and stats_array[k]['frontends'][frt] and stats_array[k]['frontends'][frt][prop] then list[k] = stats_array[k]['frontends'][frt][prop] end end -- aggregation values res['frontends'][frt][prop] = stats.merge_agg(list, mode) end end list = "" for k, v in pairs(stats_array) do if stats_array[k]['frontends'][frt] ~= nil then if list == "" then list = k else list = list .. "/" .. k end end end res['frontends'][frt]["agg"] = list -- -- listeners -- for lis, val in pairs(res['frontends'][frt]['listeners']) do for prop, mode in pairs(stats.values) do if mode[4] == "1" then -- build list of frontends list = {} for k, v in pairs(stats_array) do if v['frontends'][frt] and v['frontends'][frt]['listeners'] and v['frontends'][frt]['listeners'][lis] and v['frontends'][frt]['listeners'][lis][prop] then list[k] = v['frontends'][frt]['listeners'][lis][prop] end end -- aggregation values res['frontends'][frt]['listeners'][lis][prop] = stats.merge_agg(list, mode) end end list = "" for k, v in pairs(stats_array) do if stats_array[k]['frontends'][frt] and stats_array[k]['frontends'][frt]['listeners'] and stats_array[k]['frontends'][frt]['listeners'][lis] then if list == "" then list = k else list = list .. "/" .. k end end end res['frontends'][frt]['listeners'][lis]["agg"] = list end end -- -- backends -- for bck, val in pairs(res['backends']) do for prop, mode in pairs(stats.values) do if mode[2] == "1" then -- build list of frontends list = {} for k, v in pairs(stats_array) do if stats_array[k]['backends'] and stats_array[k]['backends'][bck] and stats_array[k]['backends'][bck][prop] then list[k] = stats_array[k]['backends'][bck][prop] end end -- aggregation values res['backends'][bck][prop] = stats.merge_agg(list, mode) end end list = "" for k, v in pairs(stats_array) do if stats_array[k]['backends'][bck] ~= nil then if list == "" then list = k else list = list .. "/" .. k end end end res['backends'][bck]["agg"] = list -- -- servers -- for srv, val in pairs(res['backends'][bck]['servers']) do for prop, mode in pairs(stats.values) do if mode[3] == "1" then -- build list of frontends list = {} for k, v in pairs(stats_array) do if v['backends'][bck] and v['backends'][bck]['servers'] and v['backends'][bck]['servers'][srv] and v['backends'][bck]['servers'][srv][prop] then list[k] = v['backends'][bck]['servers'][srv][prop] end end -- aggregation values res['backends'][bck]['servers'][srv][prop] = stats.merge_agg(list, mode) end end list = "" for k, v in pairs(stats_array) do if stats_array[k]['backends'][bck]['servers'][srv] ~= nil then if list == "" then list = k else list = list .. "/" .. k end end end res['backends'][bck]['servers'][srv]["agg"] = list end end return res end stats.tocsv = function(msg) if msg == nil then return "" end return tostring(msg) end -- This function is an applet that accept and HTTP request -- and return a aggregated CSV stats.csv_service = function(applet) local frt local bck local srv local lis local val1 local val2 local k local i local msg msg = "# " for i, k in ipairs(stats.csv) do msg = msg .. k .. "," end msg = msg .. "\n" for frt, val1 in pairs(stats.agg['frontends']) do for i, k in ipairs(stats.csv) do msg = msg .. stats.tocsv(val1[k]) .. "," end msg = msg .. "\n" for lis, val2 in pairs(val1['listeners']) do for i, k in ipairs(stats.csv) do msg = msg .. stats.tocsv(val2[k]) .. "," end msg = msg .. "\n" end end for bck, val1 in pairs(stats.agg['backends']) do for i, k in ipairs(stats.csv) do msg = msg .. stats.tocsv(val1[k]) .. "," end msg = msg .. "\n" for srv, val2 in pairs(val1['servers']) do for i, k in ipairs(stats.csv) do msg = msg .. stats.tocsv(val2[k]) .. "," end msg = msg .. "\n" end end applet:add_header("content-length", tostring(msg:len())); applet:set_status(200) applet:start_response() applet:send(msg) end -- The following function gets and merge all the known haproxy -- stats each second. stats.update = function() local k local v local vals local srv = {} while true do for k, v in pairs(stats.conf) do vals = stats.get_stats(v) if vals ~= nil then srv[k] = vals end end local g = stats.merge_stats(srv) stats.servers = srv; stats.agg = g core.sleep(stats.update_interval) end end -- register general fetches to get the merged values -- syntax is: -- -- ["f" | "b"] ":" <fe/be name> ":" <metrix> -- ["f" | "b"] ":" <fe/be name> ":" [ <server> | <listener> ] ":" <metrix> -- -- example: -- -- f:bck_webfarm:qcur -- f:bck_webfarm:server1:qcur -- stats.process_key = function(key) local path = {} local p = key local pos local febe = 0 if stats.lookup[key] ~= nil then return stats.lookup[key] end -- must start with "f:" or "b:" if p:sub(1, 2) == "f:" then table.insert(path, "frontends") febe = 0 elseif p:sub(1, 2) == "b:" then table.insert(path, "backends") febe = 1 else error("the agg key '" .. key .. "' is invalid") end p = p:sub(3) -- look for the next part, search a mandatory ":" pos = p:find(":") if pos == nil then error("the agg key '" .. key .. "' is invalid") end table.insert(path, p:sub(1, pos - 1)) p = p:sub(pos + 1) -- look for the next part, search for a ":" or the end. pos = p:find(":") if pos == nil then table.insert(path, p) stats.lookup[key] = path return path end if febe == 0 then table.insert(path, "listeners") else table.insert(path, "servers") end table.insert(path, p:sub(1, pos - 1)) p = p:sub(pos + 1) -- stores the last part table.insert(path, p:sub(1, -1)) -- index and return the result stats.lookup[key] = path return path end stats.fetch = function(txn, name) local w local path local i local v local tab -- perform a lookup path = stats.process_key(name) tab = stats.agg for i, v in pairs(path) do if tab[v] == nil then error("key '" .. name .. "' doesn't exists (or not yet exists)") end tab = tab[v] end return tab end -- ["h1"] = "http://127.0.0.1:10002/;csv", -- ["h2"] = "socket:///tmp/haproxy", stats.add = function(name, uri) stats.conf[name] = uri end -- change update interval stats.interval = function(interval) stats.update_interval = interval end core.register_service("stats", "http", stats.csv_service) core.register_fetches("stats", stats.fetch) core.register_task(stats.update)
Create /etc/haproxy/stats-config.lua
Lua configuration script.
stats.add("statistics-cpu-1", "http://stats:stats@127.0.0.1:7001/statistics;csv") stats.add("statistics-cpu-2", "http://stats:stats@127.0.0.1:7002/statistics;csv") stats.add("statistics-cpu-3", "http://stats:stats@127.0.0.1:7003/statistics;csv") stats.add("statistics-cpu-4", "http://stats:stats@127.0.0.1:7004/statistics;csv") stats.interval(5) -- calculate interval every n-seconds
Create the HAProxy configuration.
global log /dev/log local0 log /dev/log local1 notice chroot /var/lib/haproxy stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners stats timeout 30s user haproxy group haproxy daemon # Default SSL material locations ca-base /etc/ssl/certs crt-base /etc/ssl/private # Default ciphers to use on SSL-enabled listening sockets. # For more information, see ciphers(1SSL). This list is from: # https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/ # An alternative list with additional directives can be obtained from # https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS ssl-default-bind-options no-sslv3 nbproc 4 cpu-map auto:1-4 0-3 lua-load /etc/haproxy/stats.lua lua-load /etc/haproxy/stats-config.lua defaults log global mode http option httplog option dontlognull timeout connect 5000 timeout client 50000 timeout server 50000 errorfile 400 /etc/haproxy/errors/400.http errorfile 403 /etc/haproxy/errors/403.http errorfile 408 /etc/haproxy/errors/408.http errorfile 500 /etc/haproxy/errors/500.http errorfile 502 /etc/haproxy/errors/502.http errorfile 503 /etc/haproxy/errors/503.http errorfile 504 /etc/haproxy/errors/504.http userlist statistics-auth-list user stats insecure-password stats frontend sample-frontend bind :80 mode http default_backend sample-backend backend sample-backend option tcp-check server backend-server-1 192.168.88.1:80 check server backend-server-2 192.168.88.2:80 check listen statistics-1 bind 127.0.0.1:7001 mode http bind-process 1 stats enable stats realm Statistics stats uri /statistics stats show-legends # stats auth stats:stats acl statistics-auth http_auth(statistics-auth-list) http-request auth realm statistics unless statistics-auth listen statistics-2 bind 127.0.0.1:7002 mode http bind-process 2 stats enable stats realm Statistics stats uri /statistics stats show-legends # stats auth stats:stats acl statistics-auth http_auth(statistics-auth-list) http-request auth realm statistics unless statistics-auth listen statistics-3 bind 127.0.0.1:7003 mode http bind-process 3 stats enable stats realm Statistics stats uri /statistics stats show-legends # stats auth stats:stats acl statistics-auth http_auth(statistics-auth-list) http-request auth realm statistics unless statistics-auth listen statistics-4 bind 127.0.0.1:7004 mode http bind-process 4 stats enable stats realm Statistics stats uri /statistics stats show-legends # stats auth stats:stats acl statistics-auth http_auth(statistics-auth-list) http-request auth realm statistics unless statistics-auth listen statistics-0 bind 127.0.0.1:7000 mode http acl statistics-auth http_auth(statistics-auth-list) http-request auth realm statistics unless statistics-auth http-request use-service lua.stats if { path /statistics;csv }
Reload HAProxy service.
$ sudo systemctl reload haproxy
Verify configuration.
$ for p in $(seq 7000 7004); do echo; echo "Port ${p}"; curl -u stats:stats "http://127.0.0.1:${p}/statistics;csv" | sort -t, -k 1,2; done
Port 7000 # pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,agg, sample-backend,BACKEND,0,0,0,6,800,15,4630,90247,0,0,,0,0,0,0,UP/UP/UP/UP,4,4,0,,0,878.0,0.0,,3/4/1/2,3/3/3/3,,,15,,1,0,,13,,,,0,14,0,1,0,0,,,,,0,0,0,0,0,0,331.0,,,0.0,0.25,0.75,4.0,statistics-cpu-1/statistics-cpu-4/statistics-cpu-2/statistics-cpu-3, sample-backend,backend-server-1,0,0,0,6,0,15,4630,90247,,0,,0,0,0,0,UP/UP/UP/UP,4,4,0,0,0,878.0,0.0,0,3/4/1/2,3/3/3/3,1/1/1/1,0,15,0,2,0,,13,L4OK/L4OK/L4OK/L4OK,,2/3/2/3,0,14,0,1,0,0,,,,,0,0,,,,,331.0,,,0.0,0.25,0.75,4.0,statistics-cpu-1/statistics-cpu-4/statistics-cpu-2/statistics-cpu-3, sample-backend,backend-server-2,0,0,0,0,0,0,0,0,,0,,0,0,0,0,DOWN/DOWN/DOWN/DOWN,4,4,0,4,4,876.0,876.0,0,3/4/1/2,3/3/3/3,2/2/2/2,0,0,0,2,0,,0,L4CON/L4CON/L4CON/L4CON,,1070/1070/1070/1070,0,0,0,0,0,0,,,,,0,0,,,,,-1.0,No route to host at initial connection step of tcp-check/No route to host at initial connection step of tcp-check/No route to host at initial connection step of tcp-check/No route to host at initial connection step of tcp-check,,0.0,0.0,0.0,0.0,statistics-cpu-1/statistics-cpu-4/statistics-cpu-2/statistics-cpu-3, sample-frontend,FRONTEND,,,0,6,8000,6,4630,90247,0,0,0,,,,,OPEN/OPEN/OPEN/OPEN,,,,,,,,,3/4/1/2,2/2/2/2,,,,,0,0,0,6,,,,0,14,0,1,0,0,,0,13,15,,,0,0,0,0,,,,,,,,statistics-cpu-1/statistics-cpu-4/statistics-cpu-2/statistics-cpu-3, statistics-0,BACKEND,0,0,0,0,800,0,1441,38113,0,0,,0,0,0,0,UP/UP/UP/UP,0,0,0,,0,878.0,0,,3/4/1/2,8/8/8/8,,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,,0,0,0,0,0,0,39.0,,,0.0,0.0,1.0,1.0,statistics-cpu-1/statistics-cpu-4/statistics-cpu-2/statistics-cpu-3, statistics-0,FRONTEND,,,0,4,8000,11,1441,38113,0,0,0,,,,,OPEN/OPEN/OPEN/OPEN,,,,,,,,,3/4/1/2,8/8/8/8,,,,,0,0,0,4,,,,0,11,0,0,0,0,,0,4,11,,,0,0,0,0,,,,,,,,statistics-cpu-1/statistics-cpu-4/statistics-cpu-2/statistics-cpu-3, statistics-1,BACKEND,0,0,0,0,200,0,66596,1574952,0,0,,0,0,0,0,UP,0,0,0,,0,878.0,0,,1,4,,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,,0,0,0,0,0,0,0.0,,,0.0,0.0,1.0,1.0,statistics-cpu-1, statistics-1,FRONTEND,,,1,4,2000,713,66596,1574952,0,0,0,,,,,OPEN,,,,,,,,,1,4,,,,,0,1,0,6,,,,0,712,0,0,0,0,,1,6,713,,,0,0,0,0,,,,,,,,statistics-cpu-1, statistics-2,BACKEND,0,0,0,0,200,0,66596,1576350,0,0,,0,0,0,0,UP,0,0,0,,0,878.0,0,,2,5,,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,,0,0,0,0,0,0,0.0,,,0.0,0.0,0.0,1.0,statistics-cpu-2, statistics-2,FRONTEND,,,2,4,2000,714,66596,1576350,0,0,0,,,,,OPEN,,,,,,,,,2,5,,,,,0,2,0,6,,,,0,712,0,0,0,0,,1,6,713,,,0,0,0,0,,,,,,,,statistics-cpu-2, statistics-3,BACKEND,0,0,0,0,200,0,66410,1566594,0,0,,0,0,0,0,UP,0,0,0,,0,878.0,0,,3,6,,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,,0,0,0,0,0,0,0.0,,,0.0,0.0,0.0,1.0,statistics-cpu-3, statistics-3,FRONTEND,,,2,4,2000,712,66410,1566594,0,0,0,,,,,OPEN,,,,,,,,,3,6,,,,,0,2,0,5,,,,0,710,0,0,0,0,,2,5,712,,,0,0,0,0,,,,,,,,statistics-cpu-3, statistics-4,BACKEND,0,0,0,0,200,0,66410,1572072,0,0,,0,0,0,0,UP,0,0,0,,0,878.0,0,,4,7,,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,,0,0,0,0,0,0,0.0,,,0.0,0.0,0.0,0.0,statistics-cpu-4, statistics-4,FRONTEND,,,2,4,2000,712,66410,1572072,0,0,0,,,,,OPEN,,,,,,,,,4,7,,,,,0,2,0,5,,,,0,710,0,0,0,0,,2,5,712,,,0,0,0,0,,,,,,,,statistics-cpu-4, Port 7001 # pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,agent_status,agent_code,agent_duration,check_desc,agent_desc,check_rise,check_fall,check_health,agent_rise,agent_fall,agent_health,addr,cookie,mode,algo,conn_rate,conn_rate_max,conn_tot,intercepted,dcon,dses, sample-backend,BACKEND,0,0,0,1,200,3,950,17717,0,0,,0,0,0,0,UP,1,1,0,,0,883,0,,1,3,0,,3,,1,0,,2,,,,0,3,0,0,0,0,,,,3,0,0,0,0,0,0,448,,,0,0,1,2,,,,,,,,,,,,,,http,roundrobin,,,,,,, sample-backend,backend-server-1,0,0,0,1,,3,950,17717,,0,,0,0,0,0,UP,1,1,0,0,0,883,0,,1,3,1,,3,,2,0,,2,L4OK,,4,0,3,0,0,0,0,,,,,0,0,,,,,448,,,0,0,1,2,,,,Layer4 check passed,,2,3,4,,,,192.168.88.1:80,,http,,,,,,,, sample-backend,backend-server-2,0,0,0,0,,0,0,0,,0,,0,0,0,0,DOWN,1,1,0,1,1,881,881,,1,3,2,,0,,2,0,,0,L4CON,,1070,0,0,0,0,0,0,,,,,0,0,,,,,-1,No route to host at initial connection step of tcp-check,,0,0,0,0,,,,Layer4 connection problem,,2,3,0,,,,192.168.88.2:80,,http,,,,,,,, sample-frontend,FRONTEND,,,0,1,2000,1,950,17717,0,0,0,,,,,OPEN,,,,,,,,,1,2,0,,,,0,0,0,1,,,,0,3,0,0,0,0,,0,2,3,,,0,0,0,0,,,,,,,,,,,,,,,,,,,,,http,,0,1,1,0,0,0, statistics-0,BACKEND,0,0,0,0,200,0,524,13746,0,0,,0,0,0,0,UP,0,0,0,,0,883,,,1,8,0,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,0,0,0,0,0,0,0,38,,,0,0,1,1,,,,,,,,,,,,,,http,roundrobin,,,,,,, statistics-0,FRONTEND,,,0,1,2000,4,524,13746,0,0,0,,,,,OPEN,,,,,,,,,1,8,0,,,,0,0,0,1,,,,0,4,0,0,0,0,,0,1,4,,,0,0,0,0,,,,,,,,,,,,,,,,,,,,,http,,0,1,4,4,0,0, statistics-1,BACKEND,0,0,0,0,200,0,66968,1583972,0,0,,0,0,0,0,UP,0,0,0,,0,883,,,1,4,0,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,0,0,0,0,0,0,0,0,,,0,0,0,1,,,,,,,,,,,,,,http,roundrobin,,,,,,, statistics-1,FRONTEND,,,1,4,2000,717,66968,1583972,0,0,0,,,,,OPEN,,,,,,,,,1,4,0,,,,0,1,0,6,,,,0,716,0,0,0,0,,1,6,717,,,0,0,0,0,,,,,,,,,,,,,,,,,,,,,http,,1,6,717,717,0,0, Port 7002 # pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,agent_status,agent_code,agent_duration,check_desc,agent_desc,check_rise,check_fall,check_health,agent_rise,agent_fall,agent_health,addr,cookie,mode,algo,conn_rate,conn_rate_max,conn_tot,intercepted,dcon,dses, sample-backend,BACKEND,0,0,0,1,200,3,949,18913,0,0,,0,0,0,0,UP,1,1,0,,0,883,0,,2,3,0,,3,,1,0,,3,,,,0,3,0,0,0,0,,,,3,0,0,0,0,0,0,448,,,0,0,1,2,,,,,,,,,,,,,,http,roundrobin,,,,,,, sample-backend,backend-server-1,0,0,0,1,,3,949,18913,,0,,0,0,0,0,UP,1,1,0,0,0,883,0,,2,3,1,,3,,2,0,,3,L4OK,,3,0,3,0,0,0,0,,,,,0,0,,,,,448,,,0,0,1,2,,,,Layer4 check passed,,2,3,4,,,,192.168.88.1:80,,http,,,,,,,, sample-backend,backend-server-2,0,0,0,0,,0,0,0,,0,,0,0,0,0,DOWN,1,1,0,1,1,881,881,,2,3,2,,0,,2,0,,0,L4CON,,1070,0,0,0,0,0,0,,,,,0,0,,,,,-1,No route to host at initial connection step of tcp-check,,0,0,0,0,,,,Layer4 connection problem,,2,3,0,,,,192.168.88.2:80,,http,,,,,,,, sample-frontend,FRONTEND,,,0,1,2000,1,949,18913,0,0,0,,,,,OPEN,,,,,,,,,2,2,0,,,,0,0,0,1,,,,0,3,0,0,0,0,,0,3,3,,,0,0,0,0,,,,,,,,,,,,,,,,,,,,,http,,0,1,1,0,0,0, statistics-0,BACKEND,0,0,0,0,200,0,393,10429,0,0,,0,0,0,0,UP,0,0,0,,0,883,,,2,8,0,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,0,0,0,0,0,0,0,35,,,0,0,1,1,,,,,,,,,,,,,,http,roundrobin,,,,,,, statistics-0,FRONTEND,,,0,1,2000,3,393,10429,0,0,0,,,,,OPEN,,,,,,,,,2,8,0,,,,0,0,0,1,,,,0,3,0,0,0,0,,0,1,3,,,0,0,0,0,,,,,,,,,,,,,,,,,,,,,http,,0,1,3,3,0,0, statistics-2,BACKEND,0,0,0,0,200,0,66968,1585370,0,0,,0,0,0,0,UP,0,0,0,,0,883,,,2,5,0,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,0,0,0,0,0,0,0,0,,,0,0,0,0,,,,,,,,,,,,,,http,roundrobin,,,,,,, statistics-2,FRONTEND,,,1,4,2000,717,66968,1585370,0,0,0,,,,,OPEN,,,,,,,,,2,5,0,,,,0,1,0,6,,,,0,716,0,0,0,0,,1,6,717,,,0,0,0,0,,,,,,,,,,,,,,,,,,,,,http,,1,6,717,717,0,0, Port 7003 # pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,agent_status,agent_code,agent_duration,check_desc,agent_desc,check_rise,check_fall,check_health,agent_rise,agent_fall,agent_health,addr,cookie,mode,algo,conn_rate,conn_rate_max,conn_tot,intercepted,dcon,dses, sample-backend,BACKEND,0,0,0,0,200,0,0,0,0,0,,0,0,0,0,UP,1,1,0,,0,883,0,,3,3,0,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,0,0,0,0,0,0,0,-1,,,0,0,0,0,,,,,,,,,,,,,,http,roundrobin,,,,,,, sample-backend,backend-server-1,0,0,0,0,,0,0,0,,0,,0,0,0,0,UP,1,1,0,0,0,883,0,,3,3,1,,0,,2,0,,0,L4OK,,4,0,0,0,0,0,0,,,,,0,0,,,,,-1,,,0,0,0,0,,,,Layer4 check passed,,2,3,4,,,,192.168.88.1:80,,http,,,,,,,, sample-backend,backend-server-2,0,0,0,0,,0,0,0,,0,,0,0,0,0,DOWN,1,1,0,1,1,881,881,,3,3,2,,0,,2,0,,0,L4CON,,1070,0,0,0,0,0,0,,,,,0,0,,,,,-1,No route to host at initial connection step of tcp-check,,0,0,0,0,,,,Layer4 connection problem,,2,3,0,,,,192.168.88.2:80,,http,,,,,,,, sample-frontend,FRONTEND,,,0,0,2000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,3,2,0,,,,0,0,0,0,,,,0,0,0,0,0,0,,0,0,0,,,0,0,0,0,,,,,,,,,,,,,,,,,,,,,http,,0,0,0,0,0,0, statistics-0,BACKEND,0,0,0,0,200,0,524,13939,0,0,,0,0,0,0,UP,0,0,0,,0,883,,,3,8,0,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,0,0,0,0,0,0,0,0,,,0,0,1,1,,,,,,,,,,,,,,http,roundrobin,,,,,,, statistics-0,FRONTEND,,,0,1,2000,4,524,13939,0,0,0,,,,,OPEN,,,,,,,,,3,8,0,,,,0,1,0,1,,,,0,4,0,0,0,0,,1,1,4,,,0,0,0,0,,,,,,,,,,,,,,,,,,,,,http,,1,1,4,4,0,0, statistics-3,BACKEND,0,0,0,0,200,0,66782,1575534,0,0,,0,0,0,0,UP,0,0,0,,0,883,,,3,6,0,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,0,0,0,0,0,0,0,0,,,0,0,0,1,,,,,,,,,,,,,,http,roundrobin,,,,,,, statistics-3,FRONTEND,,,1,4,2000,715,66782,1575534,0,0,0,,,,,OPEN,,,,,,,,,3,6,0,,,,0,1,0,5,,,,0,714,0,0,0,0,,1,5,715,,,0,0,0,0,,,,,,,,,,,,,,,,,,,,,http,,1,5,715,715,0,0, Port 7004 # pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,agent_status,agent_code,agent_duration,check_desc,agent_desc,check_rise,check_fall,check_health,agent_rise,agent_fall,agent_health,addr,cookie,mode,algo,conn_rate,conn_rate_max,conn_tot,intercepted,dcon,dses, sample-backend,BACKEND,0,0,0,4,200,9,2731,53617,0,0,,0,0,0,0,UP,1,1,0,,0,883,0,,4,3,0,,9,,1,0,,8,,,,0,8,0,1,0,0,,,,9,0,0,0,0,0,0,444,,,0,1,1,12,,,,,,,,,,,,,,http,roundrobin,,,,,,, sample-backend,backend-server-1,0,0,0,4,,9,2731,53617,,0,,0,0,0,0,UP,1,1,0,0,0,883,0,,4,3,1,,9,,2,0,,8,L4OK,,3,0,8,0,1,0,0,,,,,0,0,,,,,444,,,0,1,1,12,,,,Layer4 check passed,,2,3,4,,,,192.168.88.1:80,,http,,,,,,,, sample-backend,backend-server-2,0,0,0,0,,0,0,0,,0,,0,0,0,0,DOWN,1,1,0,1,1,881,881,,4,3,2,,0,,2,0,,0,L4CON,,1071,0,0,0,0,0,0,,,,,0,0,,,,,-1,No route to host at initial connection step of tcp-check,,0,0,0,0,,,,Layer4 connection problem,,2,3,0,,,,192.168.88.2:80,,http,,,,,,,, sample-frontend,FRONTEND,,,0,4,2000,4,2731,53617,0,0,0,,,,,OPEN,,,,,,,,,4,2,0,,,,0,0,0,4,,,,0,8,0,1,0,0,,0,8,9,,,0,0,0,0,,,,,,,,,,,,,,,,,,,,,http,,0,4,4,0,0,0, statistics-0,BACKEND,0,0,0,0,200,0,131,3546,0,0,,0,0,0,0,UP,0,0,0,,0,883,,,4,8,0,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,0,0,0,0,0,0,0,46,,,0,0,1,1,,,,,,,,,,,,,,http,roundrobin,,,,,,, statistics-0,FRONTEND,,,0,1,2000,1,131,3546,0,0,0,,,,,OPEN,,,,,,,,,4,8,0,,,,0,0,0,1,,,,0,1,0,0,0,0,,0,1,1,,,0,0,0,0,,,,,,,,,,,,,,,,,,,,,http,,0,1,1,1,0,0, statistics-4,BACKEND,0,0,0,0,200,0,66782,1581104,0,0,,0,0,0,0,UP,0,0,0,,0,883,,,4,7,0,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,0,0,0,0,0,0,0,0,,,0,0,0,0,,,,,,,,,,,,,,http,roundrobin,,,,,,, statistics-4,FRONTEND,,,1,4,2000,715,66782,1581104,0,0,0,,,,,OPEN,,,,,,,,,4,7,0,,,,0,1,0,5,,,,0,714,0,0,0,0,,1,5,715,,,0,0,0,0,,,,,,,,,,,,,,,,,,,,,http,,1,5,715,715,0,0,
Notice, calculated values on port 7000
could be slightly delayed up to 5 seconds.