Use HAProxy stats socket to determine current application status.
Initial information
Operating system.
$ lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 10 (buster) Release: 10 Codename: buster
HAProxy version.
$ sudo haproxy -v HA-Proxy version 1.8.19-1+deb10u1 2019/11/27 Copyright 2000-2019 Willy Tarreau <willy@haproxy.org>
Default HAProxy configuration.
$ cat /etc/haproxy/haproxy.cfg
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
stats timeout 30s
user haproxy
group haproxy
daemon
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL). This list is from:
# https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
# An alternative list with additional directives can be obtained from
# https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
ssl-default-bind-options no-sslv3
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
Socket permissions.
$ ls -l /run/haproxy/admin.sock srw-rw---- 1 root haproxy 0 Jan 11 21:30 /run/haproxy/admin.sock
Stats socket configuration
The access level for a stats socket
There are three distinct access levels: admin, operator and user.
Use the following table to quickly inspect available commands and required access level, but remember that these things are subject to change.
| Socket command | Permission level | ||
| Admin | Operator | User | |
help | Yes | Yes | Yes |
| help message | |||
prompt | Yes | Yes | Yes |
| toggle interactive mode with prompt | |||
quit | Yes | Yes | Yes |
| disconnect | |||
show tls-keys [id|*] | Yes | Yes | Yes |
| show tls keys references or dump tls ticket keys when id specified | |||
set ssl tls-key [id|keyfile] | Yes | Yes | Yes |
| set the next TLS key for the or listener to | |||
show errors | Yes | Yes | |
| report last request and response errors for each proxy | |||
disable agent | Yes | ||
| disable agent checks (use ‘set server’ instead) | |||
disable health | Yes | ||
| disable health checks (use ‘set server’ instead) | |||
disable server | Yes | ||
| disable a server for maintenance (use ‘set server’ instead) | |||
enable agent | Yes | ||
| enable agent checks (use ‘set server’ instead) | |||
enable health | Yes | ||
| enable health checks (use ‘set server’ instead) | |||
enable server | Yes | ||
| enable a disabled server (use ‘set server’ instead) | |||
set maxconn server | Yes | ||
| change a server’s maxconn setting | |||
set server | Yes | ||
| change a server’s state, weight or address | |||
get weight | Yes | Yes | Yes |
| report a server’s current weight | |||
set weight | Yes | ||
| change a server’s weight (deprecated) | |||
show sess [id] | Yes | Yes | |
| report the list of current sessions or dump this session | |||
shutdown session | Yes | ||
| kill a specific session | |||
shutdown sessions server | Yes | ||
| kill sessions on a server | |||
clear table | Yes | Yes | Yes |
| remove an entry from a table | |||
set table [id] | Yes | Yes | Yes |
| update or create a table entry’s data | |||
show table [id] | Yes | Yes | Yes |
| report table usage stats or dump this table’s contents | |||
clear counters | Yes | Yes | |
| clear max statistics counters (add ‘all’ for all counters) | |||
show info | Yes | Yes | Yes |
| report information about the running process | |||
show stat | Yes | Yes | Yes |
| report counters for each proxy and server | |||
show schema json | Yes | Yes | Yes |
| report schema used for stats | |||
show startup-logs | Yes | Yes | Yes (see 869efd) |
| report logs emitted during HAProxy startup | |||
show resolvers [id] | Yes | Yes | Yes |
| dumps counters from all resolvers section and associated name servers | |||
set maxconn global | Yes | ||
| change the per-process maxconn setting | |||
set rate-limit | Yes | ||
| change a rate limiting value | |||
set severity-output [none|number|string] | Yes | Yes | Yes |
| set presence of severity level in feedback information | |||
set timeout | Yes | Yes | Yes |
| change a timeout setting | |||
show env [var] | Yes | Yes | |
| dump environment variables known to the process | |||
show cli sockets | Yes | Yes | Yes |
| dump list of cli sockets | |||
show fd [num] | Yes | Yes | |
| dump list of file descriptors in use | |||
show activity | Yes | Yes | Yes |
| show per-thread activity stats (for support/developers) | |||
disable frontend | Yes | ||
| temporarily disable specific frontend | |||
enable frontend | Yes | ||
| re-enable specific frontend | |||
set maxconn frontend | Yes | ||
| change a frontend’s maxconn setting | |||
show servers state [id] | Yes | Yes | Yes |
| dump volatile server information (for backend ) | |||
show backend | Yes | Yes | Yes |
| list backends in the current running config | |||
shutdown frontend | Yes | ||
| stop a specific frontend | |||
set dynamic-cookie-key backend | Yes | ||
| change a backend secret key for dynamic cookies | |||
enable dynamic-cookie backend | Yes | ||
| enable dynamic cookies on a specific backend | |||
disable dynamic-cookie backend | Yes | ||
| disable dynamic cookies on a specific backend | |||
show cache | Yes | ||
| show cache status | |||
add acl | Yes | Yes | Yes |
| add acl entry | |||
clear acl [id] | Yes | Yes | Yes |
| clear the content of this acl | |||
del acl | Yes | Yes | Yes |
| delete acl entry | |||
get acl | Yes | Yes | Yes |
| report the patterns matching a sample for an ACL | |||
show acl [id] | Yes | Yes | Yes |
| report available acls or dump an acl’s contents | |||
add map | Yes | Yes | Yes |
| add map entry | |||
clear map [id] | Yes | Yes | Yes |
| clear the content of this map | |||
del map | Yes | Yes | Yes |
| delete map entry | |||
get map | Yes | Yes | Yes |
| report the keys and values matching a sample for a map | |||
set map | Yes | Yes | Yes |
| modify map entry | |||
show map [id] | Yes | Yes | Yes |
| report available maps or dump a map’s contents | |||
show pools | Yes | Yes | Yes |
| report information about the memory pools usage | |||
Stats socket permissions
You can use a single socket or define multiple sockets using different permissions like mode and user/uid and group/gid ownership.
stats socket /run/haproxy/admin.sock mode 600 level admin stats socket /run/haproxy/operator.sock mode 660 level operator user haproxy group haproxy stats socket /run/haproxy/user.sock mode 660 level user uid 106 gid 112
$ ls -l /var/run/haproxy total 0 srw------- 1 root haproxy 0 Jan 12 02:04 admin.sock srw-rw---- 1 haproxy haproxy 0 Jan 12 02:04 operator.sock srw-rw---- 1 haproxy haproxy 0 Jan 12 02:04 user.sock
Stats socket operations
Required utilities
Install socat to interact with stats socket and rlwrap, expect to use interactive mode.
$ sudo apt-get install expect socat rlwrap
One-shot mode
This is the simplest mode of operation, which is especially well suited for shell scripts.
$ echo "show pools" | sudo -u haproxy socat stdio unix-connect:/var/run/haproxy/admin.sock Dumping pools usage. Use SIGQUIT to flush them. - Pool cache_st (16 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users [SHARED] - Pool pipe (32 bytes) : 5 allocated (160 bytes), 5 used, 0 failures, 2 users [SHARED] - Pool email_alert (48 bytes) : 59 allocated (2832 bytes), 6 used, 0 failures, 4 users [SHARED] - Pool tcpcheck_ru (64 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 5 users [SHARED] - Pool spoe_appctx (128 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 3 users [SHARED] - Pool spoe_ctx (144 bytes) : 47 allocated (6768 bytes), 5 used, 0 failures, 2 users [SHARED] - Pool h2s (160 bytes) : 56 allocated (8960 bytes), 14 used, 0 failures, 2 users [SHARED] - Pool h2c (240 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users [SHARED] - Pool http_txn (288 bytes) : 46 allocated (13248 bytes), 4 used, 0 failures, 1 users [SHARED] - Pool connection (384 bytes) : 59 allocated (22656 bytes), 7 used, 0 failures, 1 users [SHARED] - Pool hdr_idx (416 bytes) : 46 allocated (19136 bytes), 4 used, 0 failures, 1 users [SHARED] - Pool dns_resolut (480 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users [SHARED] - Pool dns_answer_ (576 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users [SHARED] - Pool stream (848 bytes) : 47 allocated (39856 bytes), 5 used, 0 failures, 1 users [SHARED] - Pool requri (1024 bytes) : 10 allocated (10240 bytes), 0 used, 0 failures, 1 users [SHARED] - Pool trash (16400 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users - Pool buffer (16408 bytes) : 7 allocated (114856 bytes), 2 used, 0 failures, 1 users [SHARED] Total: 17 pools, 238712 bytes allocated, 45968 used.
It works everywhere and you can use it anytime.
Interactive mode of operation
This mode of operation requires more work as Debian does not provide socat with readline support due to licensing restrictions.
READLINE
Uses GNU readline and history on stdio to allow editing and reusing input lines (example).
Due to licensing restrictions the readline feature is disabled in Debian. See BUGS.
You can use STDIO instead.
This will not work out of the box on Debian.
$ sudo -u haproxy socat readline /run/haproxy/user.sock 2020/01/12 02:51:38 socat[23784] E unknown device/address "readline"
Use rlwrap and simple expect trick to access HAProxy stats socket interactively.
To be exact, you do not need to use
expectBut this is the way I use it.$ expect -c 'log_user 0;spawn sudo rlwrap -l haproxy_stats_socket.log -S "haproxy@$(hostname)> " -c socat /run/haproxy/admin.sock stdio; send "prompt\n"; interact' prompt haproxy@example.org> prompt haproxy@example.org> show cli sockets # socket lvl processes /run/haproxy/admin.sock admin all /run/haproxy/operator.sock operator all /run/haproxy/user.sock user all haproxy@example.org> quit haproxy@example.org> (process exited)
Inspect
$ cat haproxy_stats_socket.log [rlwrap] Sun Jan 12 02:34:19 2020 prompt > show cli sockets # socket lvl processes /run/haproxy/admin.sock admin all /run/haproxy/operator.sock operator all /run/haproxy/user.sock user all > quit
Examples
Display basic information.
$ echo "show info" | sudo -u haproxy socat stdio /run/haproxy/user.sock Name: HAProxy Version: 1.8.19-1+deb10u1 Release_date: 2019/11/27 Nbthread: 1 Nbproc: 1 Process_num: 1 Pid: 23505 Uptime: 0d 0h53m14s Uptime_sec: 3194 Memmax_MB: 0 PoolAlloc_MB: 0 PoolUsed_MB: 0 PoolFailed: 0 Ulimit-n: 4051 Maxsock: 4051 Maxconn: 2000 Hard_maxconn: 2000 CurrConns: 0 CumConns: 13 CumReq: 13 MaxSslConns: 0 CurrSslConns: 0 CumSslConns: 0 Maxpipes: 0 PipesUsed: 0 PipesFree: 0 ConnRate: 0 ConnRateLimit: 0 MaxConnRate: 0 SessRate: 0 SessRateLimit: 0 MaxSessRate: 0 SslRate: 0 SslRateLimit: 0 MaxSslRate: 0 SslFrontendKeyRate: 0 SslFrontendMaxKeyRate: 0 SslFrontendSessionReuse_pct: 0 SslBackendKeyRate: 0 SslBackendMaxKeyRate: 0 SslCacheLookups: 0 SslCacheMisses: 0 CompressBpsIn: 0 CompressBpsOut: 0 CompressBpsRateLim: 0 ZlibMemUsage: 0 MaxZlibMemUsage: 0 Tasks: 4 Run_queue: 1 Idle_pct: 100 node: debian Stopping: 0 Jobs: 5 Listeners: 4
Display defined backends.
$ echo "show backend" | sudo -u haproxy socat stdio unix-connect:/var/run/haproxy/user.sock # name backend-local-blog backend-local-gitlab backend-local-netdata backend-gitlab-ssh no-match
Display servers in a specific backend.
$ echo "show servers state backend-local-gitlab" | sudo -u haproxy socat stdio /var/run/haproxy/user.sock 1 # be_id be_name srv_id srv_name srv_addr srv_op_state srv_admin_state srv_uweight srv_iweight srv_time_since_last_change srv_check_status srv_check_result srv_check_health srv_check_state srv_agent_state bk_f_forced_id srv_f_forced_id srv_fqdn srv_port 5 backend-local-gitlab 1 blog 172.16.4.4 2 0 1 1 6763 1 0 2 0 0 0 0 - 80
Display statistics.
$ echo "show stat" | sudo -u haproxy socat stdio unix-connect:/var/run/haproxy/admin.sock # pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,agent_status,agent_code,agent_duration,check_desc,agent_desc,check_rise,check_fall,check_health,agent_rise,agent_fall,agent_health,addr,cookie,mode,algo,conn_rate,conn_rate_max,conn_tot,intercepted,dcon,dses, ssh-gitlab-frontend,FRONTEND,,,2,2,2000,29,22380,44059,0,0,0,,,,,OPEN,,,,,,,,,1,2,0,,,,0,0,0,1,,,,,,,,,,,0,0,0,,,0,0,0,0,,,,,,,,,,,,,,,,,,,,,tcp,,0,1,29,,0,0, web-frontend,FRONTEND,,,6,46,2000,733,834153,14638693,0,0,0,,,,,OPEN,,,,,,,,,1,3,0,,,,0,0,0,11,,,,0,914,227,9,0,0,,0,26,1150,,,0,0,0,0,,,,,,,,,,,,,,,,,,,,,http,,0,11,733,0,0,0, backend-local-blog,gitlab,0,0,0,3,,567,321344,11708170,,0,,0,0,0,0,no check,1,1,0,,,6584,,,1,4,1,,567,,2,0,,11,,,,0,343,220,4,0,0,,,,,1,0,,,,,12,,,0,1,1,92,,,,,,,,,,,,172.16.40.1:80,,http,,,,,,,, backend-local-blog,BACKEND,0,0,0,3,200,567,321344,11708170,0,0,,0,0,0,0,UP,1,1,0,,0,6584,0,,1,4,0,,567,,1,0,,11,,,,0,343,220,4,0,0,,,,567,1,0,0,0,0,0,12,,,0,1,1,92,,,,,,,,,,,,,,http,roundrobin,,,,,,, backend-local-gitlab,blog,0,0,0,5,,17,14498,1114860,,0,,0,0,0,0,no check,1,1,0,,,6584,,,1,5,1,,17,,2,0,,8,,,,0,10,7,0,0,0,,,,,0,0,,,,,591,,,0,0,6,8,,,,,,,,,,,,172.16.34.34:80,,http,,,,,,,, backend-local-gitlab,BACKEND,0,0,0,5,200,17,14498,1114860,0,0,,0,0,0,0,UP,1,1,0,,0,6584,0,,1,5,0,,17,,1,0,,8,,,,0,10,7,0,0,0,,,,17,0,0,0,0,0,0,591,,,0,0,6,8,,,,,,,,,,,,,,http,roundrobin,,,,,,, backend-local-netdata,netdata,0,0,0,10,,565,497615,1815412,,0,,0,0,0,0,no check,1,1,0,,,6584,,,1,6,1,,565,,2,0,,26,,,,0,561,0,4,0,0,,,,,0,0,,,,,3,,,0,0,2,5478,,,,,,,,,,,,127.0.0.1:19999,,http,,,,,,,, backend-local-netdata,BACKEND,0,0,0,10,200,565,497615,1815412,0,0,,0,0,0,0,UP,1,1,0,,0,6584,0,,1,6,0,,565,,1,0,,26,,,,0,561,0,4,0,0,,,,565,0,0,0,0,0,0,3,,,0,0,2,5478,,,,,,,,,,,,,,http,roundrobin,,,,,,, backend-gitlab-ssh,netdata,0,0,2,2,,29,22380,44059,,0,,0,0,0,0,no check,1,1,0,,,6584,,,1,7,1,,29,,2,0,,1,,,,,,,,,,,,,,1,0,,,,,31,,,0,0,0,309,,,,,,,,,,,,172.16.34.34:22,,tcp,,,,,,,, backend-gitlab-ssh,BACKEND,0,0,2,2,200,29,22380,44059,0,0,,0,0,0,0,UP,1,1,0,,0,6584,0,,1,7,0,,29,,1,0,,1,,,,,,,,,,,,,,1,0,0,0,0,0,31,,,0,0,0,309,,,,,,,,,,,,,,tcp,roundrobin,,,,,,, no-match,BACKEND,0,0,0,0,200,0,0,0,0,0,,0,0,0,0,UP,0,0,0,,0,6584,,,1,8,0,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,0,0,0,0,0,0,0,-1,,,0,0,0,0,,,,,,,,,,,,,,http,roundrobin,,,,,,,
Use awk to parse HAProxy statistics and extract current sessions.
$ echo "show stat" | sudo -u haproxy socat stdio unix-connect:/var/run/haproxy/admin.sock | awk -F, '$2=="FRONTEND" {print $1 " " $5}'
ssh-gitlab-frontend 0
web-frontend 11
Display startup logs.
$ echo "show startup-logs" | sudo -u haproxy socat unix-connect:/var/run/haproxy/admin.sock stdio [WARNING] 011/010936 (30023) : Setting tune.ssl.default-dh-param to 1024 by default, if your workload permits it you should set it to at least 2048. Please set a value >= 1024 to make this warning disappear.