Keep track of network latency using SmokePing, uWSGI application server, and Nginx HTTP server. SmokePing supports many configurable probes, alerting, and live charts accessible through a web browser. It was created by Tobi Oetiker, which is known for RRDtool an Open Source data logging and graphing system for time series data.
Install and configure SmokePing
Update package index.
$ sudo apt-get update
Install the smokeping package, but skip recommended packages as it would set up software using Apache HTTP server. There is nothing wrong with that. It will perform as well as this setup.
$ sudo apt-get install --no-install-recommends smokeping dnsutils curl
This software is stable for the last two years, so there is no need to download and install it from the source code. Use the Debian package, do not waste your time.
At first, check out database settings as you cannot easily modify these later on. I will leave default settings, which means 20 pings every 5 minutes per check, unless overridden, of course.
$ cat << EOF | sudo tee /etc/smokeping/config.d/Database
*** Database ***
step = 300
pings = 20
# consfn mrhb steps total
AVERAGE 0.5 1 1008
AVERAGE 0.5 12 4320
MIN 0.5 12 4320
MAX 0.5 12 4320
AVERAGE 0.5 144 720
MAX 0.5 144 720
MIN 0.5 144 720
EOF
Update general settings.
$ cat << EOF | sudo tee /etc/smokeping/config.d/General *** General *** owner = Milosz contact = milosz@example.org mailhost = localhost # NOTE: do not put the Image Cache below cgi-bin # since all files under cgi-bin will be executed ... this is not # good for images. cgiurl = http://smoke.example.org/smokeping.cgi # specify this to get syslog logging syslogfacility = local0 # each probe is now run in its own process # disable this to revert to the old behaviour # concurrentprobes = no @include /etc/smokeping/config.d/pathnames EOF
Define probes that will be used to check specified targets.
$ cat << EOF | sudo tee /etc/smokeping/config.d/Probes *** Probes *** + FPing binary = /usr/bin/fping +DNS binary = /usr/bin/dig +Curl binary = /usr/bin/curl +SSH binary = /usr/bin/ssh-keyscan pings = 5 EOF
Define available alerts.
$ cat << EOF | sudo tee /etc/smokeping/config.d/Alerts *** Alerts *** to = root@localhost from = smokealert@company.xy +offlineatstartup type = rtt pattern = ==S,==U comment = offline at startup +hostdown_with_state type = loss pattern = >50% edgetrigger = yes comment = host down +lossdetect type = loss pattern = ==0%,==0%,==0%,==0%,==0%,>0%,>0%,>0% comment = suddenly there is packet loss +lossdetect_with_state type = loss edgetrigger = yes pattern = ==0%,==0%,==0%,==0%,==0%,>0%,>0%,>0% comment = sudden packet loss +rttdetect type = rtt pattern = <100,<100,<100,<100,<100,>100,>100,>100 comment = suddenly latency increased over 100ms +lost_5_from_20_with_state type = matcher edgetrigger = yes pattern = CheckLoss(l => 5,x => 20) comment = lost over 5 from 20 samples +rtt_avg_increased type = matcher pattern = Avgratio(historic => 20, current => 2, comparator=>'>', percentage => 150) comment = average latency from last 2 samples increased over 150% when compared to average from last 20 samples EOF
Precisely define targets using defined probes and alerts.
$ cat << EOF | sudo tee /etc/smokeping/config.d/Targets *** Targets *** menu = Top title = Network Latency Grapher probe = FPing + Example menu = Example domains title = Example domains ++ example_com menu = example.com title = Host example.com host = example.com alerts = offlineatstartup,hostdown_with_state,rttdetect,rtt_avg_increased,lossdetect ++ example_org menu = example.org title = Host example.org host = example.org alerts = offlineatstartup,hostdown_with_state,rttdetect,rtt_avg_increased,lossdetect + dns menu = DNS check title = DNS check probe = DNS ++ google menu = Google title = DNS check using Google Public DNS +++ example_org_dns_google menu = example.com title = DNS check example.com using first Google Public DNS server host = example.com pings = 5 server = 8.8.8.8 +++ example_com_dns_google menu = example.com title = DNS check example.com using second Google Public DNS server host = example.com pings = 5 server = 8.8.4.4 ++ local menu = local title = DNS check using local DNS +++ example_org_dns_local menu = example.com title = DNS check example.com using local DNS host = example.com pings = 5 server = 192.168.1.254 + ssh menu = SSH title = SSH check probe = SSH ++ example_org_ssh menu = example.org title = SSH check - example.org host = example.org keytype = rsa port = 22 alerts = hostdown_with_state,rttdetect,rtt_avg_increased + WWW menu = WWW title = WWW check probe = Curl ++ local_www menu = local title = WWW check - this website host = smoke.example.org extraargs = --user secret-user:secret-password urlformat = https://%host%/ expect = Logged in as secret-user pings = 5 alerts = offlineatstartup,hostdown_with_state ++ google_www_check menu = google.com title = WWW check - google.com host = google.com follow_redirects = yes include_redirects = yes urlformat = https://%host%/ expect = search pings = 5 alerts = offlineatstartup,hostdown_with_state,rttdetect,rtt_avg_increased EOF
Reload smokeping service.
$ sudo systemctl reload smokeping
Install and configure uWSGI
Install the uWSGI package.
$ sudo apt-get --no-install-recommends install uwsgi
Define smokeping application with simple basic-auth (user secret-user and password secret-password).
$ cat << EOF | sudo tee /etc/uwsgi/apps-available/smokeping.ini [uwsgi] plugins = router_basicauth, cgi route = ^/ basicauth:smokeping,secret-user:secret-password uid = www-data gid = www-data processes = 1 threads = 2 cgi = /usr/lib/cgi-bin/smokeping.cgi chown-socket = www-data:www-data EOF
Specified CGI script executes real SmokePing web-frontend.
$ cat /usr/lib/cgi-bin/smokeping.cgi
#!/bin/sh exec /usr/share/smokeping/smokeping.cgi /etc/smokeping/config
Enable application.
$ sudo ln -s /etc/uwsgi/apps-available/smokeping.ini /etc/uwsgi/apps-enabled/
Reload application server configuration.
$ sudo systemctl restart uwsgi
Install and configure Nginx
Install nginx package.
$ sudo apt-get install --no-install-recommends nginx
Disable default Nginx site.
$ sudo unlink /etc/nginx/sites-enabled/default
Create a directory for SSL certificates.
$ sudo mkdir /etc/nginx/ssl
Generate a self-signed SSL certificate for your domain.
$ sudo openssl req -subj "/commonName=smoke.example.org/" -x509 -nodes -days 730 -newkey rsa:2048 -keyout /etc/nginx/ssl/nginx.key -out /etc/nginx/ssl/nginx.crt
Prepare Nginx virtual host configuration.
$ cat << EOF | sudo tee /etc/nginx/sites-available/smoke.example.org
server {
listen 443 ssl;
server_name default;
ssl_certificate_key /etc/nginx/ssl/nginx.key;
ssl_certificate /etc/nginx/ssl/nginx.crt;
location / {
include uwsgi_params;
uwsgi_modifier1 9;
uwsgi_pass unix:/var/run/uwsgi/app/smokeping/socket;
}
location ^~ /smokeping/ {
alias /usr/share/smokeping/www/;
if (!-f \$request_filename) {
return 301 /;
}
}
}
EOFEnable virtual host.
$ sudo ln -s /etc/nginx/sites-available/smoke.example.org /etc/nginx/sites-enabled/
Reload HTTP server configuration.
$ sudo systemctl reload nginx
It is as simple as that.
Additional notes
Check the defined configuration.
$ sudo smokeping --check
Configuration file '/etc/smokeping/config' syntax OK.
Execute once and display performed operations.
$ sudo smokeping --debug
### parsing ssh-keyscan output...OK
### parsing dig output...OK
### Compiling alert detector pattern 'lossdetect_with_state'
### ==0%,==0%,==0%,==0%,==0%,>0%,>0%,>0%
sub {
my $d = shift;
my $y = $d->{loss};
for(1){
my $minlength = 8;
my $maxlength = 8;
next if scalar @$y < $minlength ;
next unless defined $y->[-8]
and $y->[-8] =~ /^\d/
and $y->[-8] == 0
; next unless defined $y->[-7]
and $y->[-7] =~ /^\d/
and $y->[-7] == 0
; next unless defined $y->[-6]
and $y->[-6] =~ /^\d/
and $y->[-6] == 0
; next unless defined $y->[-5]
and $y->[-5] =~ /^\d/
and $y->[-5] == 0
; next unless defined $y->[-4]
and $y->[-4] =~ /^\d/
and $y->[-4] == 0
; next unless defined $y->[-3]
and $y->[-3] =~ /^\d/
and $y->[-3] > 0
; next unless defined $y->[-2]
and $y->[-2] =~ /^\d/
and $y->[-2] > 0
; next unless defined $y->[-1]
and $y->[-1] =~ /^\d/
and $y->[-1] > 0
; return 1;
}
return 0;
}
### Compiling alert detector pattern 'rttdetect'
### <100,<100,<100,<100,<100,>100,>100,>100
sub {
my $d = shift;
my $y = $d->{rtt};
for(1){
my $minlength = 8;
my $maxlength = 8;
next if scalar @$y < $minlength ;
next unless defined $y->[-8]
and $y->[-8] =~ /^\d/
and $y->[-8] < 0.1
; next unless defined $y->[-7]
and $y->[-7] =~ /^\d/
and $y->[-7] < 0.1
; next unless defined $y->[-6]
and $y->[-6] =~ /^\d/
and $y->[-6] < 0.1
; next unless defined $y->[-5]
and $y->[-5] =~ /^\d/
and $y->[-5] < 0.1
; next unless defined $y->[-4]
and $y->[-4] =~ /^\d/
and $y->[-4] < 0.1
; next unless defined $y->[-3]
and $y->[-3] =~ /^\d/
and $y->[-3] > 0.1
; next unless defined $y->[-2]
and $y->[-2] =~ /^\d/
and $y->[-2] > 0.1
; next unless defined $y->[-1]
and $y->[-1] =~ /^\d/
and $y->[-1] > 0.1
; return 1;
}
return 0;
}
### Compiling alert detector pattern 'lossdetect'
### ==0%,==0%,==0%,==0%,==0%,>0%,>0%,>0%
sub {
my $d = shift;
my $y = $d->{loss};
for(1){
my $minlength = 8;
my $maxlength = 8;
next if scalar @$y < $minlength ;
next unless defined $y->[-8]
and $y->[-8] =~ /^\d/
and $y->[-8] == 0
; next unless defined $y->[-7]
and $y->[-7] =~ /^\d/
and $y->[-7] == 0
; next unless defined $y->[-6]
and $y->[-6] =~ /^\d/
and $y->[-6] == 0
; next unless defined $y->[-5]
and $y->[-5] =~ /^\d/
and $y->[-5] == 0
; next unless defined $y->[-4]
and $y->[-4] =~ /^\d/
and $y->[-4] == 0
; next unless defined $y->[-3]
and $y->[-3] =~ /^\d/
and $y->[-3] > 0
; next unless defined $y->[-2]
and $y->[-2] =~ /^\d/
and $y->[-2] > 0
; next unless defined $y->[-1]
and $y->[-1] =~ /^\d/
and $y->[-1] > 0
; return 1;
}
return 0;
}
### Compiling alert detector pattern 'hostdown_with_state'
### >50%
sub {
my $d = shift;
my $y = $d->{loss};
for(1){
my $minlength = 1;
my $maxlength = 1;
next if scalar @$y < $minlength ;
next unless defined $y->[-1]
and $y->[-1] =~ /^\d/
and $y->[-1] > 50
; return 1;
}
return 0;
}
### Compiling alert detector pattern 'offlineatstartup'
### ==S,==U
sub {
my $d = shift;
my $y = $d->{rtt};
for(1){
my $minlength = 2;
my $maxlength = 2;
next if scalar @$y < $minlength ;
next unless defined $y->[-2] and $y->[-2] eq 'S';
next if defined $y->[-1];
return 1;
}
return 0;
}
Smokeping version 2.006011 successfully launched.
Not entering multiprocess mode with '--debug'. Use '--debug-daemon' for that.
Curl: probing 2 targets with step 300 s and offset 161 s.
DNS: probing 3 targets with step 300 s and offset 161 s.
SSH: probing 1 targets with step 300 s and offset 161 s.
FPing: probing 2 targets with step 300 s and offset 161 s.
Curl: forks 5, timeout for each target 51
Curl: executing command list '/usr/bin/curl','-m','10','-w','Time: %{time_total} DNS time: %{time_namelookup} Redirect time: %{time_redirect}\n','--user','secret-user:secret-password','--insecure','https://smoke.example.org/'Curl: executing command list '/usr/bin/curl','-m','10','-w','Time: %{time_total} DNS time: %{time_namelookup} Redirect time: %{time_redirect}\n','-L','https://google.com/'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload T otal Spent Left Speed
% T o0t a l 0 % R e0c e i v e d0 % X f0e r d A0v e r a g e 0S p e e d 0 T-i-m:e- - : - -T i-m-e: - - : - -T i-m-e: - -C:u-r-r e n t
0 Dload Upload Total Spent Left Speed
100 3597 100 3597 0 0 8237 0 --:--:-- --:--:-- --:--:-- 8250
Curl: curl output: 'Time: 0.436645 DNS time: 0.010099 Redirect time: 0.000000', result: 0.426546
100 269 100 269 0 0 590 0 --:--:-- --:--:-- --:--:-- 591
Time Time Current
Dload Upload Total Spent Left Speed
100 3597 100 3597 0 0 8313 0 --:--:-- --:--:-- --:--:-- 8307
Curl: curl output: 'Time: 0.432650 DNS time: 0.009387 Redirect time: 0.000000', result: 0.423263
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 11128 0 11128 0 0 11075 0 --:--:-- 0:00:01 --:--:-- 47965
Curl: curl output: '</script></div></body></html>Time: 1.004703 DNS time: 0.124560 Redirect time: 0.455532', result: 1.335675
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 3597 100 3597 0 0 7860 0 --:--:-- --:--:-- --:--:-- 7870
Curl: curl output: 'Time: 0.457618 DNS time: 0.008962 Redirect time: 0.000000', result: 0.448656
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 269 100 269 0 0 487 0 --:--:-- --:--:-- --:--:-- 488
100 3597 100 3597 0 0 8571 0 --:--:-- --:--:-- --:--:-- 8564
Curl: curl output: 'Time: 0.419665 DNS time: 0.008850 Redirect time: 0.000000', result: 0.410815
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 Current
Dload Upload Total Spent Left Speed
100 11194 0 11194 0 0 9382 0 --:--:-- 0:00:01 --:--:-- 26973
Curl: curl output: '</script></div></body></html>Time: 1.193133 DNS time: 0.124532 Redirect time: 0.551845', result: 1.620446
100 3597 100 3597 0 0 8054 0 --:--:-- --:--:-- --:--:-- 8065
Curl: curl output: 'Time: 0.446580 DNS time: 0.015530 Redirect time: 0.000000', result: 0.43105
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Curl: smoke.example.org: got 0.410815 0.423263 0.426546 0.43105 0.448656
100 269 100 269 0 0 445 0 --:--:-- --:--:-- --:--:-- 444
100 11145 0 11145 0 0 9397 0 --:--:-- 0:00:01 --:--:-- 9397
Curl: curl output: '</script></div></body></html>Time: 1.185905 DNS time: 0.124787 Redirect time: 0.606740', result: 1.667858
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 269 100 269 0 0 468 0 --:--:-- --:--:-- --:--:-- 468
100 12377 0 12377 0 0 10602 0 --:--:-- 0:00:01 --:--:-- 10602
Curl: curl output: '</script></div></body></html>Time: 1.167415 DNS time: 0.124849 Redirect time: 0.575869', result: 1.618435
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 269 100 269 0 0 467 0 --:--:-- --:--:-- --:--:-- 467
100 11169 0 11169 0 0 10043 0 --:--:-- 0:00:01 --:--:-- 3635k
Curl: curl output: '</script></div></body></html>Time: 1.112116 DNS time: 0.124859 Redirect time: 0.578498', result: 1.565755
Curl: google.com: got 1.335675 1.565755 1.618435 1.620446 1.667858
DNS: forks 5, timeout for each target 26
DNS: query=/usr/bin/dig @192.168.1.1 example.comDNS: query=/usr/bin/dig @8.8.8.8 example.com
DNS: query=/usr/bin/dig @8.8.4.4 example.com
DNS: example.com: got 0.0000000000e+00 0.0000000000e+00 0.0000000000e+00 0.0000000000e+00 0.0000000000e+00
DNS: example.com: got 8.3000000000e-02 8.5000000000e-02 8.9000000000e-02 1.0700000000e-01 1.2100000000e-01
DNS: example.com: got 8.2000000000e-02 8.7000000000e-02 8.8000000000e-02 1.0600000000e-01 1.1600000000e-01
SSH: forks 5, timeout for each target 26
SSH: query=/usr/bin/ssh-keyscan -t rsa -p 22 example.org
SSH: example.org: got 1.6821000000e-02 1.8110000000e-02 1.8164000000e-02 1.8171000000e-02 1.8271000000e-02
FPing: Executing /usr/bin/fping -C 20 -q -B1 -r1 -i10 example.org example.com
FPing: Got fping output: 'example.org : 0.03 0.02 0.02 0.02 0.02 0.02 0.02 0.02 0.02 0.01 0.02 0.04 0.04 0.03 0.03 0.03 0.03 0.04 0.02 0.01'
FPing: Got fping output: 'example.com : 0.02 0.02 0.02 0.02 0.02 0.02 0.02 0.02 0.02 0.02 0.02 0.02 0.01 0.02 0.02 0.02 0.02 0.02 0.02 0.02'
Calling RRDs::update(/var/lib/smokeping/Example/example_org.rrd --template uptime:loss:median:ping1:ping2:ping3:ping4:ping5:ping6:ping7:ping8:ping9:ping10:ping11:ping12:ping13:ping14:ping15:ping16:ping17:ping18:ping19:ping20 1504565297:U:0:2.0000000000e-05:1.0000000000e-05:1.0000000000e-05:2.0000000000e-05:2.0000000000e-05:2.0000000000e-05:2.0000000000e-05:2.0000000000e-05:2.0000000000e-05:2.0000000000e-05:2.0000000000e-05:2.0000000000e-05:2.0000000000e-05:3.0000000000e-05:3.0000000000e-05:3.0000000000e-05:3.0000000000e-05:3.0000000000e-05:4.0000000000e-05:4.0000000000e-05:4.0000000000e-05)
Alert "offlineatstartup": no match for target /var/lib/smokeping/Example/example_org
Alert "hostdown_with_state": no match for target /var/lib/smokeping/Example/example_org
Alert "rttdetect": no match for target /var/lib/smokeping/Example/example_org
Alert "rtt_avg_increased": no match for target /var/lib/smokeping/Example/example_org
Alert "lossdetect": no match for target /var/lib/smokeping/Example/example_org
Calling RRDs::update(/var/lib/smokeping/Example/example_com.rrd --template uptime:loss:median:ping1:ping2:ping3:ping4:ping5:ping6:ping7:ping8:ping9:ping10:ping11:ping12:ping13:ping14:ping15:ping16:ping17:ping18:ping19:ping20 1504565297:U:0:2.0000000000e-05:1.0000000000e-05:2.0000000000e-05:2.0000000000e-05:2.0000000000e-05:2.0000000000e-05:2.0000000000e-05:2.0000000000e-05:2.0000000000e-05:2.0000000000e-05:2.0000000000e-05:2.0000000000e-05:2.0000000000e-05:2.0000000000e-05:2.0000000000e-05:2.0000000000e-05:2.0000000000e-05:2.0000000000e-05:2.0000000000e-05:2.0000000000e-05:2.0000000000e-05)
Alert "offlineatstartup": no match for target /var/lib/smokeping/Example/example_com
Alert "hostdown_with_state": no match for target /var/lib/smokeping/Example/example_com
Alert "rttdetect": no match for target /var/lib/smokeping/Example/example_com
Alert "rtt_avg_increased": no match for target /var/lib/smokeping/Example/example_com
Alert "lossdetect": no match for target /var/lib/smokeping/Example/example_com
Calling RRDs::update(/var/lib/smokeping/dns/google/example_com_dns_google.rrd --template uptime:loss:median:ping1:ping2:ping3:ping4:ping5 1504565297:U:0:8.8000000000e-02:8.2000000000e-02:8.7000000000e-02:8.8000000000e-02:1.0600000000e-01:1.1600000000e-01)
Calling RRDs::update(/var/lib/smokeping/dns/google/example_org_dns_google.rrd --template uptime:loss:median:ping1:ping2:ping3:ping4:ping5 1504565297:U:0:8.9000000000e-02:8.3000000000e-02:8.5000000000e-02:8.9000000000e-02:1.0700000000e-01:1.2100000000e-01)
Calling RRDs::update(/var/lib/smokeping/dns/local/example_org_dns_local.rrd --template uptime:loss:median:ping1:ping2:ping3:ping4:ping5 1504565297:U:0:0.0000000000e+00:0.0000000000e+00:0.0000000000e+00:0.0000000000e+00:0.0000000000e+00:0.0000000000e+00)
Calling RRDs::update(/var/lib/smokeping/ssh/example_org_ssh.rrd --template uptime:loss:median:ping1:ping2:ping3:ping4:ping5 1504565297:U:0:1.8164000000e-02:1.6821000000e-02:1.8110000000e-02:1.8164000000e-02:1.8171000000e-02:1.8271000000e-02)
Alert "hostdown_with_state": no match for target /var/lib/smokeping/ssh/example_org_ssh
Alert "rttdetect": no match for target /var/lib/smokeping/ssh/example_org_ssh
Alert "rtt_avg_increased": no match for target /var/lib/smokeping/ssh/example_org_ssh
Calling RRDs::update(/var/lib/smokeping/WWW/local_www.rrd --template uptime:loss:median:ping1:ping2:ping3:ping4:ping5 1504565297:U:0:0.426546:0.410815:0.423263:0.426546:0.43105:0.448656)
Alert "offlineatstartup": no match for target /var/lib/smokeping/WWW/local_www
Alert "hostdown_with_state": no match for target /var/lib/smokeping/WWW/local_www
Calling RRDs::update(/var/lib/smokeping/WWW/google_www_check.rrd --template uptime:loss:median:ping1:ping2:ping3:ping4:ping5 1504565297:U:0:1.618435:1.335675:1.565755:1.618435:1.620446:1.667858)
Alert "offlineatstartup": no match for target /var/lib/smokeping/WWW/google_www_check
Alert "hostdown_with_state": no match for target /var/lib/smokeping/WWW/google_www_check
Alert "rttdetect": no match for target /var/lib/smokeping/WWW/google_www_check
Alert "rtt_avg_increased": no match for target /var/lib/smokeping/WWW/google_www_check