Use OpenResty to perform health checks in your cloud environment as it incorporates Nginx, Lua and comes with batteries included.
I really love it!
Installation
Install curl utility as it is the first dependency.
$ sudo apt install curl
Import OpenResty GPG public key using curl.
$ curl --silent https://openresty.org/package/pubkey.gpg | sudo apt-key add -
Define the OpenResty repository.
$ echo "deb http://openresty.org/package/debian $(lsb_release -sc) openresty" | sudo tee /etc/apt/sources.list.d/openresty.list
Update package index.
$ sudo apt update
Install openresty package, including recommended packages.
$ sudo apt install openresty Reading package lists... Done Building dependency tree Reading state information... Done The following additional packages will be installed: openresty-openssl openresty-opm openresty-pcre openresty-resty openresty-zlib Suggested packages: openresty-restydoc The following NEW packages will be installed: openresty openresty-openssl openresty-opm openresty-pcre openresty-resty openresty-zlib 0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded. Need to get 2,938 kB of archives. After this operation, 8,886 kB of additional disk space will be used. Do you want to continue? [Y/n] Y Get:1 http://openresty.org/package/debian stretch/openresty amd64 openresty-zlib amd64 1.2.11-3~stretch1 [80.3 kB] Get:2 http://openresty.org/package/debian stretch/openresty amd64 openresty-openssl amd64 1.1.0j-1~stretch1 [1,534 kB] Get:3 http://openresty.org/package/debian stretch/openresty amd64 openresty-pcre amd64 8.42-1~stretch1 [258 kB] Get:4 http://openresty.org/package/debian stretch/openresty amd64 openresty amd64 1.15.8.1-1~stretch1 [1,034 kB] Get:5 http://openresty.org/package/debian stretch/openresty amd64 openresty-resty all 1.15.8.1-1~stretch1 [13.2 kB] Get:6 http://openresty.org/package/debian stretch/openresty amd64 openresty-opm amd64 1.15.8.1-1~stretch1 [19.0 kB] Fetched 2,938 kB in 1s (2,065 kB/s) [...]
Alternatively, install openresty package, but skip recommended packages.
$ sudo apt install --no-install-recommends openresty Reading package lists... Done Building dependency tree Reading state information... Done The following additional packages will be installed: openresty-openssl openresty-pcre openresty-zlib Suggested packages: openresty-restydoc Recommended packages: openresty-resty openresty-opm The following NEW packages will be installed: openresty openresty-openssl openresty-pcre openresty-zlib 0 upgraded, 4 newly installed, 0 to remove and 0 not upgraded. Need to get 2,906 kB of archives. After this operation, 8,712 kB of additional disk space will be used. Do you want to continue? [Y/n] Get:1 http://openresty.org/package/debian stretch/openresty amd64 openresty-zlib amd64 1.2.11-3~stretch1 [80.3 kB] Get:2 http://openresty.org/package/debian stretch/openresty amd64 openresty-openssl amd64 1.1.0j-1~stretch1 [1,534 kB] Get:3 http://openresty.org/package/debian stretch/openresty amd64 openresty-pcre amd64 8.42-1~stretch1 [258 kB] Get:4 http://openresty.org/package/debian stretch/openresty amd64 openresty amd64 1.15.8.1-1~stretch1 [1,034 kB] Fetched 2,906 kB in 1s (1,538 kB/s) [...]
I will recommend to install recommended packages on the development server but skip these on production, where I like to keep things small.
Initial configuration
OpenResty nginx core configuration is stored inside /usr/local/openresty/nginx/ directory, basic settings in conf/nginx.conf configuration file.
You should configure an SSL certificate as a basic precaution and change the listening port, so it will not affect other services.
Application is controlled by the openresty service.
$ sudo systemctl status openresty
● openresty.service - full-fledged web platform
Loaded: loaded (/lib/systemd/system/openresty.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2019-08-09 09:41:07 GMT; 2h 1min ago
Main PID: 12674 (nginx)
Tasks: 2 (limit: 4915)
CGroup: /system.slice/openresty.service
├─12674 nginx: master process /usr/local/openresty/nginx/sbin/nginx -g daemon on; master_process on;
└─12675 nginx: worker process
Aug 09 09:41:07 stretch systemd[1]: Starting full-fledged web platform...
Aug 09 09:41:07 stretch systemd[1]: Started full-fledged web platform.
Reload this service after you modify configuration or Lua scripts.
HTTP health check
Install lua-resty-http package using OpenResty Package Manager.
Install
openresty-opm package to use OpenResty Package Manager$ sudo opm install ledgetech/lua-resty-http
* Fetching ledgetech/lua-resty-http
Downloading https://opm.openresty.org/api/pkg/tarball/ledgetech/lua-resty-http-0.14.opm.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 20862 100 20862 0 0 62166 0 --:--:-- --:--:-- --:--:-- 62089
Package ledgetech/lua-resty-http 0.14 installed successfully under /usr/local/openresty/site/ .
Alter OpenResty configuration to include HTTP health check endpoint.
This configuration file location is
/usr/local/openresty/nginx/conf/nginx.conflocation /health/http {
default_type "text/html";
content_by_lua_file health/http.lua;
}
Create a Lua script to perform an HTTP health check.
Lua script location is
/usr/local/openresty/nginx/health/http.lualocal http = require("resty.http")
local httpc = http.new()
local url = "http://127.0.0.1"
local parameters = { keepalive = false }
local required_string = "Welcome to nginx!"
local res, err = httpc:request_uri(url, parameters)
if not res then
ngx.status = 503
ngx.header['Error-Message'] = err
ngx.say("Error")
return
end
if res.body and string.find(res.body, required_string) then
ngx.status = 200
ngx.say("OK")
return
else
ngx.status = 503
ngx.header['Error-Message'] = "required string not found"
ngx.say("Error")
return
end
ngx.status = 501
ngx.header['Error-Message'] = "reached end of the script"
ngx.say("Error")
Reload OpenResty service.
$ sudo systemctl reload openresty
Green (success) health check.
$ curl http://192.168.50.191:81/health/http -v * Trying 192.168.50.191... * TCP_NODELAY set * Connected to 192.168.50.191 (192.168.50.191) port 81 (#0) > GET /health/http HTTP/1.1 > Host: 192.168.50.191:81 > User-Agent: curl/7.58.0 > Accept: */* > < HTTP/1.1 200 OK < Server: openresty/1.15.8.1 < Date: Fri, 09 Aug 2019 14:39:00 GMT < Content-Type: text/html < Transfer-Encoding: chunked < Connection: keep-alive < OK * Connection #0 to host 192.168.50.191 left intact
Red (fail) health check.
$ curl http://192.168.50.191:81/health/http -v * Trying 192.168.50.191... * TCP_NODELAY set * Connected to 192.168.50.191 (192.168.50.191) port 81 (#0) > GET /health/http HTTP/1.1 > Host: 192.168.50.191:81 > User-Agent: curl/7.58.0 > Accept: */* > < HTTP/1.1 503 Service Temporarily Unavailable < Server: openresty/1.15.8.1 < Date: Fri, 09 Aug 2019 14:41:13 GMT < Content-Type: text/html < Transfer-Encoding: chunked < Connection: keep-alive < Error-Message: connection refused < Error * Connection #0 to host 192.168.50.191 left intact
I will use the same procedure to create other health checks, so I can omit some obvious information later on.
nginx health check
This is a simplified version of the previous HTTP health check.
Configure nginx service (you can use /etc/nginx/sites-enabled/ping configuration file, for example) to reply with a pong to ping request.
server {
listen 80 default_server;
location /ping {
access_log off;
default_type "text/html";
return 200 "pong\n";
}
}
Alter OpenResty configuration to include nginx health check endpoint.
location /health/nginx {
default_type "text/html";
content_by_lua_file health/nginx.lua;
}
Create a Lua script (health/nginx.lua) to perform this health check.
local http = require("resty.http")
local httpc = http.new()
local url = "http://127.0.0.1/ping"
local parameters = { keepalive = false }
local required_string = "pong"
local res, err = httpc:request_uri(url, parameters)
if not res then
ngx.status = 503
ngx.say("Error")
return
end
if res.body and string.find(res.body, required_string) then
ngx.status = 200
ngx.say("OK")
return
end
ngx.status = 501
ngx.say("Error")
Reload both services and verify this health check.
ProxySQL health check
Alter OpenResty configuration to include ProxySQL health check endpoint.
location /health/proxysql {
default_type "text/html";
content_by_lua_file health/proxysql.lua;
}
Create a Lua script (health/proxysql.lua) to perform this health check.
local mysql = require("resty.mysql")
local db, err = mysql:new()
if not db then
ngx.status = 503
ngx.header['Error-Message'] = err
ngx.say("Error")
return
end
local proxysql_host = "127.0.0.1"
local proxysql_port = 6032
local proxysql_user = "admin"
local proxysql_pass = "admin"
local proxysql_timeout = 1000
db:set_timeout(proxysql_timeout)
local ok, err, errno, sqlstate = db:connect({ host = proxysql_host, port = proxysql_port, user = proxysql_user, password = proxysql_pass })
if not ok then
ngx.status = 503
ngx.header['Error-Message'] = err
ngx.say("Error")
return
end
local res, err, errno, sqlstate = db:query("select Variable_Value from stats.stats_mysql_global where Variable_Name='ProxySQL_Uptime'")
if res and res[1]["Variable_Value"] then
ngx.status = 200
ngx.header['ProxySQL-Uptime'] = res[1]["Variable_Value"]
ngx.say("OK")
return
end
ngx.status = 501
ngx.say("Error")
db:close()
MariaDB health check
Alter OpenResty configuration to include MariaDB health check endpoint.
location /health/mariadb {
default_type "text/html";
content_by_lua_file health/mariadb.lua;
}
Create a Lua script (health/mariadb.lua) to perform this health check.
local mysql = require("resty.mysql")
local db, err = mysql:new()
if not db then
ngx.status = 503
ngx.header['Error-Message'] = err
ngx.say("Error")
return
end
local mariadb_socket = "/var/run/mysqld/mysqld.sock"
local mariadb_user = "admin"
local mariadb_pass = "admin"
local mariadb_timeout = 1000
db:set_timeout(proxysql_timeout)
local ok, err, errno, sqlstate = db:connect({ path = mariadb_socket, user = mariadb_user, password = mariadb_pass })
if not ok then
ngx.status = 503
ngx.header['Error-Message'] = err
ngx.say("Error")
return
end
local res, err, errno, sqlstate = db:query("select VARIABLE_VALUE from information_schema.GLOBAL_STATUS where VARIABLE_NAME='Uptime'")
if res and res[1]["VARIABLE_VALUE"] then
ngx.status = 200
ngx.header['MariaDB-Uptime'] = res[1]["VARIABLE_VALUE"]
ngx.say("OK")
return
end
ngx.status = 501
ngx.say("Error")
PostgreSQL health check
Install pgmoon package using OpenResty Package Manager.
$ sudo opm install leafo/pgmoon
* Fetching leafo/pgmoon
Downloading https://opm.openresty.org/api/pkg/tarball/leafo/pgmoon-1.9.0.opm.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 17242 100 17242 0 0 22860 0 --:--:-- --:--:-- --:--:-- 22837
Package leafo/pgmoon 1.9.0 installed successfully under /usr/local/openresty/site/ .
Alter OpenResty configuration to include PostgreSQL health check endpoint.
location /health/postgresql {
default_type "text/html";
content_by_lua_file health/postgresql.lua;
}
Create a Lua script (health/postgresql.lua) to perform this health check.
local pgmoon = require("pgmoon")
local pg_host = "127.0.0.1"
local pg_port = 5432
local pg_user = "admin"
local pg_pass = "admin"
local pg_database = "postgres"
local pg_timeout = 1000
local pg, err = pgmoon.new({ host = pg_host, port = pg_port, user = pg_user, password = pg_pass, database = pg_database })
if not pg then
ngx.status = 503
ngx.header['Error-Message'] = err
ngx.say("Error")
return
end
pg:settimeout(pg_timeout)
local ok, err = pg:connect()
if not ok then
ngx.status = 503
ngx.header['Error-Message'] = err
ngx.say("Error")
return
end
local res = pg:query("SELECT date_trunc('second', current_timestamp - pg_postmaster_start_time()) as pretty_uptime")
if res and res[1]["pretty_uptime"] then
ngx.status = 200
ngx.header['PostgreSQL-Uptime'] = res[1]["pretty_uptime"]
ngx.say("OK")
return
end
ngx.status = 501
ngx.say("Error")
pg:disconnect()
Redis health check
Alter OpenResty configuration to include Redis health check endpoint.
location /health/redis {
default_type "text/html";
content_by_lua_file health/redis.lua;
}
Create a Lua script (health/redis.lua) to perform this health check.
local redis = require("resty.redis")
local red = redis:new()
local redis_socket = "unix:///var/run/redis/redis.sock"
local redis_password = "password"
local redis_timeout = 1000
red:settimeout(redis_timeout)
local ok, err = red:connect(redis_socket)
if not ok then
ngx.status = 503
ngx.header['Error-Message'] = err
ngx.say("Error")
return
end
local res, err = red:auth(redis_password)
if not res then
ngx.status = 503
ngx.header['Error-Message'] = err
ngx.say("Error")
return
end
local res, err = red:info("server")
if res then
ngx.status = 200
ngx.header['Redis-Uptime'] = string.match(res, "uptime_in_seconds:([0-9]+)")
ngx.say("OK")
return
end
ngx.status = 501
ngx.say("Error")
red:close()