Categories
SysOps

How to install and configure Consul

Install and configure Consul.

Firewall configuration

Install a dynamically managed firewall daemon.

$ sudo apt install firewalld

List network interfaces.

$ ip --brief address
lo               UNKNOWN        127.0.0.1/8 ::1/128 
ens18            UP             172.16.151.111/21 metric 100 fe80::5cf7:3aff:fe6a:b34e/64 

Add network interface to the public zone.

$ sudo firewall-cmd --add-interface ens18 --zone public
success

Inspect public zone.

$ sudo firewall-cmd --list-all --zone=public
public
  target: default
  icmp-block-inversion: no
  interfaces: ens18
  sources: 
  services: dhcpv6-client ssh
  ports: 
  protocols: 
  forward: yes
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

On consul server open Server RPC port that will be used to handle requests incoming from other agents.

$ sudo firewall-cmd --add-port 8300/tcp --zone public

On consul server and agent open The Serf LAN port to handle gossip in the LAN.

$ sudo firewall-cmd --add-port 8301/tcp --zone public
$ sudo firewall-cmd --add-port 8301/udp --zone public

Inspect current configuration.

$ sudo firewall-cmd --list-all --zone=public
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: ens18
  sources: 
  services: dhcpv6-client ssh
  ports: 8300/tcp 8301/tcp 8301/udp
  protocols: 
  forward: yes
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

Ensure that firewall configuration is permanent.

$ sudo firewall-cmd --runtime-to-permanent

Inspect service status.

$ systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
     Loaded: loaded (/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2022-07-14 19:25:11 UTC; 49s ago
       Docs: man:firewalld(1)
   Main PID: 5033 (firewalld)
      Tasks: 4 (limit: 2241)
     Memory: 25.3M
        CPU: 746ms
     CGroup: /system.slice/firewalld.service
             └─5033 /usr/bin/python3 /usr/sbin/firewalld --nofork --nopid

Jul 14 19:25:11 mgmt systemd[1]: Starting firewalld - dynamic firewall daemon...
Jul 14 19:25:11 mgmt systemd[1]: Started firewalld - dynamic firewall daemon.

This is a minimal firewall configuration as there are other ports.

Install Consul application

Install dependencies.

$ sudo apt install curl unzip

Visit downloads page and download linux binary.

$ curl --silent \
       --remote-name \
       --output-dir /tmp \
       https://releases.hashicorp.com/consul/1.12.3/consul_1.12.3_linux_amd64.zip

Move archive to persistent location.

$ sudo mv /tmp/consul_1.12.3_linux_amd64.zip /opt/

Inspect downloaded archive.

$ unzip -l /opt/consul_1.12.3_linux_amd64.zip 
Archive:  /opt/consul_1.12.3_linux_amd64.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
118259119  2022-07-13 21:19   consul
---------                     -------
118259119                     1 file

Extract consul binary.

$ sudo unzip -d /usr/bin /opt/consul_1.12.3_linux_amd64.zip consul
Archive:  /opt/consul_1.12.3_linux_amd64.zip
  inflating: /usr/bin/consul

Inspect binary file permissions.

$ ls -l /usr/bin/consul 
-rwxr-xr-x 1 root root 118259119 Jul 13 21:19 /usr/bin/consul

Create consul group.

$ sudo groupadd --gid 863 consul

Create consul user.

$ sudo useradd --uid 863 --gid 863 \
               --shell /bin/false \
               --home-dir /etc/consul.d --no-create-home \
               consul

Create configuration directory.

$ sudo install --directory --group consul --owner consul --mode 700 /etc/consul.d

Create data directory.

$ sudo install --directory --group consul --owner consul --mode 700 /opt/consul

Create log directory.

$ sudo install --directory --group consul --owner consul --mode 700 /var/log/consul

Create empty environment file.

$ sudo -u consul touch /etc/consul.d/consul.env

Create empty configuration.

$ sudo -u consul touch /etc/consul.d/consul.hcl

Create systemd service – it is the official packaged one.

$ sudo tee /usr/lib/systemd/system/consul.service << 'EOF'
[Unit]
Description="HashiCorp Consul - A service mesh solution"
Documentation=https://www.consul.io/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/consul.d/consul.hcl

[Service]
EnvironmentFile=-/etc/consul.d/consul.env
User=consul
Group=consul
ExecStart=/usr/bin/consul agent -config-dir=/etc/consul.d/
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGTERM
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF
[Unit]
Description="HashiCorp Consul - A service mesh solution"
Documentation=https://www.consul.io/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/consul.d/consul.hcl

[Service]
EnvironmentFile=-/etc/consul.d/consul.env
User=consul
Group=consul
ExecStart=/usr/bin/consul agent -config-dir=/etc/consul.d/
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGTERM
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Reload systemd configuration.

$ sudo systemctl --system daemon-reload

Inspect service status.

$ systemctl status consul
○ consul.service - "HashiCorp Consul - A service mesh solution"
     Loaded: loaded (/lib/systemd/system/consul.service; disabled; vendor preset: enabled)
     Active: inactive (dead)
       Docs: https://www.consul.io/

Inspect consul version.

$ consul version
Consul v1.12.3
Revision 2308c75e
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

Configure server application

Create server configuration. You can split it into multiple files, it is just easier to present it now this way.

$ sudo -u consul tee /etc/consul.d/consul.hcl << EOF
# datacenter
datacenter = "dc-lab-1"

# data directory
data_dir = "/opt/consul"

# server mode
server = true

# bind address
bind_addr = "0.0.0.0"

# single-node server
bootstrap = true

# number of expected servers in the datacenter
#bootstrap_expect = 3

# servers in the datacenter
#retry_join = ["172.16.151.117", "172.16.151.118", "172.16.151.119"]

# logging
log_level = "info"
log_file = "/var/log/consul/"
log_rotate_duration = "24h"
log_rotate_max_files = 30
EOF

Validate configuration.

$ sudo -u consul consul validate /etc/consul.d
skipping file /etc/consul.d/consul.env, extension must be .hcl or .json, or config format must be set
bootstrap_expect > 0: expecting 3 servers
Configuration is valid!

Enable server service.

$ sudo systemctl enable --now consul
Created symlink /etc/systemd/system/multi-user.target.wants/consul.service → /lib/systemd/system/consul.service.

Inspect service status.

$ systemctl status consul
● consul.service - "HashiCorp Consul - A service mesh solution"
     Loaded: loaded (/lib/systemd/system/consul.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-07-14 18:44:08 UTC; 1min 13s ago
       Docs: https://www.consul.io/
   Main PID: 4208 (consul)
      Tasks: 9 (limit: 2241)
     Memory: 26.6M
        CPU: 324ms
     CGroup: /system.slice/consul.service
             └─4208 /usr/bin/consul agent -config-dir=/etc/consul.d/

Jul 14 18:44:14 jammy consul[4208]: 2022-07-14T18:44:14.176Z [INFO]  agent.leader: started routine: routine="intermediate cert renew watch"
Jul 14 18:44:14 jammy consul[4208]: 2022-07-14T18:44:14.176Z [INFO]  agent.leader: started routine: routine="CA root pruning"
Jul 14 18:44:14 jammy consul[4208]: 2022-07-14T18:44:14.176Z [INFO]  agent.leader: started routine: routine="CA root expiration metric"
Jul 14 18:44:14 jammy consul[4208]: 2022-07-14T18:44:14.176Z [INFO]  agent.leader: started routine: routine="CA signing expiration metric"
Jul 14 18:44:14 jammy consul[4208]: 2022-07-14T18:44:14.176Z [INFO]  agent.leader: started routine: routine="virtual IP version check"
Jul 14 18:44:14 jammy consul[4208]: 2022-07-14T18:44:14.184Z [INFO]  agent.server: member joined, marking health alive: member=jammy partition=default
Jul 14 18:44:14 jammy consul[4208]: 2022-07-14T18:44:14.186Z [INFO]  agent.leader: stopping routine: routine="virtual IP version check"
Jul 14 18:44:14 jammy consul[4208]: 2022-07-14T18:44:14.186Z [INFO]  agent.leader: stopped routine: routine="virtual IP version check"
Jul 14 18:44:14 jammy consul[4208]: 2022-07-14T18:44:14.194Z [INFO]  agent.server: federation state anti-entropy synced
Jul 14 18:44:14 jammy consul[4208]: 2022-07-14T18:44:14.530Z [INFO]  agent: Synced node info

List cluster members.

$ consul members
Node   Address              Status  Type    Build   Protocol  DC        Partition  Segment
jammy  172.16.151.111:8301  alive   server  1.12.3  2         dc-lab-1  default    <all>

List current raft peers.

$ consul operator raft list-peers
Node   ID                                    Address              State   Voter  RaftProtocol
jammy  1c49ba0a-8bac-78bb-18f4-34df6adca9cd  172.16.151.111:8300  leader  true   3

Configure other servers the same way, just remove bootstrap and uncomment bootstrap_expect and retry_join options.

Configure Consul web user interface

Install Consul on different machine and configure it.

Create server configuration.

$ sudo -u consul tee /etc/consul.d/consul.hcl << EOF
# datacenter
datacenter = "dc-lab-1"

# data directory
data_dir = "/opt/consul"

# servers in the datacenter
retry_join = ["172.16.151.111"]

# logging
log_level = "info"
log_file = "/var/log/consul/"
log_rotate_duration = "24h"
log_rotate_max_files = 30

ui_config{
  enabled = true
}
EOF

Enable agent service.

$ sudo systemctl enable --now consul
Created symlink /etc/systemd/system/multi-user.target.wants/consul.service → /lib/systemd/system/consul.service.

Inspect service status.

$ systemctl status consul
● consul.service - "HashiCorp Consul - A service mesh solution"
     Loaded: loaded (/lib/systemd/system/consul.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-07-14 19:23:54 UTC; 33s ago
       Docs: https://www.consul.io/
   Main PID: 2206 (consul)
      Tasks: 8 (limit: 2241)
     Memory: 21.4M
        CPU: 205ms
     CGroup: /system.slice/consul.service
             └─2206 /usr/bin/consul agent -config-dir=/etc/consul.d/

Jul 14 19:23:54 mgmt consul[2206]: 2022-07-14T19:23:54.795Z [INFO]  agent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws azure digita>
Jul 14 19:23:54 mgmt consul[2206]: 2022-07-14T19:23:54.795Z [INFO]  agent: Joining cluster...: cluster=LAN
Jul 14 19:23:54 mgmt consul[2206]: 2022-07-14T19:23:54.795Z [INFO]  agent: (LAN) joining: lan_addresses=[172.16.151.111]
Jul 14 19:23:54 mgmt consul[2206]: 2022-07-14T19:23:54.795Z [WARN]  agent.router.manager: No servers available
Jul 14 19:23:54 mgmt consul[2206]: 2022-07-14T19:23:54.796Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No known Consul servers"
Jul 14 19:23:54 mgmt consul[2206]: 2022-07-14T19:23:54.797Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: jammy 172.16.151.111
Jul 14 19:23:54 mgmt consul[2206]: 2022-07-14T19:23:54.797Z [INFO]  agent: (LAN) joined: number_of_nodes=1
Jul 14 19:23:54 mgmt consul[2206]: 2022-07-14T19:23:54.797Z [INFO]  agent: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=1
Jul 14 19:23:54 mgmt consul[2206]: 2022-07-14T19:23:54.797Z [INFO]  agent.client: adding server: server="jammy (Addr: tcp/172.16.151.111:8300) (DC: dc-lab-1)"
Jul 14 19:23:57 mgmt consul[2206]: 2022-07-14T19:23:57.318Z [INFO]  agent: Synced node info

List cluster members.

$ consul members
Node   Address              Status  Type    Build   Protocol  DC        Partition  Segment
jammy  172.16.151.111:8301  alive   server  1.12.3  2         dc-lab-1  default    <all>
mgmt   172.16.151.120:8301  alive   client  1.12.3  2         dc-lab-1  default    <default>

Install haproxy.

$ sudo apt install haproxy

Append haproxy configuration.

$ sudo tee --append /etc/haproxy/haproxy.cfg << EOF

frontend http_frontend
   bind *:443 ssl crl /etc/haproxy/consul.pem
   default_backend consul_ui_backend

backend consul_ui_backend
   server ui 127.0.0.1:8500
EOF

Generate certificate.

$ sudo -u vault openssl req -subj "/O=HashiCorp/CN=Consul" -x509 -nodes -days 1095 -newkey rsa:4096 -keyout /etc/haproxy/tls.key -out /etc/haproxy/tls.crt
$ cat /etc/haproxy/tls.crt /etc/haproxy/tls.crt | sudo tee /etc/haproxy/consul.pem

Reload service.

$ sudo systemctl reload haproxy

Use browser to access Consul UI.

Remember to open HTTPS port on the firewall and access web-ui.