Install and configure HashiCorp Vault.
Install Consul application
Create consul cluster, configure encryption and access control lists.
Create access control lists and tokens
Define policy for Vault application.
$ tee vault-app-policy.hcl << EOF
# Enable the agent to update vault-X nodes
node_prefix "vault" {
policy = "write"
}
# Provide KV visibility to all agents.
agent_prefix "" {
policy = "read"
}
# Enable resources prefixed with 'vault/' to write to the KV
key_prefix "vault/" {
policy = "write"
}
# Enable the vault service to write to the KV
service "vault" {
policy = "write"
}
# Enable the agent to initialize a new session.
session_prefix "" {
policy = "write"
}
EOF
Define vault application policy.
$ consul acl policy create -name "vault-app-policy" -description "vault application policy" -rules @vault-app-policy.hcl
ID: a9b19576-966f-657a-0399-8e06f183cf41
Name: vault-app-policy
Description: vault application policy
Datacenters:
Rules:
# Enable the agent to update vault-X nodes
node_prefix "vault" {
policy = "write"
}
# Provide KV visibility to all agents.
agent_prefix "" {
policy = "read"
}
# Enable resources prefixed with 'vault/' to write to the KV
key_prefix "vault/" {
policy = "write"
}
# Enable the vault service to write to the KV
service "vault" {
policy = "write"
}
# Enable the agent to initialize a new session.
session_prefix "" {
policy = "write"
}
Create Vault application token.
$ consul acl token create -description "vault application token" -policy-name "vault-app-policy"
AccessorID: 42f99064-454d-cdc9-c0a2-6527d90febae SecretID: b12b3e74-bdf9-738b-6a81-bc35be39d769 Description: vault application token Local: false Create Time: 2022-07-16 23:26:06.157117158 +0000 UTC Policies: a9b19576-966f-657a-0399-8e06f183cf41 - vault-app-policy
Define vault haproxy policy.
$ tee vault-haproxy-policy.hcl << EOF
acl = "read"
node_prefix "haproxy" {
policy = "write"
}
node_prefix "vault" {
policy = "read"
}
service "vault" {
policy = "read"
}
EOF
Create Vault haproxy policy.
$ consul acl policy create -name "vault-haproxy-policy" -description "vault haproxy policy" -rules @vault-haproxy-policy.hcl
ID: 54358c22-6d4c-41dc-c2ec-0a02471fd75d
Name: vault-haproxy-policy
Description: vault haproxy policy
Datacenters:
Rules:
acl = "read"
node_prefix "haproxy" {
policy = "write"
}
node_prefix "vault" {
policy = "read"
}
service "vault" {
policy = "read"
}
Create Vault haproxy token.
$ consul acl token create -description "vault haproxy token" -policy-name "vault-haproxy-policy"
AccessorID: dc6d127e-41b5-d8ac-cfa6-862834a3717f SecretID: 86bb29c0-bc1d-13dd-1940-86e492da20fe Description: vault haproxy token Local: false Create Time: 2022-07-16 23:24:28.248763082 +0000 UTC Policies: 54358c22-6d4c-41dc-c2ec-0a02471fd75d - vault-haproxy-policy
Apply tokens
Update agents on haproxy (e.g. haproxy-1, haproxy-X) machines.
$ sudo -u consul tee /etc/consul.d/acl.hcl << EOF
# acl
acl = {
enabled = true
default_policy = "deny"
enable_token_persistence = true
tokens {
default = "86bb29c0-bc1d-13dd-1940-86e492da20fe"
}
}
EOF
Update agents on vault (e.g. vault-1 and so on) machines.
$ sudo -u consul tee /etc/consul.d/acl.hcl << EOF
# acl
acl = {
enabled = true
default_policy = "deny"
enable_token_persistence = true
tokens {
default = "b12b3e74-bdf9-738b-6a81-bc35be39d769"
}
}
EOF
Install Vault application
curl and unzip should be already installed during Consul agent install process.
Visit downloads page and download linux binary.
$ curl --silent \
--remote-name \
--output-dir /tmp \
https://releases.hashicorp.com/vault/1.11.0/vault_1.11.0_linux_amd64.zip
Move archive to persistent location.
$ sudo mv /tmp/vault_1.11.0_linux_amd64.zip /opt/
Inspect downloaded archive.
$ unzip -l /opt/vault_1.11.0_linux_amd64.zip
Archive: /opt/vault_1.11.0_linux_amd64.zip Length Date Time Name --------- ---------- ----- ---- 199518230 2022-06-17 23:04 vault --------- ------- 199518230 1 file
Extract vault binary.
$ sudo unzip -d /usr/bin /opt/vault_1.11.0_linux_amd64.zip vault
Archive: /opt/vault_1.11.0_linux_amd64.zip inflating: /usr/bin/vault
Inspect binary file permissions.
$ ls -l /usr/bin/vault
-rwxr-xr-x 1 root root 199518230 Jun 17 23:04 /usr/bin/vault
Display command usage.
$ vault
Usage: vault <command> [args]
Common commands:
read Read data and retrieves secrets
write Write data, configuration, and secrets
delete Delete secrets and configuration
list List data or secrets
login Authenticate locally
agent Start a Vault agent
server Start a Vault server
status Print seal and HA status
unwrap Unwrap a wrapped secret
Other commands:
audit Interact with audit devices
auth Interact with auth methods
debug Runs the debug command
kv Interact with Vault's Key-Value storage
lease Interact with leases
monitor Stream log messages from a Vault server
namespace Interact with namespaces
operator Perform operator-specific tasks
path-help Retrieve API help for paths
plugin Interact with Vault plugins and catalog
policy Interact with policies
print Prints runtime configurations
secrets Interact with secrets engines
ssh Initiate an SSH session
token Interact with tokens
version-history Prints the version history of the target Vault server
Create vault group.
$ sudo groupadd --gid 864 vault
Create vault user.
$ sudo useradd --uid 864 --gid 864 \
--shell /bin/false \
--home-dir /etc/vault.d --no-create-home \
vault
Create configuration directory.
$ sudo install --directory --group vault --owner vault --mode 700 /etc/vault.d
Create tls directory.
$ sudo install --directory --group vault --owner vault --mode 700 /opt/vault/tls
Create empty environment file.
$ sudo -u vault touch /etc/vault.d/vault.env
Create empty configuration.
$ sudo -u vault touch /etc/vault.d/vault.hcl
Generate certificate for web user interface.
$ sudo -u vault openssl req -subj "/O=HashiCorp/CN=Consul" -x509 -nodes -days 1095 -newkey rsa:4096 -keyout /opt/vault/tls/tls.key -out /opt/vault/tls/tls.crt
Limit access to these files.
$ chmod 600 /opt/vault/tls/tls.{key,crt}
Prevent memory from being swapped to disk.
$ sudo setcap cap_ipc_lock=+ep /usr/bin/vault
Create systemd service – it is the official packaged one.
$ sudo tee /usr/lib/systemd/system/vault.service << 'EOF' [Unit] Description="HashiCorp Vault - A tool for managing secrets" Documentation=https://www.vaultproject.io/docs/ Requires=network-online.target After=network-online.target ConditionFileNotEmpty=/etc/vault.d/vault.hcl StartLimitIntervalSec=60 StartLimitBurst=3 [Service] Type=notify EnvironmentFile=/etc/vault.d/vault.env User=vault Group=vault ProtectSystem=full ProtectHome=read-only PrivateTmp=yes PrivateDevices=yes SecureBits=keep-caps AmbientCapabilities=CAP_IPC_LOCK CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK NoNewPrivileges=yes ExecStart=/usr/bin/vault server -config=/etc/vault.d/vault.hcl ExecReload=/bin/kill --signal HUP $MAINPID KillMode=process KillSignal=SIGINT Restart=on-failure RestartSec=5 TimeoutStopSec=30 LimitNOFILE=65536 LimitMEMLOCK=infinity [Install] WantedBy=multi-user.target EOF
[Unit] Description="HashiCorp Vault - A tool for managing secrets" Documentation=https://www.vaultproject.io/docs/ Requires=network-online.target After=network-online.target ConditionFileNotEmpty=/etc/vault.d/vault.hcl StartLimitIntervalSec=60 StartLimitBurst=3 [Service] Type=notify EnvironmentFile=/etc/vault.d/vault.env User=vault Group=vault ProtectSystem=full ProtectHome=read-only PrivateTmp=yes PrivateDevices=yes SecureBits=keep-caps AmbientCapabilities=CAP_IPC_LOCK CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK NoNewPrivileges=yes ExecStart=/usr/bin/vault server -config=/etc/vault.d/vault.hcl ExecReload=/bin/kill --signal HUP $MAINPID KillMode=process KillSignal=SIGINT Restart=on-failure RestartSec=5 TimeoutStopSec=30 LimitNOFILE=65536 LimitMEMLOCK=infinity [Install] WantedBy=multi-user.target
Reload systemd configuration.
$ sudo systemctl --system daemon-reload
Inspect service status.
$ systemctl status vault
○ vault.service - "HashiCorp Vault - A tool for managing secrets"
Loaded: loaded (/lib/systemd/system/vault.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: https://www.vaultproject.io/docs/
Inspect consul version.
$ vault version
Vault v1.11.0 (ea296ccf58507b25051bc0597379c467046eb2f1), built 2022-06-17T15:48:44Z
Configure server application
Create server configuration. Update systemd service to use directory instead of a single file.
$ sudo -u vault tee /etc/vault.d/vault.hcl << EOF
ui = true
storage "consul" {
address = "127.0.0.1:8500"
path = "vault"
}
# HTTPS listener
listener "tcp" {
address = "0.0.0.0:8200"
tls_cert_file = "/opt/vault/tls/tls.crt"
tls_key_file = "/opt/vault/tls/tls.key"
}
# Enterprise license_path
#license_path = "/etc/vault.d/vault.hclic"
EOF
Enable server service.
$ sudo systemctl enable --now vault
Created symlink /etc/systemd/system/multi-user.target.wants/vault.service → /lib/systemd/system/vault.service.
Inspect service status.
$ systemctl status vault
● vault.service - "HashiCorp Vault - A tool for managing secrets"
Loaded: loaded (/lib/systemd/system/vault.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2022-07-16 23:51:42 UTC; 19s ago
Docs: https://www.vaultproject.io/docs/
Main PID: 2674 (vault)
Tasks: 7 (limit: 2241)
Memory: 58.1M
CPU: 112ms
CGroup: /system.slice/vault.service
└─2674 /usr/bin/vault server -config=/etc/vault.d/vault.hcl
Jul 16 23:51:42 vault-1 vault[2674]: ==> Vault server configuration:
Jul 16 23:51:42 vault-1 vault[2674]: Api Address: https://172.16.148.3:8200
Jul 16 23:51:42 vault-1 vault[2674]: Cgo: disabled
Jul 16 23:51:42 vault-1 vault[2674]: Cluster Address: https://172.16.148.3:8201
Jul 16 23:51:42 vault-1 vault[2674]: Go Version: go1.17.11
Jul 16 23:51:42 vault-1 vault[2674]: Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: ">
Jul 16 23:51:42 vault-1 vault[2674]: Log Level: info
Jul 16 23:51:42 vault-1 vault[2674]: Mlock: supported: true, enabled: true
Jul 16 23:51:42 vault-1 vault[2674]: Recovery Mode: false
Jul 16 23:51:42 vault-1 vault[2674]: Storage: consul (HA available)
Jul 16 23:51:42 vault-1 vault[2674]: Version: Vault v1.11.0, built 2022-06-17T15:48:44Z
Jul 16 23:51:42 vault-1 vault[2674]: Version Sha: ea296ccf58507b25051bc0597379c467046eb2f1
Jul 16 23:51:42 vault-1 vault[2674]: ==> Vault server started! Log data will stream in below:
Jul 16 23:51:42 vault-1 vault[2674]: 2022-07-16T23:51:42.303Z [INFO] proxy environment: http_proxy="" https_proxy="" no_proxy=""
Jul 16 23:51:42 vault-1 vault[2674]: 2022-07-16T23:51:42.304Z [WARN] storage.consul: appending trailing forward slash to path
Jul 16 23:51:42 vault-1 vault[2674]: 2022-07-16T23:51:42.306Z [WARN] no `api_addr` value specified in config or in VAULT_API_ADDR; falling back to detection if possible, but this value should be manually set
Jul 16 23:51:42 vault-1 vault[2674]: 2022-07-16T23:51:42.338Z [INFO] core: Initializing version history cache for core
Jul 16 23:51:42 vault-1 systemd[1]: Started "HashiCorp Vault - A tool for managing secrets".
Display seal and HA status.
$ vault status -tls-skip-verify
Key Value --- ----- Seal Type shamir Initialized false Sealed true Total Shares 0 Threshold 0 Unseal Progress 0/0 Unseal Nonce n/a Version 1.11.0 Build Date 2022-06-17T15:48:44Z Storage Type consul HA Enabled true
Initialize vault server.
$ vault operator init -tls-skip-verify
Unseal Key 1: SLpEi9hrk52mgxay66UnW0ZS2u1tpoQMT7Dj4sYVFoWq Unseal Key 2: uC0VAgbWjLPYGf5mothT9bqLjSa5HW83F7VdWuFnLfAs Unseal Key 3: jy7w3CanszT2b7SgfFAtcyq7IBOHqiOMstq+TtaZAz2G Unseal Key 4: COo33dElKW8qLwbc/c+HbKkDe9UryCaWvd+PDcugUP+7 Unseal Key 5: sCJnTYQfYif5aUB9o49RAt6jpfVlMQSLivsDS+VFxJc4 Initial Root Token: hvs.BKpRwvIYboA6C9QrY5V7OnxA Vault initialized with 5 key shares and a key threshold of 3. Please securely distribute the key shares printed above. When the Vault is re-sealed, restarted, or stopped, you must supply at least 3 of these keys to unseal it before it can start servicing requests. Vault does not store the generated root key. Without at least 3 keys to reconstruct the root key, Vault will remain permanently sealed! It is possible to generate new unseal keys, provided you have a quorum of existing unseal keys shares. See "vault operator rekey" for more information.
Display seal and HA status.
$ vault status -tls-skip-verify
Key Value --- ----- Seal Type shamir Initialized true Sealed true Total Shares 5 Threshold 3 Unseal Progress 0/3 Unseal Nonce n/a Version 1.11.0 Build Date 2022-06-17T15:48:44Z Storage Type consul HA Enabled true
Unseal server.
$ vault operator unseal -tls-skip-verify
Unseal Key (will be hidden): Key Value --- ----- Seal Type shamir Initialized true Sealed true Total Shares 5 Threshold 3 Unseal Progress 1/3 Unseal Nonce 197f6b1d-562b-353d-f40e-1e96c772adf8 Version 1.11.0 Build Date 2022-06-17T15:48:44Z Storage Type consul HA Enabled true
$ vault operator unseal -tls-skip-verify
Unseal Key (will be hidden): Key Value --- ----- Seal Type shamir Initialized true Sealed true Total Shares 5 Threshold 3 Unseal Progress 2/3 Unseal Nonce 197f6b1d-562b-353d-f40e-1e96c772adf8 Version 1.11.0 Build Date 2022-06-17T15:48:44Z Storage Type consul HA Enabled true
$ vault operator unseal -tls-skip-verify
Unseal Key (will be hidden): Key Value --- ----- Seal Type shamir Initialized true Sealed false Total Shares 5 Threshold 3 Version 1.11.0 Build Date 2022-06-17T15:48:44Z Storage Type consul Cluster Name vault-cluster-d49add11 Cluster ID d73bc1ff-d088-e602-959c-cc72be323da9 HA Enabled true HA Cluster n/a HA Mode standby Active Node Address <none>
Display seal and HA status.
$ vault status -tls-skip-verify
Key Value --- ----- Seal Type shamir Initialized true Sealed false Total Shares 5 Threshold 3 Version 1.11.0 Build Date 2022-06-17T15:48:44Z Storage Type consul Cluster Name vault-cluster-d49add11 Cluster ID d73bc1ff-d088-e602-959c-cc72be323da9 HA Enabled true HA Cluster https://172.16.148.3:8201 HA Mode active Active Since 2022-07-17T19:07:08.353351812Z
Configure haproxy
Install haproxy.
$ sudo apt install haproxy
Generate certificate.
$ sudo openssl req -subj "/O=HashiCorp/CN=Vault" -x509 -nodes -days 1095 -newkey rsa:4096 -keyout /etc/haproxy/tls.key -out /etc/haproxy/tls.crt
$ cat /etc/haproxy/tls.crt /etc/haproxy/tls.crt | sudo tee /etc/haproxy/vault.pem
Append haproxy configuration.
$ sudo tee --append /etc/haproxy/haproxy.cfg << EOF
resolvers consul
nameserver consul 127.0.0.1:8600
accepted_payload_size 8192
hold valid 5s
frontend https_frontend
bind *:443 ssl crt /etc/haproxy/vault.pem
default_backend vault_ui_backend
backend vault_ui_backend
server-template vault 1-3 _vault._tcp.service.consul resolvers consul resolve-prefer ipv4 check ssl verify none
EOF
Validate configuration.
$ haproxy -c -f /etc/haproxy/haproxy.cfg
Configuration file is valid
Reload service.
$ sudo systemctl reload haproxy
Configure DNS service and access service.
