Install and configure HashiCorp Vault.
Install Consul application
Create consul cluster, configure encryption and access control lists.
Create access control lists and tokens
Define policy for Vault application.
$ tee vault-app-policy.hcl << EOF # Enable the agent to update vault-X nodes node_prefix "vault" { policy = "write" } # Provide KV visibility to all agents. agent_prefix "" { policy = "read" } # Enable resources prefixed with 'vault/' to write to the KV key_prefix "vault/" { policy = "write" } # Enable the vault service to write to the KV service "vault" { policy = "write" } # Enable the agent to initialize a new session. session_prefix "" { policy = "write" } EOF
Define vault application policy.
$ consul acl policy create -name "vault-app-policy" -description "vault application policy" -rules @vault-app-policy.hcl
ID: a9b19576-966f-657a-0399-8e06f183cf41 Name: vault-app-policy Description: vault application policy Datacenters: Rules: # Enable the agent to update vault-X nodes node_prefix "vault" { policy = "write" } # Provide KV visibility to all agents. agent_prefix "" { policy = "read" } # Enable resources prefixed with 'vault/' to write to the KV key_prefix "vault/" { policy = "write" } # Enable the vault service to write to the KV service "vault" { policy = "write" } # Enable the agent to initialize a new session. session_prefix "" { policy = "write" }
Create Vault application token.
$ consul acl token create -description "vault application token" -policy-name "vault-app-policy"
AccessorID: 42f99064-454d-cdc9-c0a2-6527d90febae SecretID: b12b3e74-bdf9-738b-6a81-bc35be39d769 Description: vault application token Local: false Create Time: 2022-07-16 23:26:06.157117158 +0000 UTC Policies: a9b19576-966f-657a-0399-8e06f183cf41 - vault-app-policy
Define vault haproxy policy.
$ tee vault-haproxy-policy.hcl << EOF acl = "read" node_prefix "haproxy" { policy = "write" } node_prefix "vault" { policy = "read" } service "vault" { policy = "read" } EOF
Create Vault haproxy policy.
$ consul acl policy create -name "vault-haproxy-policy" -description "vault haproxy policy" -rules @vault-haproxy-policy.hcl
ID: 54358c22-6d4c-41dc-c2ec-0a02471fd75d Name: vault-haproxy-policy Description: vault haproxy policy Datacenters: Rules: acl = "read" node_prefix "haproxy" { policy = "write" } node_prefix "vault" { policy = "read" } service "vault" { policy = "read" }
Create Vault haproxy token.
$ consul acl token create -description "vault haproxy token" -policy-name "vault-haproxy-policy"
AccessorID: dc6d127e-41b5-d8ac-cfa6-862834a3717f SecretID: 86bb29c0-bc1d-13dd-1940-86e492da20fe Description: vault haproxy token Local: false Create Time: 2022-07-16 23:24:28.248763082 +0000 UTC Policies: 54358c22-6d4c-41dc-c2ec-0a02471fd75d - vault-haproxy-policy
Apply tokens
Update agents on haproxy (e.g. haproxy-1, haproxy-X) machines.
$ sudo -u consul tee /etc/consul.d/acl.hcl << EOF # acl acl = { enabled = true default_policy = "deny" enable_token_persistence = true tokens { default = "86bb29c0-bc1d-13dd-1940-86e492da20fe" } } EOF
Update agents on vault (e.g. vault-1 and so on) machines.
$ sudo -u consul tee /etc/consul.d/acl.hcl << EOF # acl acl = { enabled = true default_policy = "deny" enable_token_persistence = true tokens { default = "b12b3e74-bdf9-738b-6a81-bc35be39d769" } } EOF
Install Vault application
curl
and unzip
should be already installed during Consul agent install process.
Visit downloads page and download linux binary.
$ curl --silent \ --remote-name \ --output-dir /tmp \ https://releases.hashicorp.com/vault/1.11.0/vault_1.11.0_linux_amd64.zip
Move archive to persistent location.
$ sudo mv /tmp/vault_1.11.0_linux_amd64.zip /opt/
Inspect downloaded archive.
$ unzip -l /opt/vault_1.11.0_linux_amd64.zip
Archive: /opt/vault_1.11.0_linux_amd64.zip Length Date Time Name --------- ---------- ----- ---- 199518230 2022-06-17 23:04 vault --------- ------- 199518230 1 file
Extract vault binary.
$ sudo unzip -d /usr/bin /opt/vault_1.11.0_linux_amd64.zip vault
Archive: /opt/vault_1.11.0_linux_amd64.zip inflating: /usr/bin/vault
Inspect binary file permissions.
$ ls -l /usr/bin/vault
-rwxr-xr-x 1 root root 199518230 Jun 17 23:04 /usr/bin/vault
Display command usage.
$ vault
Usage: vault <command> [args] Common commands: read Read data and retrieves secrets write Write data, configuration, and secrets delete Delete secrets and configuration list List data or secrets login Authenticate locally agent Start a Vault agent server Start a Vault server status Print seal and HA status unwrap Unwrap a wrapped secret Other commands: audit Interact with audit devices auth Interact with auth methods debug Runs the debug command kv Interact with Vault's Key-Value storage lease Interact with leases monitor Stream log messages from a Vault server namespace Interact with namespaces operator Perform operator-specific tasks path-help Retrieve API help for paths plugin Interact with Vault plugins and catalog policy Interact with policies print Prints runtime configurations secrets Interact with secrets engines ssh Initiate an SSH session token Interact with tokens version-history Prints the version history of the target Vault server
Create vault group.
$ sudo groupadd --gid 864 vault
Create vault user.
$ sudo useradd --uid 864 --gid 864 \ --shell /bin/false \ --home-dir /etc/vault.d --no-create-home \ vault
Create configuration directory.
$ sudo install --directory --group vault --owner vault --mode 700 /etc/vault.d
Create tls directory.
$ sudo install --directory --group vault --owner vault --mode 700 /opt/vault/tls
Create empty environment file.
$ sudo -u vault touch /etc/vault.d/vault.env
Create empty configuration.
$ sudo -u vault touch /etc/vault.d/vault.hcl
Generate certificate for web user interface.
$ sudo -u vault openssl req -subj "/O=HashiCorp/CN=Consul" -x509 -nodes -days 1095 -newkey rsa:4096 -keyout /opt/vault/tls/tls.key -out /opt/vault/tls/tls.crt
Limit access to these files.
$ chmod 600 /opt/vault/tls/tls.{key,crt}
Prevent memory from being swapped to disk.
$ sudo setcap cap_ipc_lock=+ep /usr/bin/vault
Create systemd service – it is the official packaged one.
$ sudo tee /usr/lib/systemd/system/vault.service << 'EOF' [Unit] Description="HashiCorp Vault - A tool for managing secrets" Documentation=https://www.vaultproject.io/docs/ Requires=network-online.target After=network-online.target ConditionFileNotEmpty=/etc/vault.d/vault.hcl StartLimitIntervalSec=60 StartLimitBurst=3 [Service] Type=notify EnvironmentFile=/etc/vault.d/vault.env User=vault Group=vault ProtectSystem=full ProtectHome=read-only PrivateTmp=yes PrivateDevices=yes SecureBits=keep-caps AmbientCapabilities=CAP_IPC_LOCK CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK NoNewPrivileges=yes ExecStart=/usr/bin/vault server -config=/etc/vault.d/vault.hcl ExecReload=/bin/kill --signal HUP $MAINPID KillMode=process KillSignal=SIGINT Restart=on-failure RestartSec=5 TimeoutStopSec=30 LimitNOFILE=65536 LimitMEMLOCK=infinity [Install] WantedBy=multi-user.target EOF
[Unit] Description="HashiCorp Vault - A tool for managing secrets" Documentation=https://www.vaultproject.io/docs/ Requires=network-online.target After=network-online.target ConditionFileNotEmpty=/etc/vault.d/vault.hcl StartLimitIntervalSec=60 StartLimitBurst=3 [Service] Type=notify EnvironmentFile=/etc/vault.d/vault.env User=vault Group=vault ProtectSystem=full ProtectHome=read-only PrivateTmp=yes PrivateDevices=yes SecureBits=keep-caps AmbientCapabilities=CAP_IPC_LOCK CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK NoNewPrivileges=yes ExecStart=/usr/bin/vault server -config=/etc/vault.d/vault.hcl ExecReload=/bin/kill --signal HUP $MAINPID KillMode=process KillSignal=SIGINT Restart=on-failure RestartSec=5 TimeoutStopSec=30 LimitNOFILE=65536 LimitMEMLOCK=infinity [Install] WantedBy=multi-user.target
Reload systemd configuration.
$ sudo systemctl --system daemon-reload
Inspect service status.
$ systemctl status vault
○ vault.service - "HashiCorp Vault - A tool for managing secrets" Loaded: loaded (/lib/systemd/system/vault.service; disabled; vendor preset: enabled) Active: inactive (dead) Docs: https://www.vaultproject.io/docs/
Inspect consul version.
$ vault version
Vault v1.11.0 (ea296ccf58507b25051bc0597379c467046eb2f1), built 2022-06-17T15:48:44Z
Configure server application
Create server configuration. Update systemd service to use directory instead of a single file.
$ sudo -u vault tee /etc/vault.d/vault.hcl << EOF ui = true storage "consul" { address = "127.0.0.1:8500" path = "vault" } # HTTPS listener listener "tcp" { address = "0.0.0.0:8200" tls_cert_file = "/opt/vault/tls/tls.crt" tls_key_file = "/opt/vault/tls/tls.key" } # Enterprise license_path #license_path = "/etc/vault.d/vault.hclic" EOF
Enable server service.
$ sudo systemctl enable --now vault
Created symlink /etc/systemd/system/multi-user.target.wants/vault.service → /lib/systemd/system/vault.service.
Inspect service status.
$ systemctl status vault
● vault.service - "HashiCorp Vault - A tool for managing secrets" Loaded: loaded (/lib/systemd/system/vault.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2022-07-16 23:51:42 UTC; 19s ago Docs: https://www.vaultproject.io/docs/ Main PID: 2674 (vault) Tasks: 7 (limit: 2241) Memory: 58.1M CPU: 112ms CGroup: /system.slice/vault.service └─2674 /usr/bin/vault server -config=/etc/vault.d/vault.hcl Jul 16 23:51:42 vault-1 vault[2674]: ==> Vault server configuration: Jul 16 23:51:42 vault-1 vault[2674]: Api Address: https://172.16.148.3:8200 Jul 16 23:51:42 vault-1 vault[2674]: Cgo: disabled Jul 16 23:51:42 vault-1 vault[2674]: Cluster Address: https://172.16.148.3:8201 Jul 16 23:51:42 vault-1 vault[2674]: Go Version: go1.17.11 Jul 16 23:51:42 vault-1 vault[2674]: Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "> Jul 16 23:51:42 vault-1 vault[2674]: Log Level: info Jul 16 23:51:42 vault-1 vault[2674]: Mlock: supported: true, enabled: true Jul 16 23:51:42 vault-1 vault[2674]: Recovery Mode: false Jul 16 23:51:42 vault-1 vault[2674]: Storage: consul (HA available) Jul 16 23:51:42 vault-1 vault[2674]: Version: Vault v1.11.0, built 2022-06-17T15:48:44Z Jul 16 23:51:42 vault-1 vault[2674]: Version Sha: ea296ccf58507b25051bc0597379c467046eb2f1 Jul 16 23:51:42 vault-1 vault[2674]: ==> Vault server started! Log data will stream in below: Jul 16 23:51:42 vault-1 vault[2674]: 2022-07-16T23:51:42.303Z [INFO] proxy environment: http_proxy="" https_proxy="" no_proxy="" Jul 16 23:51:42 vault-1 vault[2674]: 2022-07-16T23:51:42.304Z [WARN] storage.consul: appending trailing forward slash to path Jul 16 23:51:42 vault-1 vault[2674]: 2022-07-16T23:51:42.306Z [WARN] no `api_addr` value specified in config or in VAULT_API_ADDR; falling back to detection if possible, but this value should be manually set Jul 16 23:51:42 vault-1 vault[2674]: 2022-07-16T23:51:42.338Z [INFO] core: Initializing version history cache for core Jul 16 23:51:42 vault-1 systemd[1]: Started "HashiCorp Vault - A tool for managing secrets".
Display seal and HA status.
$ vault status -tls-skip-verify
Key Value --- ----- Seal Type shamir Initialized false Sealed true Total Shares 0 Threshold 0 Unseal Progress 0/0 Unseal Nonce n/a Version 1.11.0 Build Date 2022-06-17T15:48:44Z Storage Type consul HA Enabled true
Initialize vault server.
$ vault operator init -tls-skip-verify
Unseal Key 1: SLpEi9hrk52mgxay66UnW0ZS2u1tpoQMT7Dj4sYVFoWq Unseal Key 2: uC0VAgbWjLPYGf5mothT9bqLjSa5HW83F7VdWuFnLfAs Unseal Key 3: jy7w3CanszT2b7SgfFAtcyq7IBOHqiOMstq+TtaZAz2G Unseal Key 4: COo33dElKW8qLwbc/c+HbKkDe9UryCaWvd+PDcugUP+7 Unseal Key 5: sCJnTYQfYif5aUB9o49RAt6jpfVlMQSLivsDS+VFxJc4 Initial Root Token: hvs.BKpRwvIYboA6C9QrY5V7OnxA Vault initialized with 5 key shares and a key threshold of 3. Please securely distribute the key shares printed above. When the Vault is re-sealed, restarted, or stopped, you must supply at least 3 of these keys to unseal it before it can start servicing requests. Vault does not store the generated root key. Without at least 3 keys to reconstruct the root key, Vault will remain permanently sealed! It is possible to generate new unseal keys, provided you have a quorum of existing unseal keys shares. See "vault operator rekey" for more information.
Display seal and HA status.
$ vault status -tls-skip-verify
Key Value --- ----- Seal Type shamir Initialized true Sealed true Total Shares 5 Threshold 3 Unseal Progress 0/3 Unseal Nonce n/a Version 1.11.0 Build Date 2022-06-17T15:48:44Z Storage Type consul HA Enabled true
Unseal server.
$ vault operator unseal -tls-skip-verify
Unseal Key (will be hidden): Key Value --- ----- Seal Type shamir Initialized true Sealed true Total Shares 5 Threshold 3 Unseal Progress 1/3 Unseal Nonce 197f6b1d-562b-353d-f40e-1e96c772adf8 Version 1.11.0 Build Date 2022-06-17T15:48:44Z Storage Type consul HA Enabled true
$ vault operator unseal -tls-skip-verify
Unseal Key (will be hidden): Key Value --- ----- Seal Type shamir Initialized true Sealed true Total Shares 5 Threshold 3 Unseal Progress 2/3 Unseal Nonce 197f6b1d-562b-353d-f40e-1e96c772adf8 Version 1.11.0 Build Date 2022-06-17T15:48:44Z Storage Type consul HA Enabled true
$ vault operator unseal -tls-skip-verify
Unseal Key (will be hidden): Key Value --- ----- Seal Type shamir Initialized true Sealed false Total Shares 5 Threshold 3 Version 1.11.0 Build Date 2022-06-17T15:48:44Z Storage Type consul Cluster Name vault-cluster-d49add11 Cluster ID d73bc1ff-d088-e602-959c-cc72be323da9 HA Enabled true HA Cluster n/a HA Mode standby Active Node Address <none>
Display seal and HA status.
$ vault status -tls-skip-verify
Key Value --- ----- Seal Type shamir Initialized true Sealed false Total Shares 5 Threshold 3 Version 1.11.0 Build Date 2022-06-17T15:48:44Z Storage Type consul Cluster Name vault-cluster-d49add11 Cluster ID d73bc1ff-d088-e602-959c-cc72be323da9 HA Enabled true HA Cluster https://172.16.148.3:8201 HA Mode active Active Since 2022-07-17T19:07:08.353351812Z
Configure haproxy
Install haproxy.
$ sudo apt install haproxy
Generate certificate.
$ sudo openssl req -subj "/O=HashiCorp/CN=Vault" -x509 -nodes -days 1095 -newkey rsa:4096 -keyout /etc/haproxy/tls.key -out /etc/haproxy/tls.crt
$ cat /etc/haproxy/tls.crt /etc/haproxy/tls.crt | sudo tee /etc/haproxy/vault.pem
Append haproxy configuration.
$ sudo tee --append /etc/haproxy/haproxy.cfg << EOF resolvers consul nameserver consul 127.0.0.1:8600 accepted_payload_size 8192 hold valid 5s frontend https_frontend bind *:443 ssl crt /etc/haproxy/vault.pem default_backend vault_ui_backend backend vault_ui_backend server-template vault 1-3 _vault._tcp.service.consul resolvers consul resolve-prefer ipv4 check ssl verify none EOF
Validate configuration.
$ haproxy -c -f /etc/haproxy/haproxy.cfg
Configuration file is valid
Reload service.
$ sudo systemctl reload haproxy
Configure DNS service and access service.