How to install HasiCorp Vault

Install and configure HashiCorp Vault.

Install Consul application

Create consul cluster, configure encryption and access control lists.

Create access control lists and tokens

Define policy for Vault application.

$ tee vault-app-policy.hcl << EOF
# Enable the agent to update vault-X nodes
node_prefix "vault" {
  policy = "write"
}

# Provide KV visibility to all agents.
agent_prefix "" {
  policy = "read"
}

# Enable resources prefixed with 'vault/' to write to the KV
key_prefix "vault/" {
  policy = "write"
}

# Enable the vault service to write to the KV
service "vault" {
  policy = "write"
}

# Enable the agent to initialize a new session.
session_prefix "" {
  policy = "write"
}
EOF

Define vault application policy.

$ consul acl policy create -name "vault-app-policy" -description "vault application policy" -rules @vault-app-policy.hcl

ID:           a9b19576-966f-657a-0399-8e06f183cf41
Name:         vault-app-policy
Description:  vault application policy
Datacenters:  
Rules:
# Enable the agent to update vault-X nodes
node_prefix "vault" {
  policy = "write"
}

# Provide KV visibility to all agents.
agent_prefix "" {
  policy = "read"
}

# Enable resources prefixed with 'vault/' to write to the KV
key_prefix "vault/" {
  policy = "write"
}

# Enable the vault service to write to the KV
service "vault" {
  policy = "write"
}

# Enable the agent to initialize a new session.
session_prefix "" {
  policy = "write"
}

Create Vault application token.

$ consul acl token create -description "vault application token" -policy-name "vault-app-policy"

AccessorID:       42f99064-454d-cdc9-c0a2-6527d90febae
SecretID:         b12b3e74-bdf9-738b-6a81-bc35be39d769
Description:      vault application token
Local:            false
Create Time:      2022-07-16 23:26:06.157117158 +0000 UTC
Policies:
   a9b19576-966f-657a-0399-8e06f183cf41 - vault-app-policy

Define vault haproxy policy.

$ tee vault-haproxy-policy.hcl << EOF
acl = "read"

node_prefix "haproxy" {
  policy = "write"
}

node_prefix "vault" {
  policy = "read"
}

service "vault" {
  policy = "read"
}
EOF

Create Vault haproxy policy.

$ consul acl policy create -name "vault-haproxy-policy" -description "vault haproxy policy" -rules @vault-haproxy-policy.hcl

ID:           54358c22-6d4c-41dc-c2ec-0a02471fd75d
Name:         vault-haproxy-policy
Description:  vault haproxy policy
Datacenters:  
Rules:
acl = "read"

node_prefix "haproxy" {
  policy = "write"
}

node_prefix "vault" {
  policy = "read"
}

service "vault" {
  policy = "read"
}

Create Vault haproxy token.

$ consul acl token create -description "vault haproxy token" -policy-name "vault-haproxy-policy"

AccessorID:       dc6d127e-41b5-d8ac-cfa6-862834a3717f
SecretID:         86bb29c0-bc1d-13dd-1940-86e492da20fe
Description:      vault haproxy token
Local:            false
Create Time:      2022-07-16 23:24:28.248763082 +0000 UTC
Policies:
   54358c22-6d4c-41dc-c2ec-0a02471fd75d - vault-haproxy-policy

Apply tokens

Update agents on haproxy (e.g. haproxy-1, haproxy-X) machines.

$ sudo -u consul tee /etc/consul.d/acl.hcl << EOF
# acl
acl = {
  enabled = true
  default_policy = "deny"
  enable_token_persistence = true
  tokens {
    default = "86bb29c0-bc1d-13dd-1940-86e492da20fe"
  }
}
EOF

Update agents on vault (e.g. vault-1 and so on) machines.

$ sudo -u consul tee /etc/consul.d/acl.hcl << EOF
# acl
acl = {
  enabled = true
  default_policy = "deny"
  enable_token_persistence = true
  tokens {
    default = "b12b3e74-bdf9-738b-6a81-bc35be39d769"
  }
}
EOF

Install Vault application

curl and unzip should be already installed during Consul agent install process.

Visit downloads page and download linux binary.

$ curl --silent \
       --remote-name \
       --output-dir /tmp \
       https://releases.hashicorp.com/vault/1.11.0/vault_1.11.0_linux_amd64.zip

Move archive to persistent location.

$ sudo mv /tmp/vault_1.11.0_linux_amd64.zip /opt/

Inspect downloaded archive.

$ unzip -l /opt/vault_1.11.0_linux_amd64.zip

Archive:  /opt/vault_1.11.0_linux_amd64.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
199518230  2022-06-17 23:04   vault
---------                     -------
199518230                     1 file

Extract vault binary.

$ sudo unzip -d /usr/bin /opt/vault_1.11.0_linux_amd64.zip vault

Archive:  /opt/vault_1.11.0_linux_amd64.zip
  inflating: /usr/bin/vault

Inspect binary file permissions.

$ ls -l /usr/bin/vault

-rwxr-xr-x 1 root root 199518230 Jun 17 23:04 /usr/bin/vault

Display command usage.

$ vault

Usage: vault <command> [args]

Common commands:
    read        Read data and retrieves secrets
    write       Write data, configuration, and secrets
    delete      Delete secrets and configuration
    list        List data or secrets
    login       Authenticate locally
    agent       Start a Vault agent
    server      Start a Vault server
    status      Print seal and HA status
    unwrap      Unwrap a wrapped secret

Other commands:
    audit                Interact with audit devices
    auth                 Interact with auth methods
    debug                Runs the debug command
    kv                   Interact with Vault's Key-Value storage
    lease                Interact with leases
    monitor              Stream log messages from a Vault server
    namespace            Interact with namespaces
    operator             Perform operator-specific tasks
    path-help            Retrieve API help for paths
    plugin               Interact with Vault plugins and catalog
    policy               Interact with policies
    print                Prints runtime configurations
    secrets              Interact with secrets engines
    ssh                  Initiate an SSH session
    token                Interact with tokens
    version-history      Prints the version history of the target Vault server

Create vault group.

$ sudo groupadd --gid 864 vault

Create vault user.

$ sudo useradd --uid 864 --gid 864 \
               --shell /bin/false \
               --home-dir /etc/vault.d --no-create-home \
               vault

Create configuration directory.

$ sudo install --directory --group vault --owner vault --mode 700 /etc/vault.d

Create tls directory.

$ sudo install --directory --group vault --owner vault --mode 700 /opt/vault/tls

Create empty environment file.

$ sudo -u vault touch /etc/vault.d/vault.env

Create empty configuration.

$ sudo -u vault touch /etc/vault.d/vault.hcl

Generate certificate for web user interface.

$ sudo -u vault openssl req -subj "/O=HashiCorp/CN=Consul" -x509 -nodes -days 1095 -newkey rsa:4096 -keyout /opt/vault/tls/tls.key -out /opt/vault/tls/tls.crt

Limit access to these files.

$ chmod 600 /opt/vault/tls/tls.{key,crt}

Prevent memory from being swapped to disk.

$ sudo setcap cap_ipc_lock=+ep /usr/bin/vault

Create systemd service – it is the official packaged one.

$ sudo tee /usr/lib/systemd/system/vault.service << 'EOF'
[Unit]
Description="HashiCorp Vault - A tool for managing secrets"
Documentation=https://www.vaultproject.io/docs/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/vault.d/vault.hcl
StartLimitIntervalSec=60
StartLimitBurst=3

[Service]
Type=notify
EnvironmentFile=/etc/vault.d/vault.env
User=vault
Group=vault
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep-caps
AmbientCapabilities=CAP_IPC_LOCK
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
NoNewPrivileges=yes
ExecStart=/usr/bin/vault server -config=/etc/vault.d/vault.hcl
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
LimitNOFILE=65536
LimitMEMLOCK=infinity

[Install]
WantedBy=multi-user.target
EOF

[Unit]
Description="HashiCorp Vault - A tool for managing secrets"
Documentation=https://www.vaultproject.io/docs/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/vault.d/vault.hcl
StartLimitIntervalSec=60
StartLimitBurst=3

[Service]
Type=notify
EnvironmentFile=/etc/vault.d/vault.env
User=vault
Group=vault
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep-caps
AmbientCapabilities=CAP_IPC_LOCK
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
NoNewPrivileges=yes
ExecStart=/usr/bin/vault server -config=/etc/vault.d/vault.hcl
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
LimitNOFILE=65536
LimitMEMLOCK=infinity

[Install]
WantedBy=multi-user.target

Reload systemd configuration.

$ sudo systemctl --system daemon-reload

Inspect service status.

$ systemctl status vault

○ vault.service - "HashiCorp Vault - A tool for managing secrets"
     Loaded: loaded (/lib/systemd/system/vault.service; disabled; vendor preset: enabled)
     Active: inactive (dead)
       Docs: https://www.vaultproject.io/docs/

Inspect consul version.

$ vault version

Vault v1.11.0 (ea296ccf58507b25051bc0597379c467046eb2f1), built 2022-06-17T15:48:44Z

Configure server application

Create server configuration. Update systemd service to use directory instead of a single file.

$ sudo -u vault tee /etc/vault.d/vault.hcl << EOF
ui = true

storage "consul" {
  address = "127.0.0.1:8500"
  path    = "vault"
}

# HTTPS listener
listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_cert_file = "/opt/vault/tls/tls.crt"
  tls_key_file  = "/opt/vault/tls/tls.key"
}

# Enterprise license_path
#license_path = "/etc/vault.d/vault.hclic"
EOF

Enable server service.

$ sudo systemctl enable --now vault

Created symlink /etc/systemd/system/multi-user.target.wants/vault.service → /lib/systemd/system/vault.service.

Inspect service status.

$ systemctl status vault

● vault.service - "HashiCorp Vault - A tool for managing secrets"
     Loaded: loaded (/lib/systemd/system/vault.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2022-07-16 23:51:42 UTC; 19s ago
       Docs: https://www.vaultproject.io/docs/
   Main PID: 2674 (vault)
      Tasks: 7 (limit: 2241)
     Memory: 58.1M
        CPU: 112ms
     CGroup: /system.slice/vault.service
             └─2674 /usr/bin/vault server -config=/etc/vault.d/vault.hcl

Jul 16 23:51:42 vault-1 vault[2674]: ==> Vault server configuration:
Jul 16 23:51:42 vault-1 vault[2674]:              Api Address: https://172.16.148.3:8200
Jul 16 23:51:42 vault-1 vault[2674]:                      Cgo: disabled
Jul 16 23:51:42 vault-1 vault[2674]:          Cluster Address: https://172.16.148.3:8201
Jul 16 23:51:42 vault-1 vault[2674]:               Go Version: go1.17.11
Jul 16 23:51:42 vault-1 vault[2674]:               Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: ">
Jul 16 23:51:42 vault-1 vault[2674]:                Log Level: info
Jul 16 23:51:42 vault-1 vault[2674]:                    Mlock: supported: true, enabled: true
Jul 16 23:51:42 vault-1 vault[2674]:            Recovery Mode: false
Jul 16 23:51:42 vault-1 vault[2674]:                  Storage: consul (HA available)
Jul 16 23:51:42 vault-1 vault[2674]:                  Version: Vault v1.11.0, built 2022-06-17T15:48:44Z
Jul 16 23:51:42 vault-1 vault[2674]:              Version Sha: ea296ccf58507b25051bc0597379c467046eb2f1
Jul 16 23:51:42 vault-1 vault[2674]: ==> Vault server started! Log data will stream in below:
Jul 16 23:51:42 vault-1 vault[2674]: 2022-07-16T23:51:42.303Z [INFO]  proxy environment: http_proxy="" https_proxy="" no_proxy=""
Jul 16 23:51:42 vault-1 vault[2674]: 2022-07-16T23:51:42.304Z [WARN]  storage.consul: appending trailing forward slash to path
Jul 16 23:51:42 vault-1 vault[2674]: 2022-07-16T23:51:42.306Z [WARN]  no `api_addr` value specified in config or in VAULT_API_ADDR; falling back to detection if possible, but this value should be manually set
Jul 16 23:51:42 vault-1 vault[2674]: 2022-07-16T23:51:42.338Z [INFO]  core: Initializing version history cache for core
Jul 16 23:51:42 vault-1 systemd[1]: Started "HashiCorp Vault - A tool for managing secrets".

Display seal and HA status.

$ vault status -tls-skip-verify

Key                Value
---                -----
Seal Type          shamir
Initialized        false
Sealed             true
Total Shares       0
Threshold          0
Unseal Progress    0/0
Unseal Nonce       n/a
Version            1.11.0
Build Date         2022-06-17T15:48:44Z
Storage Type       consul
HA Enabled         true

Initialize vault server.

$ vault operator init  -tls-skip-verify

Unseal Key 1: SLpEi9hrk52mgxay66UnW0ZS2u1tpoQMT7Dj4sYVFoWq
Unseal Key 2: uC0VAgbWjLPYGf5mothT9bqLjSa5HW83F7VdWuFnLfAs
Unseal Key 3: jy7w3CanszT2b7SgfFAtcyq7IBOHqiOMstq+TtaZAz2G
Unseal Key 4: COo33dElKW8qLwbc/c+HbKkDe9UryCaWvd+PDcugUP+7
Unseal Key 5: sCJnTYQfYif5aUB9o49RAt6jpfVlMQSLivsDS+VFxJc4

Initial Root Token: hvs.BKpRwvIYboA6C9QrY5V7OnxA

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated root key. Without at least 3 keys to
reconstruct the root key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.

Display seal and HA status.

$ vault status -tls-skip-verify

Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    0/3
Unseal Nonce       n/a
Version            1.11.0
Build Date         2022-06-17T15:48:44Z
Storage Type       consul
HA Enabled         true

Unseal server.

$ vault operator unseal -tls-skip-verify

Unseal Key (will be hidden): 
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    1/3
Unseal Nonce       197f6b1d-562b-353d-f40e-1e96c772adf8
Version            1.11.0
Build Date         2022-06-17T15:48:44Z
Storage Type       consul
HA Enabled         true

$ vault operator unseal -tls-skip-verify

Unseal Key (will be hidden): 
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    2/3
Unseal Nonce       197f6b1d-562b-353d-f40e-1e96c772adf8
Version            1.11.0
Build Date         2022-06-17T15:48:44Z
Storage Type       consul
HA Enabled         true

$ vault operator unseal -tls-skip-verify

Unseal Key (will be hidden): 
Key                    Value
---                    -----
Seal Type              shamir
Initialized            true
Sealed                 false
Total Shares           5
Threshold              3
Version                1.11.0
Build Date             2022-06-17T15:48:44Z
Storage Type           consul
Cluster Name           vault-cluster-d49add11
Cluster ID             d73bc1ff-d088-e602-959c-cc72be323da9
HA Enabled             true
HA Cluster             n/a
HA Mode                standby
Active Node Address    <none>

Display seal and HA status.

$ vault status -tls-skip-verify

Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    5
Threshold       3
Version         1.11.0
Build Date      2022-06-17T15:48:44Z
Storage Type    consul
Cluster Name    vault-cluster-d49add11
Cluster ID      d73bc1ff-d088-e602-959c-cc72be323da9
HA Enabled      true
HA Cluster      https://172.16.148.3:8201
HA Mode         active
Active Since    2022-07-17T19:07:08.353351812Z

Configure haproxy

Install haproxy.

$ sudo apt install haproxy

Generate certificate.

$ sudo openssl req -subj "/O=HashiCorp/CN=Vault" -x509 -nodes -days 1095 -newkey rsa:4096 -keyout /etc/haproxy/tls.key -out /etc/haproxy/tls.crt

$ cat /etc/haproxy/tls.crt /etc/haproxy/tls.crt | sudo tee /etc/haproxy/vault.pem

Append haproxy configuration.

$ sudo tee --append /etc/haproxy/haproxy.cfg << EOF

resolvers consul
    nameserver consul 127.0.0.1:8600
    accepted_payload_size 8192
    hold valid 5s

frontend https_frontend
  bind *:443 ssl crt /etc/haproxy/vault.pem
  default_backend vault_ui_backend
   
backend vault_ui_backend
  server-template vault 1-3 _vault._tcp.service.consul resolvers consul resolve-prefer ipv4 check ssl verify none
EOF

Validate configuration.

$ haproxy -c -f /etc/haproxy/haproxy.cfg

Configuration file is valid

Reload service.

$ sudo systemctl reload haproxy

Configure DNS service and access service.

Install Consul application#

Create access control lists and tokens#

Apply tokens#

Install Vault application#

Configure server application#

Configure haproxy#

Install Consul application

Create access control lists and tokens

Apply tokens

Install Vault application

Configure server application

Configure haproxy