How to combine group variables by parsing group configuration files

Group variables are overwritten, not merged, this is an expected behaviour, but can by circumvented by parsing group configuration files.

Ansible directory tree used to describe this solution.

$ tree --dirsfirst

.
├── inventory
│   ├── group_vars
│   │   ├── all
│   │   │   ├── package_sources.yml
│   │   │   └── users.yml
│   │   ├── application
│   │   │   ├── labels.yml
│   │   │   ├── package_sources.yml
│   │   │   └── users.yml
│   │   ├── backend
│   │   │   ├── labels.yml
│   │   │   ├── package_sources.yml
│   │   │   └── users.yml
│   │   └── frontend
│   │       ├── labels.yml
│   │       └── package_sources.yml
│   └── hosts.ini
├── labels_playbook.yml
├── package_sources_playbook.yml
└── users_playbook.yml

Split up group configuration into many small files to apply this solution.

The inventory

Inventory file with overlapping groups.

$ cat inventory/hosts.ini

[application]
beaver
wombat
[frontend]
rabbit
[backend]
beaver
wombat

Group variables defined for every host.

$ cat inventory/group_vars/all/users.yml

---
user__definitions:
  - name: milosz
    home: /home/milosz
    shell: /bin/bash
  - name: ansible
    home: /home/ansible
    shell: /bin/bash

$ cat inventory/group_vars/all/package_sources.yml

---
package__sources:
  - debian

Frontend group variables.

$ cat inventory/group_vars/frontend/package_sources.yml

---
package__sources:
  - haproxy
  - nginx
  - debian

$ cat inventory/group_vars/frontend/labels.yml

---
label__description: "frontend server"

Application group variables.

$ cat inventory/group_vars/application/users.yml

---
user__definitions:
  - name: django
    home: /home/django
    shell: /bin/bash

$ cat inventory/group_vars/application/package_sources.yml

---
package__sources:
  - mysql
  - django
  - nginx

$ cat  inventory/group_vars/application/labels.yml

---
label__description: "django application"

Backend group variables.

$ cat inventory/group_vars/backend/users.yml

---
user__definitions:
  - name: milosz
    home: /home/milosz
    shell: /bin/bash
  - name: ansible
    home: /home/ansible
    shell: /bin/bash

$ cat inventory/group_vars/backend/package_sources.yml

---
package__sources:
  - debian

$ cat inventory/group_vars/backend/labels.yml

---
label__description: "backend server"

Play with labels

Create playbook to extract group labels.

$ cat labels_playbook.yml

---
- hosts: all
  tasks:
    - name: "Define files that contain group labels"
      set_fact:
        files_with_group_labels: >-
          {%- set files_with_group_labels = [] -%}
          {%- for host_group in ['all'] + group_names -%}
            {%- set file = "inventory/group_vars/" + host_group + "/labels.yml" -%}
            {%- if file is exists -%}
              {%- set _ = files_with_group_labels.extend([file]) -%}
            {%- endif -%}
          {%- endfor -%}
          {{ files_with_group_labels }}
    - name: "Display files that contain group labels"
      debug:
        msg: "{{ files_with_group_labels }}"
    - name: "Define empty group labels"
      set_fact:
        group_labels: []
    - name: "Extract group labels"
      set_fact:
        group_labels: >-
          {%- set file = item | from_yaml -%}
          {%- if file.label__description is defined -%}
            {%- set element = file.label__description -%}
            {%- if element != "" and element not in group_labels -%}
              {%- set _ = group_labels.extend([element]) -%}
            {%- endif -%}
          {%- endif -%}
          {{ group_labels }}
      with_file: "{{ files_with_group_labels }}"
    - name: "Display group labels"
      debug:
        msg: "{{ group_labels | join(', ') }}"

Run playbook to inspect group labels.

$ ansible-playbook -i inventory/hosts.ini labels_playbook.yml

PLAY [all] *********************************************************************************************************************************************************************
TASK [Gathering Facts] *********************************************************************************************************************************************************
ok: [beaver]
ok: [rabbit]
ok: [wombat]
TASK [Define files that contain group labels] **********************************************************************************************************************************
ok: [beaver]
ok: [wombat]
ok: [rabbit]
TASK [Display files that contain group labels] *********************************************************************************************************************************
ok: [beaver] => {
    "msg": [
        "inventory/group_vars/application/labels.yml",
        "inventory/group_vars/backend/labels.yml"
    ]
}
ok: [wombat] => {
    "msg": [
        "inventory/group_vars/application/labels.yml",
        "inventory/group_vars/backend/labels.yml"
    ]
}
ok: [rabbit] => {
    "msg": [
        "inventory/group_vars/frontend/labels.yml"
    ]
}
TASK [Define empty group labels] ***********************************************************************************************************************************************
ok: [beaver]
ok: [wombat]
ok: [rabbit]
TASK [Extract group labels] ****************************************************************************************************************************************************
ok: [beaver] => (item=---
label__description: "django application")
ok: [wombat] => (item=---
label__description: "django application")
ok: [beaver] => (item=---
label__description: "backend server")
ok: [wombat] => (item=---
label__description: "backend server")
ok: [rabbit] => (item=---
label__description: "frontend server")
TASK [Display group labels] ****************************************************************************************************************************************************
ok: [beaver] => {
    "msg": "django application, backend server"
}
ok: [wombat] => {
    "msg": "django application, backend server"
}
ok: [rabbit] => {
    "msg": "frontend server"
}
PLAY RECAP *********************************************************************************************************************************************************************
beaver                     : ok=6    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
rabbit                     : ok=6    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
wombat                     : ok=6    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Play with users

Create playbook to extract group users.

$ cat users_playbook.yml

---
- hosts: all
  tasks:
    - name: "Define files that contain group users"
      set_fact:
        files_with_group_users: >-
          {%- set files_with_group_users = [] -%}
          {%- for host_group in ['all'] + group_names -%}
            {%- set file = "inventory/group_vars/" + host_group + "/users.yml" -%}
            {%- if file is exists -%}
              {%- set _ = files_with_group_users.extend([file]) -%}
            {%- endif -%}
          {%- endfor -%}
          {{ files_with_group_users }}
    - name: "Display files that contain group users"
      debug:
        msg: "{{ files_with_group_users }}"
    - name: "Define empty group users"
      set_fact:
        group_users: []
    - name: "Extract group users"
      set_fact:
        group_users: >-
          {%- set file = item | from_yaml -%}
          {%- if file.user__definitions is defined -%}
            {%- for element in file.user__definitions -%}
              {%- if element != "" and element not in group_users -%}
                {%- set _ = group_users.extend([element]) -%}
              {%- endif -%}
            {% endfor %}
          {%- endif -%}
          {{ group_users }}
      with_file: "{{ files_with_group_users }}"
    - name: "Display group users"
      debug:
        msg: "{{ group_users }}"

Run playbook to inspect group labels.

$ ansible-playbook -i inventory/hosts.ini users_playbook.yml

PLAY [all] ************************************************************************************************************************************************************[85/1905]
TASK [Gathering Facts] *********************************************************************************************************************************************************
ok: [rabbit]
ok: [wombat]
ok: [beaver]
TASK [Define files that contain group users] ***********************************************************************************************************************************
ok: [beaver]
ok: [wombat]
ok: [rabbit]
TASK [Display files that contain group users] **********************************************************************************************************************************
ok: [beaver] => {
    "msg": [
        "inventory/group_vars/all/users.yml",
        "inventory/group_vars/application/users.yml",
        "inventory/group_vars/backend/users.yml"
    ]
}
ok: [wombat] => {
    "msg": [
        "inventory/group_vars/all/users.yml",
        "inventory/group_vars/application/users.yml",
        "inventory/group_vars/backend/users.yml"
    ]
}
ok: [rabbit] => {
    "msg": [
        "inventory/group_vars/all/users.yml"
    ]
}
TASK [Define empty group users] ************************************************************************************************************************************************
ok: [beaver]
ok: [wombat]
ok: [rabbit]
TASK [Extract group users] *******************************************************************************************************************************************[47/1905]
ok: [beaver] => (item=---
user__definitions:
  - name: milosz
    home: /home/milosz
    shell: /bin/bash
  - name: ansible
    home: /home/ansible
    shell: /bin/bash)
ok: [beaver] => (item=---
user__definitions:
  - name: django
    home: /home/django
    shell: /bin/bash)
ok: [beaver] => (item=---
user__definitions:
  - name: milosz
    home: /home/milosz
    shell: /bin/bash
  - name: ansible
    home: /home/ansible
    shell: /bin/bash)
ok: [wombat] => (item=---
user__definitions:
  - name: milosz
    home: /home/milosz
    shell: /bin/bash
  - name: ansible
    home: /home/ansible
    shell: /bin/bash)
ok: [wombat] => (item=---
user__definitions:
  - name: django
    home: /home/django
    shell: /bin/bash)
ok: [rabbit] => (item=---
user__definitions:
  - name: milosz
    home: /home/milosz
    shell: /bin/bash
  - name: ansible
    home: /home/ansible
    shell: /bin/bash)
ok: [wombat] => (item=---
user__definitions:
  - name: milosz
    home: /home/milosz
    shell: /bin/bash
  - name: ansible
    home: /home/ansible
    shell: /bin/bash)
TASK [Display group users] *****************************************************************************************************************************************************
ok: [beaver] => {
    "msg": [
        {
            "home": "/home/milosz",
            "name": "milosz",
            "shell": "/bin/bash"
        },
        {
            "home": "/home/ansible",
            "name": "ansible",
            "shell": "/bin/bash"
        },
        {
            "home": "/home/django",
            "name": "django",
            "shell": "/bin/bash"
        }
    ]
}
ok: [wombat] => {
    "msg": [
        {
            "home": "/home/milosz",
            "name": "milosz",
            "shell": "/bin/bash"
        },
        {
            "home": "/home/ansible",
            "name": "ansible",
            "shell": "/bin/bash"
        },
        {
            "home": "/home/django",
            "name": "django",
            "shell": "/bin/bash"
        }
    ]
}
ok: [rabbit] => {
    "msg": [
        {
            "home": "/home/milosz",
            "name": "milosz",
            "shell": "/bin/bash"
        },
        {
            "home": "/home/ansible",
            "name": "ansible",
            "shell": "/bin/bash"
        }
    ]
}
PLAY RECAP *********************************************************************************************************************************************************************
beaver                     : ok=6    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
rabbit                     : ok=6    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
wombat                     : ok=6    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Play with package sources

Create playbook to extract group labels.

$ cat package_sources_playbook.yml

---
- hosts: all
  tasks:
    - name: "Define files that contain package sources"
      set_fact:
        files_with_group_package_sources: >-
          {%- set files_with_group_package_sources = [] -%}
          {%- for host_group in ['all'] + group_names -%}
            {%- set file = "inventory/group_vars/" + host_group + "/package_sources.yml" -%}
            {%- if file is exists -%}
              {%- set _ = files_with_group_package_sources.extend([file]) -%}
            {%- endif -%}
          {%- endfor -%}
          {{ files_with_group_package_sources }}
    - name: "Display files that contain group package sources"
      debug:
        msg: "{{ files_with_group_package_sources }}"
    - name: "Define empty group package sources"
      set_fact:
        group_package_sources: []
    - name: "Extract group package sources"
      set_fact:
        group_package_sources: >-
          {%- set file = item | from_yaml -%}
          {%- if file.package__sources is defined -%}
            {%- for element in file.package__sources -%}
              {%- if element != "" and element not in group_package_sources -%}
                {%- set _ = group_package_sources.extend([element]) -%}
              {%- endif -%}
            {% endfor %}
          {%- endif -%}
          {{ group_package_sources }}
      with_file: "{{ files_with_group_package_sources }}"
    - name: "Display group users"
      debug:
        msg: "{{ group_package_sources }}"

Run playbook to inspect group labels.

$ ansible-playbook -i inventory/hosts.ini package_sources_playbook.yml

PLAY [all] ************************************************************************************************************************************************************[37/1917]
TASK [Gathering Facts] *********************************************************************************************************************************************************
ok: [beaver]
ok: [wombat]
ok: [rabbit]
TASK [Define files that contain package sources] *******************************************************************************************************************************
ok: [beaver]
ok: [wombat]
ok: [rabbit]
TASK [Display files that contain group package sources] ************************************************************************************************************************
ok: [beaver] => {
    "msg": [
        "inventory/group_vars/all/package_sources.yml",
        "inventory/group_vars/application/package_sources.yml",
        "inventory/group_vars/backend/package_sources.yml"
    ]
}
ok: [wombat] => {
    "msg": [
        "inventory/group_vars/all/package_sources.yml",
        "inventory/group_vars/application/package_sources.yml",
        "inventory/group_vars/backend/package_sources.yml"
    ]
}
ok: [rabbit] => {
    "msg": [
        "inventory/group_vars/all/package_sources.yml",
        "inventory/group_vars/frontend/package_sources.yml"
    ]
}
TASK [Define empty group package sources] **************************************************************************************************************************************
ok: [beaver]
ok: [wombat]
ok: [rabbit]
TASK [Extract group package sources] *******************************************************************************************************************************************
ok: [beaver] => (item=---
package__sources:
  - debian)
ok: [beaver] => (item=---
package__sources:
  - mysql
  - django
  - nginx)
ok: [beaver] => (item=---
package__sources:
  - debian)
ok: [wombat] => (item=---
package__sources:
  - debian)
ok: [wombat] => (item=---
package__sources:
  - mysql
  - django
  - nginx)
ok: [rabbit] => (item=---
package__sources:
  - debian)
ok: [wombat] => (item=---
package__sources:
  - debian)
ok: [rabbit] => (item=---
package__sources:
  - haproxy
  - nginx
  - debian)
TASK [Display group users] *****************************************************************************************************************************************************
ok: [beaver] => {
    "msg": [
        "debian",
        "mysql",
        "django",
        "nginx"
    ]
}
ok: [wombat] => {
    "msg": [
        "debian",
        "mysql",
        "django",
        "nginx"
    ]
}
ok: [rabbit] => {
    "msg": [
        "debian",
        "haproxy",
        "nginx"
    ]
}
PLAY RECAP *********************************************************************************************************************************************************************
beaver                     : ok=6    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
rabbit                     : ok=6    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
wombat                     : ok=6    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Additional notes

This is just an example. You can define these variables using group configuration files or roles directly.

The best solution is to use package_sources_all (single instance), package_sources_group (merge multiple instances) and package_sources_host (single instance) and combine these into a single list inside a role.

The inventory#

Play with labels#

Play with users#

Play with package sources#

Additional notes#

The inventory

Play with labels

Play with users

Play with package sources

Additional notes