Ansible playbook starting with large forked host list, but spawning serial tasks for smaller subsets of hosts

Solution 1:

It's not possible to achieve the goal in a single playbook. A dedicated playbook for each group is needed if you want to process the groups in parallel. Use Ansible to create the code.

"Smaller subsets of hosts"

Given the inventory to simplify the testing

test_01
test_02
test_03
test_04
test_05
test_06

[GUID_01]
test_01
test_02
test_03

[GUID_02]
test_04
test_05
test_06

[foo]
test_01
test_02
test_04
test_05
srvX

[bar]
test_03
test_06
srvY

Create a dictionary of all GUID_* and a list of hosts both in particular GUID_x and foo groups. For example

- hosts: GUID_*:&foo
  gather_facts: false
  tasks:
    - set_fact:
        my_groups: "{{ my_groups|default({})|
                       combine({item: my_hosts|dict2items|
                                      selectattr('value', 'eq', item)|
                                      map(attribute='key')|list}) }}"
      loop: "{{ my_guids|unique }}"
      vars:
        my_guids: "{{ ansible_play_hosts_all|
                      map('extract', hostvars, 'group_names')|
                      map('select', 'match', 'GUID')|
                      map('first')|flatten }}"
        my_hosts: "{{ dict(ansible_play_hosts_all|zip(my_guids)) }}"
      run_once: true
    - debug:
        var: my_groups
      run_once: true

gives

  my_groups:
    GUID_01:
    - test_01
    - test_02
    GUID_02:
    - test_04
    - test_05

The same dictionary can be created without patterns

- hosts: all
  gather_facts: false
  tasks:
    - set_fact:
        my_groups: "{{ my_groups|default({})|
                       combine({item: groups[item]|
                                      intersect(groups['foo'])}) }}"
      loop: "{{ groups|select( 'match', 'GUID')|list }}"
      run_once: true

Q: "Create parallel workers that execute serial: 1 playbooks for a subset of hosts."

A: Use Ansible templates to create the code. For example, given the role

shell> cat roles/reboot/tasks/main.yml
- command: date "+%H:%M:%S"
  register: result
- debug:
    msg: "{{ result.stdout }} {{ inventory_hostname }} Reboot"
- command: sleep 3
- command: date "+%H:%M:%S"
  register: result
- debug:
    msg: "{{ result.stdout }} {{ inventory_hostname }} Ready"

The templates

shell> cat templates/my_hosts.j2
{{ lookup('file', 'hosts') }}

# ---------------------------------------------------------------------
# Groups for my_wrapper. See template my_hosts.j2

{% for group,hosts in my_groups.items() %}
[my_{{ group }}]
{% for host in hosts %}
{{ host }}
{% endfor %}

{% endfor %}

shell> cat templates/my_pb.j2
- hosts: my_{{ item }}
  gather_facts: false
  serial: 1
  roles:
    - reboot

shell> cat templates/my_wrapper.j2
#!/bin/bash
{% for group,hosts in my_groups.items() %}
nohup ansible-playbook -i my_hosts my_pb_{{ group }}.yml &
{% endfor %}

and the playbook

- hosts: all
  gather_facts: false
  tasks:
    - set_fact:
        my_groups: "{{ my_groups|default({})|
                       combine({item: groups[item]|
                                      intersect(groups['foo'])}) }}"
      loop: "{{ groups|select( 'match', 'GUID')|list }}"
      run_once: true
    - block:
        - template:
            src: my_hosts.j2
            dest: my_hosts
        - template:
            src: my_wrapper.j2
            dest: my_wrapper.sh
            mode: "a+x"
        - template:
            src: my_pb.j2
            dest: "my_pb_{{ item }}.yml"
          loop: "{{ my_groups.keys()|list }}"
      run_once: true
      delegate_to: localhost

create the files with the code

shell> cat my_hosts

   ...

# ---------------------------------------------------------------------
# Groups for my_wrapper. See template my_hosts.j2

[my_GUID_01]
test_01
test_02

[my_GUID_02]
test_04
test_05

shell> cat my_wrapper.sh
#!/bin/bash
nohup ansible-playbook -i my_hosts my_pb_GUID_01.yml &
nohup ansible-playbook -i my_hosts my_pb_GUID_02.yml &

shell> cat my_pb_GUID_01.yml 
- hosts: my_GUID_01
  gather_facts: false
  serial: 1
  roles:
    - reboot

shell> cat my_pb_GUID_02.yml 
- hosts: my_GUID_02
  gather_facts: false
  serial: 1
  roles:
    - reboot

Then, running the wrapper gives

shell> grep msg nohup.out 
    "msg": "13:03:51 test_01 Reboot"
    "msg": "13:03:51 test_04 Reboot"
    "msg": "13:03:56 test_01 Ready"
    "msg": "13:03:56 test_04 Ready"
    "msg": "13:03:58 test_02 Reboot"
    "msg": "13:03:58 test_05 Reboot"
    "msg": "13:04:04 test_02 Ready"
    "msg": "13:04:04 test_05 Ready"