Simple strategy to run Ansible tasks in parallel

This post shows how to run some Ansible playbook tasks in parallel. The approach requires tags for all tasks you want to run in parallel.

How It Works

The script runs multiple Ansible tasks simultaneously by:

Defining a list of tags to run in parallel
Using a ThreadPoolExecutor to create a thread for each tag
Running the Ansible playbook (as a subprocess) with a specific tag in each thread

The example code (below) assumes the playbook files are part of the Python package, and uses importlib.resources to access them. But you can use any method to access the playbook files.

The Process

For each tag in your predefined list:

The script creates a thread in the thread pool
The task submitted to the thread runs the Ansible playbook with its specific tag as a subprocess. Something equivalent to:

python3 -m ansible playbook -v <playbook_path> --tags <some_tag>

The main thread waits for all tasks to complete

This approach lets you run multiple tagged tasks simultaneously instead of sequentially.

Example code:

import concurrent.futures
import logging
import subprocess
import sys
from importlib.resources import as_file
from importlib.resources import files
from pathlib import Path

logger = logging.getLogger(__name__)

AVAILABLE_TAGS = ("tag_one", "tag_two", "tag_three")


def run_ansible_playbook_tag(playbook_path: Path, tag: str):
    """
    Execute the ansible playbook tasks with the given tag.
    """
    logger.warning("Executing ansible playbook, for tag: %s", tag)

    cmd = ["-v", str(playbook_path.resolve()), "--tags", tag]

    full_cmd = [sys.executable, "-m", "ansible", "playbook"] + cmd
    logger.warning("Executing command: %s", " ".join(full_cmd))

    subprocess.run(full_cmd, check=True)


def main():
    with as_file(files("package_name").joinpath("playbooks")) as playbooks:
        playbook_path = playbooks.joinpath("my-ansible-playbook.yaml")

        with concurrent.futures.ThreadPoolExecutor() as executor:
            futures = []
            for tag in AVAILABLE_TAGS:
                fut = executor.submit(
                    run_ansible_playbook_tag, playbook_path, tag
                )
                futures.append(fut)

            for future in concurrent.futures.as_completed(futures):
                try:
                    future.result()
                except Exception as e:
                    logger.error("Error during parallel execution: %s", e)
                    raise

    return 0


if __name__ == "__main__":
    sys.exit(main())

rand[om]

Simple strategy to run Ansible tasks in parallel

How It Works

The Process

Example code: