Beyond static inventories in Ansible
If you ever used Ansible, you know than one of its fundamental pieces is the inventory; the inventory is nothing more than a list of machines and possibly variables where you can run your ansible playbooks.
An inventory file can be written in YAML, JSON or Windows INI format and can describe groups of machines as follows:
---
all:
children:
servers:
hosts:
macmini2:
raspberrypi:
vars:
description: Linux servers for the Nunez family
desktops:
hosts:
dmaf5:
mac-pro-1-1:
vars:
description: Desktops for the Nunez family
And you can confirm than it has the right structure too; for example filter only the machines that belong to the 'desktops' pattern:
[josevnz@dmaf5 ExtendingAnsibleWithPython]$ ansible-inventory --yaml --inventory /home/josevnz/EnableSysadmin/BashHere/hosts.yaml --graph desktops
@desktops:
|--dmaf5
|--mac-pro-1-1
Having a static YAML inventory may not be entirely practical for the following reasons:
- You host inventory is really large. Admit it, you have better things to do than to edit YAML files, right?
- Your inventory is on a format that is not compatible with Ansible YAML; It may be on a database, a plain text file
- The servers that are part of your inventory is, well really dynamic; You create machines on your private cloud as you need them and their IP address change all the time, or you have a home network with lots of roaming devices (tables, phones). You want o maintain that by hand?
What are you going to learn on this article?
There are many ways to manage your inventories in Ansible, will cover a few alternatives here:
- Converting inventories from legacy formats into Ansible
- Using dynamic inventories with plugins, specifically NMAP
- Writing our own inventory script to generate inventories dynamically
- Writing an Ansible inventory plugin
All this while following good practices of packaging our tools, using virtual environments and unit testing our code.
Don't repeat yourself: Check first if someone wrote it for you!
And chances are they did. You can quickly see if someone wrote a plugin that can handle inventory from a different source like this:
ansible-doc -t inventory -l
For example:
[josevnz@dmaf5 ExtendingAnsibleWithPython]$ ansible-doc -t inventory -l
advanced_host_list Parses a 'host list' with ranges
auto Loads and executes an inventory plugin specified in a YAML config
aws_ec2 EC2 inventory source
aws_rds rds instance source
azure_rm Azure Resource Manager inventory plugin
cloudscale cloudscale.ch inventory source
constructed Uses Jinja2 to construct vars and groups based on existing inventory
docker_machine Docker Machine inventory source
docker_swarm Ansible dynamic inventory plugin for Docker swarm nodes
foreman foreman inventory source
gcp_compute Google Cloud Compute Engine inventory source
generator Uses Jinja2 to construct hosts and groups from patterns
gitlab_runners Ansible dynamic inventory plugin for GitLab runners
hcloud Ansible dynamic inventory plugin for the Hetzner Cloud
host_list Parses a 'host list' string
ini Uses an Ansible INI file as inventory source
k8s Kubernetes (K8s) inventory source
kubevirt KubeVirt inventory source
linode Ansible dynamic inventory plugin for Linode
netbox NetBox inventory source
nmap Uses nmap to find hosts to target
online Online inventory source
openshift OpenShift inventory source
openstack OpenStack inventory source
scaleway Scaleway inventory source
script Executes an inventory script that returns JSON
toml Uses a specific TOML file as an inventory source
tower Ansible dynamic inventory plugin for Ansible Tower
virtualbox virtualbox inventory source
vmware_vm_inventory VMware Guest inventory source
vultr Vultr inventory source
yaml Uses a specific YAML file as an inventory source
Host_list plugin
It is the simplest one. You pass a list of machines or IP addresses, and you're good to go. Let's try it with the ping module, and the remote user 'josevnz':
[josevnz@dmaf5 ExtendingAnsibleWithPython]$ cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.17 mac-pro-1-1
192.168.1.16 macmini2
192.168.1.11 raspberrypi
[josevnz@dmaf5 ExtendingAnsibleWithPython]$ cat /etc/hosts| /bin/cut -f1 -d' '|/bin/grep -P '^[a-z1]'
127.0.0.1
192.168.1.17
192.168.1.16
192.168.1.11
[josevnz@dmaf5 ExtendingAnsibleWithPython]$ ansible -u josevnz -i $(/bin/cat /etc/hosts| /bin/cut -f1 -d' '|/bin/grep -P '^[a-z1]'|/bin/xargs|/bin/sed 's# #,#g') -m ping all
127.0.0.1 | SUCCESS => {
"changed": false,
"ping": "pong"
}
192.168.1.11 | SUCCESS => {
"changed": false,
"ping": "pong"
}
192.168.1.16 | SUCCESS => {
"changed": false,
"ping": "pong"
}
192.168.1.17 | SUCCESS => {
"changed": false,
"ping": "pong"
}
No surprises here. But as you can see, this is not very convenient as we had to do a little bit of Bash scripting to generate the hostlist; also the inventory is static (in the sense that is coming from the /etc/host file).
Let's move on to a more interesting plug-ing, using Nmap
Nmap plugin
The Nmap plugin allows you to use the well known network scanner to build your inventory list.
But first let's see how this works, by running nmap by hand
Crash course on Nmap
As a refresher, you can use Nmap on the command line to get a very good idea of what machines and services are on your network:
[josevnz@dmaf5 ExtendingAnsibleWithPython]$ sudo nmap -v -n -p- -sT -sV -O --osscan-limit --max-os-tries 1 -oX $HOME/home_scan.xml 192.168.1.0/24
Starting Nmap 7.80 ( https://nmap.org ) at 2022-03-05 10:29 EST
NSE: Loaded 45 scripts for scanning.
Initiating ARP Ping Scan at 10:29
Scanning 254 hosts [1 port/host]
Completed ARP Ping Scan at 10:29, 5.10s elapsed (254 total hosts)
Nmap scan report for 192.168.1.0 [host down]
Nmap scan report for 192.168.1.2 [host down]
Initiating Connect Scan at 10:29
Scanning 4 hosts [65535 ports/host]
Discovered open port 443/tcp on 192.168.1.1
Discovered open port 8080/tcp on 192.168.1.1
Discovered open port 445/tcp on 192.168.1.1
Discovered open port 139/tcp on 192.168.1.1
Discovered open port 80/tcp on 192.168.1.1
Discovered open port 80/tcp on 192.168.1.4
Discovered open port 35387/tcp on 192.168.1.4
Keep in mind than the scan above is a time-consuming operation; You are checking every port and every possible host on your network, so this may take minutes or even hours if you don't tune up your query.
With that in mind, let's keep this useful links around, will use then to tune our arguments for Nmap:
And for our inventory, we really care about machines where Ansible can SSH and perform operations. Limiting the port number to TCP 22 will speed up considerable our scanning:
# '-n': 'Never do DNS resolution',
# '-p-': 'All ports. Use -p22 to limit scan to port 22',
# '-sV': 'Probe open ports to determine service/version info',
# '-T4': 'Aggressive timing template',
# '-PE': 'Enable this echo request behavior. Good for internal networks',
# '--version-intensity 1': 'Set version scan intensity. Default is 7',
# '--disable-arp-ping': 'No ARP or ND Ping',
# '--max-hostgroup 100': 'Hostgroup (batch of hosts scanned concurrently) size',
# '--min-parallelism 20': 'Number of probes that may be outstanding for a host group',
# '--osscan-limit': 'Limit OS detection to promising targets',
# '--max-os-tries 1': 'Maximum number of OS detection tries against a target',
# '-oX -': 'Send XML output to STDOUT, avoid creating a temp file'
[josevnz@dmaf5 ExtendingAnsibleWithPython]$ nmap -v -n -p22 -sT -sV --osscan-limit --max-os-tries 1 -oX $HOME/home_scan.xml 192.168.1.0/24
Starting Nmap 7.80 ( https://nmap.org ) at 2022-03-05 10:51 EST
NSE: Loaded 45 scripts for scanning.
Initiating Ping Scan at 10:51
Scanning 256 hosts [2 ports/host]
Completed Ping Scan at 10:51, 2.31s elapsed (256 total hosts)
Nmap scan report for 192.168.1.0 [host down]
Nmap scan report for 192.168.1.2 [host down]
Nmap scan report for 192.168.1.5 [host down]
Nmap scan report for 192.168.1.7 [host down]
...
Completed NSE at 10:51, 0.00s elapsed
Nmap scan report for 192.168.1.1
Host is up (0.0024s latency).
PORT STATE SERVICE VERSION
22/tcp closed ssh
Nmap scan report for 192.168.1.3
Host is up (0.070s latency).
...
Nmap scan report for 192.168.1.11
Host is up (0.00036s latency).
PORT STATE SERVICE VERSION
22/tcp open ssh OpenSSH 8.2p1 Ubuntu 4ubuntu0.4 (Ubuntu Linux; protocol 2.0)
Service Info: OS: Linux; CPE: cpe:/o:linux:linux_kernel
...
Read data files from: /usr/bin/../share/nmap
Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 256 IP addresses (8 hosts up) scanned in 2.71 seconds
But If we don't care at all about port scanning, then we can replace the '-p22' with the '-sn' (Ping scan) flag:
[josevnz@dmaf5 ExtendingAnsibleWithPython]$ time nmap -v -n -sn --osscan-limit --max-os-tries 1 -oX $HOME/home_scan.xml 192.168.1.0/24
Read data files from: /usr/bin/../share/nmap
Nmap done: 256 IP addresses (8 hosts up) scanned in 2.52 seconds
This small difference may be a factor for bigger networks, but for our example will keep the port scanning on 22.
One more thing: going forward I will not use the '-n' switch. If passed, the DNS resolution will be disabled, and we will never know the names of the machines we just scanned.
It is possible to extend Nmap? Yes, Ansible has a nice Open Source plugin that wraps the Nmap command line program, and parses the results; To illustrate how that works I want to share with you a Python script called 'nmap_scan_rpt.py' script that can correlate services found with Nmap with security advisories (that's one of the ways you can extend the original tool).
git clone git@github.com:josevnz/home_nmap.git $HOME/home_nmap.git
pushd home_nmap/
python3 -m venv $HOME/virtualenv/home_nmap/
. ~/virtualenv/home_nmap/bin/activate
nmap_scan_rpt.py $HOME/home_scan.xml
Feel free to play with the code, you can also run Nmap as a web service, but right now let's move back to Ansible with Nmap
The Ansible nmap plugin
Now we are ready to explore the Ansible Nmap plugin;
# We do not want to do a port scan, only get the list of hosts dynamically
---
plugin: nmap
address: 192.168.1.0/24
strict: False
ipv4: yes
ports: no
groups:
appliance: "'Amazon' in hostname"
regular: "'host' in hostname"
Then let test it:
[josevnz@dmaf5 EnableSysadmin]$ ansible-inventory -i ExtendingAnsibleWithPython/Inventories/home_nmap_inventory.yaml --lis
That produces a nice JSON that will be consumed by Ansible:
{
"_meta": {
"hostvars": {
"android-1c5660ab7065af69.home": {
"ip": "192.168.1.4",
"ports": []
},
"dmaf5.home": {
"ip": "192.168.1.26",
"ports": []
}
},
"all": {
"children": [
"ungrouped"
]
},
"ungrouped": {
"hosts": [
"android-1c5660ab7065af69.home",
"dmaf5.home",
"macmini2",
"new-host-2.home",
"new-host-6.home",
"raspberrypi"
]
}
}
}
NOTE: I could not get to work the jinja2 'groups' feature, to put hosts into dynamic groups based on their hostname.
There are any other ways we can get a dynamic inventory in Ansible?
We cover a lot of material, here is a quick summary of what we learned:
- Introduced the host_list plugin and saw some of its obvious limitations, specifically when you want to use a large number of hosts
- Did a quick check on Nmap and how it can be used to scan all the hosts in your network; with that knowledge we moved on into creating an inventory using this tool.
- Configured and ran the community Ansible Nmap and compared its functionality with our manual Nmap scan.
On the next article of this series I'll show you how to write your own dynamic inventory script, and why this may be a better option than using the ready to use plugins.
Remember, you can download the code and experiment!. The best way to learn is by doing and making mistakes.