Deploying a Red Hat OpenStack Platform 12 lab (part II)

Introduction

Before the holidays, I gave sample directions for setting up a virtual infrastructure through KVM. Your set up may vary somewhat, depending on networking needs and virtualization strategy; you should, however, have the base operating system for OpenStack Director installed at this point, as well as some blank virtual machines.

In this post, I’ll walk through the installation of the undercloud, also known as the Red Hat OpenStack Director. The first thing you’ll need to do is provide your system with a valid subscription. This will look differently if you’re using Satellite – I’ll focus on registering through the Red Hat CDN.

Red Hat subscription

[root@undercloud ~]# subscription-manager register
Registering to: subscription.rhsm.redhat.com:443/subscription
Username: [username]
Password: *********

Start by listing the available pools so that you can attach to right one, you’re looking for the appropriate repositories you’ll be enabling below.

[root@undercloud ~]# subscription-manager list --all --available
[root@undercloud ~]# subscription-manager attach --pool=[POOLNUM]

[root@undercloud ~]# subscription-manager repos --disable=*
subscription-manager repos \
--enable=rhel-7-server-rpms \
--enable=rhel-7-server-extras-rpms \
--enable=rhel-7-server-rh-common-rpms \
--enable=rhel-ha-for-rhel-7-server-rpms \
--enable=rhel-7-server-openstack-12-rpm

Once registered and subscribed to the correct repositories you’ll want to update before going further. Be sure to reboot as well – as any kernel updates will need this step to take affect.

yum -y update ; reboot

Configure provisioning network interface

When the VM was created in the last post, it was done so with two networks. eth1 was configured during the creation of the VM for bridged (outside Internet) access. eth0 will be used for provisioning. The provisioning network can be assigned any subnet or IP address, use what makes sense to you, and keep in mind consistency as you assign addresses to other VMs’ interfaces on this netork.

cat > /etc/sysconfig/network-scripts/ifcfg-eth0 << EOF DEVICE="eth0" NAME="eth0" HWADDR="$(ip addr show eth0 | awk '/ether/ {print $2}')" IPADDR="192.168.101.101" NETMASK="255.255.255.0" EOF; ifup eth0

Install preliminary packages

Here we'll install python-tripleoclient and crudini. The former is the package which will assist in the installation of the OpenStack Director, while the latter is for modifying ini based files. We'll need the crudini package for a later step.

sudo yum install -y python-tripleoclient crudini

A few others steps needed for preparation...

First, you won't be able to run the installation process for Director as root, the script will check your login and halt the process if you attempt to do so. Instead, it is fairly common convention to create a user "stack" for this purpose. This user will also need full sudo privileges.

useradd stack
echo "redhat" | passwd --stdin stack
echo "stack ALL=(root) NOPASSWD:ALL" | tee -a /etc/sudoers.d/stack
chmod 0440 /etc/sudoers.d/stack

Next, configure the Director for routing. Routing can be useful for the compute nodes as opposed to giving them direct internet access through the external network.

cat >> /etc/sysctl.conf << EOF
net.ipv4.ip_forward = 1
EOF
sysctl -p /etc/sysctl.conf

If the Director's hostname is not in /etc/hosts, the installation will fail.

echo -e "192.168.101.101\tundercloud.servermain.local\tundercloud" >> /etc/hosts

Login as stack for the next steps: su - stack

Before the actual installation, you'll need an undercloud.conf file in the directory you're running the installation from. Typically this will be stack's home directory. You can copy a sample from /usr/share/instack-undercloud/undercloud.conf.sample, and modify that for your custom needs. If you've done this before, you can take note of all the settings, and apply them in a script. Take a look first at the settings we'll apply to this file:

Overview of settings

These settings are referenced from documentation found here.

local_ip: The IP address defined for the director’s Provisioning NIC.

undercloud_public_host: The IP address defined for the director’s Public API when using SSL/TLS.

undercloud_admin_host: The IP address defined for the director’s Admin API when using SSL/TLS.

local_interface: The chosen interface for the director’s Provisioning NIC.

masquerade_network: Defines the network that will masquerade for external access.

dhcp_start & dhcp_end: The start and end of the DHCP allocation range for overcloud nodes.

inspection_iprange: A range of IP address that the director’s introspection service uses during the PXE boot and provisioning process.

network_cidr: The network that the director uses to manage overcloud instances. This is the Provisioning network.

network_gateway: The gateway for the overcloud instances. This is the undercloud host.

Remember when we installed crudini? Using this tool we can plug our values into a script that will build undercloud.conf. The syntax works like this:

crudini --set <FILE> <SECTION> <SETTING> <VALUE>

The resulting script for my set up then, looks like this:

crudini --set undercloud.conf DEFAULT local_ip 192.168.101.101/24
crudini --set undercloud.conf DEFAULT undercloud_public_host 192.168.101.102/2
crudini --set undercloud.conf DEFAULT undercloud_admin_host 192.168.101.103/24
crudini --set undercloud.conf DEFAULT local_interface eth0
crudini --set undercloud.conf DEFAULT masquerade_network 192.168.101.0/24
crudini --set undercloud.conf DEFAULT dhcp_start 192.168.101.10
crudini --set undercloud.conf DEFAULT dhcp_end 192.168.101.20
crudini --set undercloud.conf DEFAULT inspection_iprange 192.168.101.21,192.168.101.30
crudini --set undercloud.conf DEFAULT network_cidr 192.168.101.0/24
crudini --set undercloud.conf DEFAULT network_gateway 192.168.101.101

These commands create the following file:

[DEFAULT]
local_ip = 192.168.101.101/24
undercloud_public_host = 192.168.101.102/2
undercloud_admin_host = 192.168.101.103/24
local_interface = eth0
masquerade_network = 192.168.101.0/24
dhcp_start = 192.168.101.10
dhcp_end = 192.168.101.20
inspection_iprange = 192.168.101.21,192.168.101.30
network_cidr = 192.168.101.0/24
network_gateway = 192.168.101.101

Finally, run the command that installs the undercloud.

The actual installation: openstack undercloud install

Be patient as the actual undercloud installation takes some time. Now would be a good time for a coffee break. Got it? OK - we're almost there!

Uploading overcloud images into Glance

The last step will be aquire the images to be copied to your overcloud servers, and get them into Glance. Specifically, this is the undercloud's image service, not the overcloud's. Start with the installation:

sudo yum -y install rhosp-director-images rhosp-director-images-ipa

Next, create an images directory to hold the extracted images; the compressed files you need are in /usr/share/rhosp-director-images.

mkdir ~/images; cd images

for i in /usr/share/rhosp-director-images/overcloud-full-* \
/usr/share/rhosp-director-images/ironic-python-agent-latest-12.0.tar; \
do tar xvf $i; done

Upload the images to be glance:

cd ..; source stackrc
openstack overcloud image upload --image-path /home/stack/images/

In the next post, we'll prepare for and deploy the overcloud.

< Previous | Next >

Deploying a Red Hat OpenStack Platform 12 lab (part I)

Introduction

Red Hat OpenStack Platform is installed and configured through a management node called RHOSP Director, also known as the undercloud. This node is based on the Ironic project for baremetal node management, and Tripleo (OpenStack on OpenStack) for overcloud deployment and configuration. This post will walk through a very simple installation of Red Hat’s newest distribution of OpenStack.

Prerequisites

This example installation will be done via a virtualized platform, KVM specifically. This post will mostly address setting up this virtual environment, you will need:

  • Five internal networks
  • One bridged network
  • 3 – 7 VMs with 8 GBs of memory a piece
  • Appropriate Red Hat Enterprise Linux subscriptions

Description of virtual networks

Provisioning: This network is used by the RHOSP Director to push Red Hat images to other servers (the overcloud), which will make up the actual OpenStack installation.

Tenant: This, typically a VXLAN network, is used for instance traffic.

Internal & External API networks: The Internal API network is used for internal cluster communications, the External API is used for API and Horizon access for external access.

Storage & Storage_Cluster:
One network is for access to Storage, the other network is for internal storage cluster communication.

As this will begin with a very simple installation, many of these networks will initially go unused. Creating them now will allow for more complex deployments going forward.

Configuring the Virtual environment

On your virtualization host, create a directory structure to keep a back up of the configuration files created. These can be used for reference, as well as backed up if the environment needs to be rebuilt later. Mine looks something like the below.

files/
├── host_interfaces
├── undercloud_interfaces
├── virtualNetworks
└── vms

Start by creating a bridge interface for the external API network. To create a bridge interface, start by configuring the IP address in /etc/sysconfig/network-scripts/ifcfg-br0. The below configuration is an example only – update this to work for your needs.

cat > ./files/host_interfaces/ifcfg-br0 << EOF
TYPE=Bridge
DEVICE=br0
BOOTPROTO=none
NAME=br0
ONBOOT=yes
DELAY=0
IPADDR=192.168.0.84
PREFIX=24
GATEWAY=192.168.0.1
DNS1=8.8.8.8
DNS2=8.8.4.4
EOF

cp files/host_interfaces/ifcfg-br0 /etc/sysconfig/network-scripts/ifcfg-br0

Then, link a physical interface to it. Example:

cat > ./files/host_interfaces/ifcfg-enp5s0 << EOF
TYPE=Ethernet
DEVICE=enp5s0
BOOTPROTO=none
HWADDR=1d:1c:0a:e4:16:f1
ONBOOT=yes
BRIDGE=br0
NM_CONTROLLED=no
EOF

cp ./files/host_interfaces/ifcfg-enp5s0 /etc/sysconfig/network-scripts/ifcfg-enp5s0

Create a file to contain the configuration for the bridged virtual network.

cat > ./files/virtualNetworks/net-br0 < <name>net-br0</name>
<forward mode=’bridge’/>
<bridge name=’br0’/>
</network>
EOF

Using this file, create the virtual network.

virsh net-define ./files/virtualNetworks/net-br0
virsh net-start net-br0
virh net-autostart net-br0

Create the rest of the networks through virt-manager. These networks should be basic, internal-only networks without any DHCP enabled. It should look something like this:

Name the network:

The subnet doesn’t matter, keep your specific networking requirements in minde:

No, just no.

Nothing else needed here, just finish up.

Don’t forget to add these networks to the directory structure created earlier for backups.
Example:

virsh net-dumpxml Provisioning > ./files/virtualNetworks/Provisioning.txt

Once these networks are created, I create the VMs starting with the Director. The Director needs more attention than the other VMs because, unlike the others, an operating needs to be installed and configured. Instead of going through the installation process, I like cloning Red Hat’s downloadable cloud image.

You can download Red Hat’s cloud image here. (Subscription needed.)

Start by creating the disk:

qemu-img create -f qcow2 /var/lib/libvirt/images/undercloud.qcow2 100G

For whatever reason, I get errors if I don’t run this command before using virt-resize or virt-customize. I haven’t looked into it, I just run this and move on.

export LIBGUESTFS_BACKEND=direct

Expand the /dev/sda1 partition of the downloaded cloud image on to that newly created qcow2.

virt-resize --expand /dev/sda1 \
/var/lib/libvirt/images/vanilla/rhel-server-7.4-x86_64-kvm.qcow2 \
/var/lib/libvirt/images/undercloud.qcow2

I like to use virt-customize to enable password authentication for easy access, removing cloud-init, and doing some intitial network interface configuration.

virt-customize -a \
/var/lib/libvirt/images/undercloud.qcow2 \
--root-password password:redhat \
--hostname undercloud.servermain.local \
--edit /etc/ssh/sshd_config:s/PasswordAuthentication\ no/PasswordAuthentication\ yes/g \
--run-command '/bin/yum -y remove cloud-init' \
--upload ./files/undercloud_interfaces/ifcfg-eth1:/etc/sysconfig/network-scripts/ifcfg-eth1 \
--selinux-relabel

Use virt-install to create the undercloud VM. Note that not all VMs need access to all networks. This one just needs two. The external API (net-br0) will serve two functions. It will allow you to access the undercloud without using your host as a jumpbox – I also use this network to hook back into the virtualization host for IPMI access. More on that further down.

/usr/bin/virt-install \
--disk path=/var/lib/libvirt/images/undercloud.qcow2 \
--import \
--cpu qemu64 \
--vcpus 2 \
--network network=Provisioning \
--network network=net-br0 \
--name undercloud \
--ram 8192 \
--os-type=linux \
--os-variant=rhel7

Below I’m defining 3 controllers using a loop. You don’t really have to hard code MAC addresses here, I do so because I imagine it will make troubleshooting easier one day.

The controllers in a traditional RHOSP environment will run control plane functions for your overcloud, some of which are clustered under pacemaker. Remember when I said this set up would need 3 – 7 VMs? Here you have the option of using 1, 3, or a greater odd number of controllers. Most deployments I see have three, and even in test environments, I recommend this, so you can get a good feel for a production environment.

for i in 1 2 3; do
  qemu-img create -f qcow2 /var/lib/libvirt/images/controller-${i}.qcow2 80G
  /usr/bin/virt-install \
  --disk path=/var/lib/libvirt/images/controller-${i}.qcow2 \
  --network network=Provisioning,mac=52:54:81:00:a0:0${i} \
  --network network=Tenant,mac=52:54:82:00:a0:0${i} \
  --network network=InternalApi,mac=52:54:83:00:a0:0${i} \
  --network network=net-br0,mac=52:54:FF:00:a0:0${i} \
  --network network=Storage,mac=52:54:84:00:a0:0${i} \
  --network network=Storage_Cluster,mac=52:54:85:00:a0:0${i} \
  --name controller-${i} \
  --vcpus 2 \
  --ram 8192 \
  --noautoconsole \
  --os-type=linux \
  --os-variant=rhel7 \
  --dry-run --print-xml > /root/files/vms/controller-${i}.xml
  virsh define --file /root/files/vms/controller-${i}.xml
done

Here I’m creating 3 compute nodes. You could get away with just one, possibly two if you wanted to test migration or other features. Note that I’m using the host CPU configuration with these. You’ll want to pass the virtualization features of your CPU to your guests so you can test running instances later.

for i in 1 2 3; do
  qemu-img create -f qcow2 /var/lib/libvirt/images/compute-${i}.qcow2 120G
  /usr/bin/virt-install \
  --disk path=/var/lib/libvirt/images/compute-${i}.qcow2 \
  --network network=Provisioning,mac=52:54:81:01:a0:0${i} \
  --network network=Tenant,mac=52:54:82:01:a0:0${i} \
  --network network=InternalApi,mac=52:54:83:01:a0:0${i} \
  --network network=Storage,mac=52:54:84:01:a0:0${i} \
  --network network=Storage_Cluster,mac=52:54:85:01:a0:0${i} \
  --name compute-${i} \
  --cpu host,+svm \
  --vcpus 2 \
  --ram 8192 \
  --noautoconsole \
  --os-type=linux \
  --os-variant=rhel7 \
  --dry-run --print-xml > /root/files/vms/compute-${i}.xml
  virsh define --file /root/files/vms/compute-${i}.xml
done

We’re almost there!

I’ve found that with RHOSP 12, pxe_ssh is deprecated, so I needed another way for Ironic to hook into my VMs. I found our how to create a virtual BMC here.

On your virtualization host, you’ll to install the python-virtualbmc package, available via the below repository from the rdoproject.

sudo yum install https://www.rdoproject.org/repos/rdo-release.rpm
sudo yum install -y python-virtualbmc

Add the ports that these BMCs will run on, along with a username and password. The below are examples only, use whatever is needed for your security stance involving your lab.

vbmc add compute-1 --port 6231 --username admin --password password
vbmc add compute-2 --port 6232 --username admin --password password
vbmc add compute-3 --port 6233 --username admin --password password
vbmc add controller-1 --port 6211 --username admin --password password
vbmc add controller-2 --port 6212 --username admin --password password
vbmc add controller-3 --port 6213 --username admin --password password

This will need to be redone anytime the virtualization host is restarted. Forgetting about this will cause unnecessary headaches later. Consider creating a systemd service to handle this.

for i in {1..3}; do
    vbmc start controller-${i}
    vbmc start compute-${i}
done

If you don’t open ports on your host, this setup won’t work! I like to create a firewalld service for custom setups such as these, making it easier to duplicate this setup later, close all ports for a given service at once, and generally easier for management.

cat << EOF > ./files/openstack-ipmi.xml
<?xml version"1.0" encoding="utf-8"?>
<service>
  <short>IPMI control</short>
  <description>IPMI ports for openstack domains.</description>>
  <port protocol="udp" port="6231"/> <!-- compute-1 -->
  <port protocol="udp" port="6232"/> <!-- compute-2 -->
  <port protocol="udp" port="6233"/> <!-- compute-3 -->
  <port protocol="udp" port="6211"/> <!-- controller-1 -->
  <port protocol="udp" port="6212"/> <!-- controller-2 -->
  <port protocol="udp" port="6213"/>> <!-- controller-3 -->
</service>
EOF

firewall-offline-cmd \
--new-service-from-file=./files/openstack-ipmi.xml \
--name=openstack-ipmi
cp /etc/firewalld/services/openstack-ipmi.xml /usr/lib/firewalld/services/

Give a few seconds after restarting firewalld for the service to become available.

systemctl restart firewalld.service

firewall-cmd --add-service openstack-ipmi
firewall-cmd --add-service openstack-ipmi --permanent

Next >

Using Ansible’s dry run with complex workloads

Ansible’s dry run, also known as check mode, can be initiated simply by running a playbook followed by the command switch --check. Using this feature you are able to see what changes will be made in your environment, without making any configuration changes. (This isn’t obvious at first, as the output is indistinguishable from an actual run. Subsequent runs however, will show the same number of changes, as nothing is being done).

For complicated playbooks wherein some tasks rely on variables passed from previous tasks, things start to break down. (No changes being made means those variables are unset, leading to a failed run).

You can work around this in several ways. One way is to to make an assumption about the variables that will be returned, and pass those manually (don’t do this:). For example:

ansible-playbook test.yml \
--extra-vars="{"grep_result": {"rc": "0"}}" --check

There’s two things wrong with this method. First, it will get clumsy real fast if there are a lot of dependencies. Second, if someone accidentally tries this and forgets the --check keyword, they may have potentially run plays with bogus values against your production systems.

A better way would be to have a dry run variable file. It might look like this:

---
grep_result:
  rc: 0

Once this is in place, run your check from a shell script, as this will keep accidents from happening.

#!/bin/bash
ansible-playbook test.yml -e "@dry-run.yml" --check

This allows one to run a check using a complex playbook and reasonable assumptions about about the state of the environment. This is one not-so-perfect solution.

Another imperfect solution would be using the ansible_check_mode boolean variable as shown here. This value is set to true whenever a check is being run, and can allow you to skip some tasks with dependencies. This will prevent the failures, with the obvious drawback of providing incomplete information.

Installing AWX

Just a few weeks ago, on the the 7th of September, Red Hat announced that Tower was opensourced, the upstream project called ‘AWX’. The install relies on either a working Openshift environment, or docker – parting from the monolithic architecture of a simple Tower install. These steps were completed on in a freshly spawned instance of Fedora-Cloud 26. If you’re using a different envionment the steps may vary.

Installing AWX

You’ll need some software to start, the Fedora cloud image is a tad bare.
dnf -y install ansible docker git python libselinux-python

dnf couldn’t install docker-py, so I used pip for this one:
pip install docker-py

Clone the upstream repository.
git clone https://github.com/ansible/awx.git

If docker isn’t running, the install will fail.
systemctl start docker; systemctl enable docker

Swith to the awx/install directory and run the install.yml. There’s no ansible.cfg in this directory to point to the local inventory file, so you’ll need to declare it explicitly when running the play.
cd awx/installer/
ansible-playbook -i inventory install.yml

It will take some time for the install to complete, but once done you should see five images created, and will be able to log into your VM via it’s hostname (if DNS is configured) or IP address.

I found that it was still updating when first attempting to login – but loving the logo so far 🙂

After some time the login screen posted:

The UI looks a tad different than the last version of Tower that I’ve used, but not bad at all!

Interactive OpenStack RC files

In my own environment, I like to password protect my overcloudrc file by removing the line that exports in plain text the OS_PASSWORD value. To do this, you’ll need to first change your password to something more human memorable than the default.

openstack user set admin -⁠-password-prompt

Then change the line that looks something like the following:
export OS_PASSWORD=a268105c2af73c85

To something like this:

read -s -p ‘password: ‘ PASSWD && echo ”
export OS_PASSWORD=$PASSWD

This way, when you source the rc file, you’re asked once for your password, and then it’s immediately exported as an environmental variable. Keep in mind, your password is still available to be viewed in plain text by observing the environmental variables, until you either log out or manually run ‘unset OS_PASSWORD’. With some further modifications, you can have all of the environmental variables unset, along with a visual indication to your authentication status.

A visual authentication clue is provided by default if you install Openstack from scratch. If deployed with director however, the resulting overcloudrc (or stackrc) files do no such thing. This also helps distinguish sessions that are authenticated to the undercloud vs. overcloud – as I’ve seen many folks mired in configuration to realize they’re configuring the wrong environnment!

Your default PS1 variable probably looks somthing like this:

export PS1="[\u@\h \W]\\$ "

I like to add the following to stackrc (uc is for ‘undercloud’):

export PS1="[\u@\h \W](uc)\\$ "

Here’s what I add to overcloudrc:

export PS1="[\u@\h \W](oc)\\$ "

Finally, have all the variables unset by exiting from your session without closing out your current shell.

Start by adding this line at the bottom of the file:

/bin/bash

This opens up a new shell, and stops execution of the script. Below this, you can then unset all of your variables. Remember to also reset your prompt to it’s original state, else the visual indicator that was set up will cause more confusion than clarification.

The result? Source the file, type password and get visual confirmation that you’re authenticated. Type exit, and the shell remains open, with none of the cloud’s authentication information set. As an example, here’s what my stackrc file looks like in my lab:

[stack@undercloud ~](uc)$ cat stackrc
export PS1=”[\u@\h \W](uc)\\$ ”
NOVA_VERSION=1.1
export NOVA_VERSION
read -s -p ‘password: ‘ PASSWD && echo ”
export OS_PASSWORD=$PASSWD
OS_AUTH_URL=http://192.168.99.100:5000/v2.0
export OS_AUTH_URL
OS_USERNAME=admin
OS_TENANT_NAME=admin
COMPUTE_API_VERSION=1.1
OS_BAREMETAL_API_VERSION=1.15
OS_NO_CACHE=True
OS_CLOUDNAME=undercloud
OS_IMAGE_API_VERSION=1
export OS_USERNAME
export OS_TENANT_NAME
export COMPUTE_API_VERSION
export OS_BAREMETAL
export OS_NO_CACHE
export OS_CLOUDNAME
export OS_IMAGE_API_VERSION

/bin/bash

export PS1=”[\u@\h \W]\\$ ”
unset NOVA_VERSION
unset OS_PASSWORD
unset OS_AUTH_URL
unset OS_USERNAME
unset OS_TENANT_NAME
unset COMPUTE_API_VERSION
unset OS_BAREMETAL_API_VERSION
unset OS_NO_CACHE
unset OS_CLOUDNAME
unset OS_IMAGE_API_VERSION

Here’s what it looks like:
[stack@undercloud ~]$ source stackrc
password:
[stack@undercloud ~](uc)$ exit
exit
[stack@undercloud ~]$

Keep in mind, this isn’t real authentication – that happens when you attempt to run commands, and the password is read from the environment. So if you have trouble after sourcing your RC file, you may have entered the wrong password. Simply exit and re-source.

Bringing infrastructure-as-code to the home lab

I’m not new to automating the creation, nor configuaration of my own VMs, but recently I’ve become interested in looking into the best methods by which to combine these two. The idea being to have a “push button” process to recreate any sufficiently documented environment.

I’ve found that too often I need some common service (DNS for example) that I do not have, and that I don’t much care to keep unused VMs arount much longer than needed.

Enter infrastructure as code:

The first step in bootstrapping any given service is to create the VM. I avoid installing an OS by creating the virtual machine from a template QCOW2, like this:

/usr/bin/qemu-img create -f qcow2 -b ${MY-TEMPLATE} \
/var/lib/libvirt/images/dns-server.qcow2

Before Ansible can take over however, I need an IP address and SSH keys injected. There are two tools to choose from here, cloud-init and virt-customize (via the libguestfs-tools-c package); I use both. For me, virt-customize is best at creating customized templates for re-use, but more specific one-off changes I’m injecting with cloud-init.

I start with the default Fedora 25 cloud image, and run the following:

virt-customize \
--format qcow2 \
-a ./Fedora-Cloud-Base-25-1.3.x86_64.qcow2 \
--run-command 'sudo dnf -y install python libselinux-python python2-dnf' \
--selinux-relabel

You may want to copy and rename the default image before running the above, depending on your modus operandi.

Getting metadata together for cloud-init:

You’ll need two files to combine into a metadata configuration iso, meta-data and user-data. This metadata will apply a desired IP address as well an inject SSH keys. Here’s what these files might look like.

meta-data
———
instance-id: 1
local-hostname: dns-server
public-keys:
  - [PASTE SSH PUB KEY 1 HERE]
  - [PASTE SSH PUB KEY 2 HERE]
  - [...]

Strictly speaking, you need just a single public key injected; I like to copy both my user’s and root’s public ssh key into the image. This way if I need to ssh into the VM from either account, I won’t break my workflow.

user-data
———
#cloud-config
write_files:
  - path: /etc/sysconfig/network-scripts/ifcfg-eth0
    content: |
      DEVICE="eth0"
      BOOTPROTO="none"
      ONBOOT="yes"
      TYPE="Ethernet"
      USERCTL="yes"
      IPADDR="192.168.122.101"
      NETMASK="255.255.255.0"
      GATEWAY="192.168.122.1"
      DNS1="192.168.122.1"
runcmd:
  - ifdown eth0; ifup eth0

Combine these files into an ISO file that will be the configuration drive using the following command:

genisoimage -output config.iso -volid cidata -joliet -rock meta-data user-data

This concludes the prep work – now we just tie it in with a script that will do two things.

    1. It should create the VM, and check for network responsiveness.

    2. After recieving input that the VM is on the network, hand the logic over to your Ansible playbook.

Because the terminal will hang waiting on the creation of the VM to finish, seperate the VM creation into it’s own file:

/usr/bin/qemu-img create -f qcow2 -b ${MY-TEMPLATE} /var/lib/libvirt/images/dns-server.qcow2
/usr/bin/virt-install \
--disk path=/var/lib/libvirt/images/dns-server.qcow \
--disk path=./config.iso,device=cdrom \
--network network=default \
--name ${VM_NAME} \
--ram 1024 \
--import \
--os-type=linux \
--os-variant=rhel7

Then call it using a separate runme.sh script:

IMAGE_CREATE=/home/rheslop/offlineKB/Lab/automation/scripts/mkimage
INSTANCE_IP=192.168.122.1
$IMAGE_CREATE &
echo -n "Waiting for instance..."
function INIT_WAIT {
if ping -c 1 $INSTANCE_IP > /dev/null 2>&1 ; then
    sleep 2 && ansible-playbook dns-default-init.yaml
else
    echo -n "."
    INIT_WAIT
fi
}

Assuming your Ansible playbook fully configures the DNS server, it should be safe to delete it, and rerun ‘runme.sh’ to get the same result from scratch every time – with the following caveat: The host will be cached in your users’ ~/.ssh/known_hosts file. If the entry is not deleted, then Ansible fails to authenticate and run your configured playbook.

To sum:
    1. Start by getting a qcow2 image you can work with, customize if necessary.
    2. Create a config.iso file for metadata injection with cloud-init
    3. Create runme.sh bash script to:
        a. Create VM
        b. Launch playbook
    4. Test delete and recreate, document if necessary.

Route to libvirt networks WITHOUT bridging or NAT’ing

It took a few resources to figure this out, so I figured I’d document this in one place for anyone else looking to achieve something similar. The basic goal is to have a method to route out of libvirt virtual networks using the host as a router, and without using NAT, so that VMs can exist in their own network yet still be able to accept traffic initiated from another network. Some modification of the below steps may enable routing between virtual networks as well.

The first steps are basic preparations for normal routing – start by configuring your system to route by editing /etc/sysctl.conf:

net.ipv4.ip_forward=1

Run sysctl -p to make these changes take affect immediately.

Make firewalld aware of the virtual network by adding the libvirt bridge to the trusted zone. Based on experiences from a ‘mrguitar‘, this may be necessary for guests to be able to speak to the host, which will be necessary so that the host can route.1

firewall-cmd --permanent --zone=trusted --add-interface=virbr1
firewall-cmd --reload

You’ll want iptables-like rules inserted so that firewalld will allow routing between the desired interfaces. The first example I found was provdided by a ‘banjo67xxx‘ over at fedoraforum.org,2 (you can find official documentation here.

This amounts to placing an xml file with IPtables rules into firewallds configuration directory.

/etc/firewalld/direct.xml:
<?xml version="1.0" encoding="utf-8"?>
<direct>
  [
    <rule ipv="ipv4" table="filter" chain="FORWARD_direct" priority="0"> -i wlp2s0 -o virbr1 -j ACCEPT </rule>
    <rule ipv="ipv4" table="filter" chain="FORWARD_direct" priority="0"> -i virbr1 -o wlp2s0 -j ACCEPT </rule>
  ]
</direct>

Next, change the forward mode to ‘open’. This is new as of libvirt 2.20, released this last September. The virtual network driver in libvirt uses iptables to describe behaviors for forwarding modes like ‘route’ and ‘nat’.3 This new ‘open’ forwarding mode simply connects guest network traffic to the virtual bridge without any interfering firewall rules.4

Here’s what my network ‘net0‘ looks like. This can be easily replicated by creating an internal network with virt-manager, and using virsh net-edit <network> to add <forward mode=’open’/>.

virsh net-dumpxml net0
<network>
  <name>net0</name>
  <uuid>a5c64edd-e395-4303-91f3-c1c23ed6c401</uuid>
  <forward mode='open'/>
  <bridge name='virbr1' stp='on' delay='0'>
  <mac address='52:54:00:2e:1e:2d'/>
  <domain name='net0'/>
  <ip address='192.168.110.1' netmask='255.255.255.0'>
  </ip>
</network>

Any host on the network that will be connecting to the VM will need to have knowledge of where to send packets destined for your libvirt network. This means that your workstation running the VM will need a static IP. The IP of the laptop my VM is on is 192.168.0.4, so the route on the remote node (stored in /etc/sysconfig/network-scripts/route-<interface>) looks like this:

192.168.110.0/24 via 192.168.0.4

Once this is done VMs on a node, on their own network, can send and receive inbound traffic. This does have limitations. If you cannot insert a return route into your gateway for example, these systems won’t be able to get online. The ability to cleanly route between virtual networks however, opens up all kinds of neat possibilities. 🙂

1. http://mrguitar.net/blog/?p=720
2. http://forums.fedoraforum.org/showthread.php?t=294446
3. https://libvirt.org/firewall.html
4. https://libvirt.org/formatnetwork.html

KVM and Dynamic DNS

OR: Configuring a lab server, part III

Having a PXE install server is a traditional convenience for provisioning baremetal, but if you’re a KVM warrior, cloning is the more sensible option.

Using a hand full of scripts, we can combine cloning, hostname assignment, with dynamic DNS to have a full network configuration for any VM on first boot – effectively cloudifying things a bit. (By cloudify I only mean the process of automating and orchestrating the creation of VMs, treating them less like pets and more like cattle; this is not to say that KVM is by itself a proper cloud platform).

I’m sure there are a few ways to go about all of this, but first, you need your base image. Many vendors provide a qcow2 image for just this purpose, a few links have been curated by openstack.org here.

The image can then be customized using the virt-customize command provided by the libguestfs-tools. I like to have a simple default login, that will be allowed via ssh; I’ll also remove cloud-init, as I’m not using it for this setup:

virt-customize -a \
/var/lib/libvirt/images/templates/CentOS-7-x86_64-template.qcow2 \
--root-password password:redhat \
--edit /etc/ssh/sshd_config:s/PasswokrdAuthentication\ no/PasswordAuthentication\ yes/g \
--run-command '/bin/yum -y remove cloud-init'

Once this image is modified, another image can be cloned from it using the -b option of qemu-img, as shown:
qemu-img create -f qcow2 -b base_image.qcow2 image_you_are_creating.qcow

This assures that the only disk space used by the newly created disk are the differences between it and the base. This part of course will be scripted, as well as the creation of the VM. By using another virt-customize command, the hostname of the VM will match the VM’s name – specified by the first argument provided to the script, nice!

This is what I normally use, but experiment and see what works for you.

#!/bin/bash

if [ -z $1 ]; then
echo "Please provide the VMs name. Example:"
echo "$0 cent-1"
exit
fi

VM_NAME=$1
FILENAME=${VM_NAME}.qcow2
DOMAIN=workstation.local

# Use template stored in /var/lib/libvirt/images/templates/
TEMPLATE=/var/lib/libvirt/images/templates/CentOS-7-x86_64-template.qcow2

# Create image in /var/lib/libvirt/images/
IMAGES_PATH=/var/lib/libvirt/images

if [ -e $IMAGES_PATH/$FILENAME ]; then
echo "$FILENAME already exists in $IMAGES_PATH"
echo "Please choose another name"
exit
fi

/usr/bin/qemu-img create -f qcow2 -b $TEMPLATE $IMAGES_PATH/$FILENAME

# Set hostname
virt-customize -a $IMAGES_PATH/$FILENAME --hostname $1.$DOMAIN

# Create the virtual machine
/usr/bin/virt-install \
--disk path=/var/lib/libvirt/images/$FILENAME \
--network network=net1 \
--name $VM_NAME \
--ram 1024 \
--import \
--os-type=linux \
--os-variant=rhel7

With this, we have just about all qualifications mentioned above, except for DNS. For this I’ll set up a dynamic DNS server, so once an instance boots – it will insert it’s name into the record when it’s getting its address.

For this I developed a bash/ansible solution that will deploy such a server with minimal input. Before setting this up, I hadn’t used Dynamic DNS, so I based my installation steps on this instructional video posted by Lewis from SSN.

Here’s how it works – I’ve created a new network, it’s an isoloated network for my lab. I’ll edit that script above (called mkcent) and change this:

--network network=net1 \

to this:

--network network=default \
--network network=net1 \

That way the VM will have its primary NIC on the outside, and one NIC on the inside – it’ll route for the other VMs. Then just call the script to spin it up.

mkcent router-net1

Next clone the following project:

git clone https://github.com/rheslop/dynamic-dns.git

Navigating to the root directory, I’ll create a hosts file for ansible and populate it with the IP address the VM has recieved from the default network. (You can get this via virsh net-dhcp-leases defaut)

echo "router-net1 ansible_host=192.168.124.62" > hosts

Make sure your ssh key is copied over for Ansible communications:

ssh-copy-id root@192.168.124.62

Finally, configure the Dynamic DNS server by running the Ansible playbook from script. (Ansible will need to be installed on your system for this to work).

deploy-ddns

Remember to remove the aforementioned extra network line from the mkcent script, so additional VMs only boot with net1. Because ‘router-net1’ also routes, the VMs will have internet access, even while on an internal network.

Configuring a lab server (part II)

Continuing where we left off, the next services I’ll want to configure will be routing and DNS. It’ll be important going forward that the server have the correct FQDN, as well NICs assigned to the appropriate zones.

I’m going to set the ens4 interface to point to itself as the DNS server. DNS forwarding will be configured, so that the lab server will be able to address both lab servers and the outside network. Once more for reference, the ens4 interface in this setup faces our lab network, and eth0 connects to the greater external network.

nodes

Configure routing:

Our ens4 interface will remain in the firewalld zone ‘public’, and eth0 will be moved to ‘external’. Because the external zone has masquerading enabled by default, the only thing left will be enabling the kernel’s routing capabilities.

hostnamectl set-hostname pxe-boot.testlab.local
echo "ZONE=external" >> /etc/sysconfig/network-scripts/ifcfg-eth0
cat >> /etc/sysconfig/network-scripts/ifcfg-ens4 << EOF
DNS1=192.168.112.254
ZONE=public
EOF
echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
sysctl -p

Install bind:

yum -y install bind

Configure DNS:

Because name services can be a bit involved, I’ve highlighted some key fields you may want to customize for your own setup.

cat > /etc/named.conf << EOF
options {
listen-on port 53 { 127.0.0.1; 192.168.112.254; };
listen-on-v6 port 53 { ::1; };
directory "/var/named";
dump-file "/var/named/data/cache_dump.db";
statistics-file "/var/named/data/named_stats.txt";
memstatistics-file "/var/named/data/named_mem_stats.txt";
allow-query { localhost; 192.168.112.0/24; };
recursion yes;
    forwarders {
        8.8.8.8;
        8.8.4.4;
    };

dnssec-enable yes;
dnssec-validation yes;
dnssec-lookaside auto;

/* Path to ISC DLV key */
bindkeys-file "/etc/named.iscdlv.key";
};

logging {
channel default_debug {
file "data/named.run";
severity dynamic;
    };
};

zone "." IN {
type hint;
file "named.ca";
};

include "/etc/named.rfc1912.zones";

zone "testlab.local" IN {
type master;
file "/var/named/testlab.local.zone";
};

zone "112.168.192.in-addr.arpa" IN {
type master;
file "/var/named/zone.112.168.192.in-addr.arpa";
};
EOF

The station10 host entered in the configuration below is done only to test name resolution – currently this host does not exist on my network so I will expect pings to fail, but they should correctly resolve to 192.168.112.10.

cat > /var/named/testlab.local.zone << EOF
\$ORIGIN testlab.local.
\$TTL 86400
testlab.local. IN SOA pxe-boot.testlab.local. root.pxe-boot.testlab.local. (
    20120710 ; serial
    21600 ;refresh after 6 hours
    3600 ; retry after 1 hour
    604800 ; expire after 1 week
    86400 ) ; minimum TTL of 1 day

;DNS Server
testlab.local. IN NS pxe-boot.testlab.local.

;Clients
station10.testlab.local. IN A 192.168.112.10
pxe-boot.testlab.local. IN A 192.168.112.254
EOF

cat > /var/named/zone.112.168.192.in-addr.arpa << EOF
\$ORIGIN 112.168.192.in-addr.arpa.
\$TTL 86400
112.168.192.in-addr.arpa. IN SOA pxe-boot.testlab.local. root.pxe-boot.testlab.local. (
    20120710 ; serial
    21600 ; refresh after 6 hours
    3600 ; retry after 1 hour
    604800 ; expire after 1 week
    86400 ) ; minimum TTL of 1 day

;DNS Server
112.168.192.in-addr.arpa. IN NS pxe-boot.testlab.local.

;Clients
10 IN PTR station10.testlab.local.
254 IN PTR pxe-boot.testlab.local.
EOF

It will be important that permissions are set correctly on the following files:
chown root:named /var/named/zone.112.168.192.in-addr.arpa
chown root:named /var/named/testlab.local.zone

We can now restart NetworkManager, as well as the network interfaces. This way the zone and other configuration changes come into affect.

systemctl restart NetworkManager
ifdown eth0 && ifup eth0
ifdown ens4 && ifup ens4

Finally, let’s add DNS to our firewall rules and start named.

firewall-cmd --permanent --zone public --add-service="dns"
firewall-cmd --reload

systemctl enable named
systemctl start named

Go to: Part I

Configuring a lab server (part I)

Or: PXE install services

The example installation will be done on a KVM hosted virtual machine, and will mimic a setup wherin a single server will act as a gateway, kickstart server, as well as providing any other lab dependent services.

nodes

The below will focus exclusively on the configuration of kickstart and DHCP – further configuration will be done in another post.

Starting with a vanilla installation, start by configuring the network. Since DHCP and other default settings are fine for the public port, we’ll just set ‘ONBOOT’ to yes on that one.

sed -i 's/ONBOOT=no/ONBOOT=yes/' \
/etc/sysconfig/network-scripts/ifcfg-eth0
systemctl restart NetworkManager
ifdown eth0 && ifup eth0


cat > /etc/sysconfig/network-scripts/ifcfg-ens4 << EOF
DEVICE="ens4"
BOOTPROTO="none"
HWADDR="$(ip addr show ens4 | awk '/ether/ {print $2}')"
IPV6INIT="no"
MTU="1500"
NM_CONTROLLED="no"
ONBOOT="yes"
TYPE="Ethernet"
IPADDR="192.168.112.254"
NETMASK="255.255.255.0"
DOMAIN="testlab.local"
EOF

systemctl restart NetworkManager; ifdown ens4

Install services:
yum -y install vsftpd dhcp tftp tftp-server* xinetd* syslinux

Next you’ll need a copy of the CentOS (or RedHat) install media to copy over to the FTP server. Create /var/ftp/inst and a /var/ftp/pub directories, which will hold installation packages and kickstart files respectively. I would recommend additional folders inside /var/ftp/inst for each operating system that will be offered by the server.

mkdir -p /var/ftp/{pub,inst/centos7}
mount -t iso9660 /dev/cdrom /mnt
cp -var /mnt/. /var/ftp/inst/centos7/
chcon -R -t public_content_t /var/ftp/inst/

Kickstart file:

cat > /var/ftp/pub/centos-7_ks.cfg << EOF
#version=DEVEL
install
url --url ftp://192.168.112.254/inst/centos7
lang en_US.UTF-8
keyboard us
network --onboot yes --device eth0 --bootproto dhcp
rootpw --plaintext redhat
firewall --service=ssh
authconfig --enableshadow --passalgo=sha512
selinux --enforcing
timezone US/Central --isUtc --nontp
bootloader --append=" crashkernel=auto" --location=mbr --boot-drive=vda
zerombr
clearpart --all --drives=vda
autopart --type=lvm
%packages --nobase
@core
%end
EOF

DHCP configuration:

cat > /etc/dhcp/dhcpd.conf << EOF
subnet 192.168.112.0 netmask 255.255.255.0 {
range 192.168.112.100 192.168.112.200 ;
option routers 192.168.112.254;
option domain-name "testlab.local";
option domain-name-servers 192.168.112.254;
default-lease-time 86400;
max-lease-time 129600;
allow booting;
allow bootp;
class "pxeclients" {
    match if substring(option vendor-class-identifier, 0,9) = "PXEClient";
    next-server 192.168.112.254;
    filename "pxelinux.0";
    }
}
EOF

TFTP server configuration:

mkdir -p /var/lib/tftpboot/centos7
mkdir -p /var/lib/tftpboot/pxelinux.cfg
cp /usr/share/syslinux/{pxelinux.0,menu.c32} /var/lib/tftpboot/
cp /var/ftp/inst/centos7/images/pxeboot/* /var/lib/tftpboot/centos7

sed -i 's/no/yes/' /etc/xinetd.d/tftp
cat > /var/lib/tftpboot/pxelinux.cfg/default << EOF
default menu.c32
prompt 0
timeout 300
ONTIMEOUT local
LABEL local
    MENU LABEL Boot to hard drive
    LOCALBOOT 0
LABEL centos7
    MENU LABEL CentOS 7.2 x64
    kernel centos7/vmlinuz
    append initrd=centos7/initrd.img noipv6 ks=ftp://192.168.112.254/pub/centos-7_ks.cfg
EOF

Finally, open the firewall and launch services.

firewall-cmd --permanent --add-service="ftp"
firewall-cmd --permanent --add-service="tftp"
firewall-cmd --permanent --add-service="dhcp"
firewall-cmd --reload
ifup ens4
systemctl enable vsftpd
systemctl enable dhcpd
systemctl enable xinetd
systemctl enable tftp
systemctl start vsftpd
systemctl start dhcpd
systemctl start xinetd
systemctl start tftp

The only step left is to test the set up by network booting a system on the 192.168.112.0/24 network.

pxe_booting

🙂

Go to: Part II