Build a multi Node Openstack Environment with Openvswitch

Since I have some time to write again I would like to share a small HowTo, how to Install a multi Node Openstack install with Openvswitch (ovs) and provider Networks as vlans for mixed Environments. At the end I will also add a subsection how to add a second vlan and a script how to add tenants, users and internal Networks for the tenants.

The whole Setup is made with Devstack, some bash-scripts. What you should look at, is for sure multi homing (iproute2) and a general documentation about
openvswitch and HowTo operate Openstack with provider networks. Over all this Documentation should be doable also without this (hopefully) and hopefully I have no typos 😉

Lets assume we have 3 networks / vlans
1. Network is the Management Network (for the communication between the nodes and for the webinterface -> Horizon)
The network is: 192.168.29.0/24 lets say vlan 29 this must be tagged (access port) on the switch / bladecenter –> as vlan 29 HINT: this is the „Segmentation ID“ the gateway is at the end: 192.168.29.254

2. This network is the provider network where the instances (virtual machines) are getting their public IP.
This Network is: 192.168.28.0/24 lets say vlan 28 IMPORTANT: This network should NOT be tagged so this should be on a trunk port the gateway is at the end: 192.168.28.254

3. This network is a second provider network, just to show the idea how to extend an openstack installation.
Like above: This Network is: 192.168.27.0/24 lets say vlan 27 IMPORTANT: This network should NOT be tagged so this should be on a trunk port the gateway is at the end: 192.168.27.254

We have minimum 2 nodes up to X nodes … the first is the controller (+compute) node and all others are going to be compute nodes.

Following steps have to be done on ALL nodes (I will say, when we have specific things to do) :

First you have to Install Ubuntu with openssh-server and with working internet in vlan 29 on the first interface (the other interface are configured later, don’t worry).

Login and do a:

sudo su

Do all basic stuff in a bunch update server, install basic packages, add the initial stack user under which we are doing all following.

apt-get update -y && apt-get install -y git sudo htop iotop libxslt1-dev libxml2-dev openvswitch-common openvswitch-switch openvswitch-controller && groupadd stack && useradd -g stack -s /bin/bash -d /opt/stack -m stack && echo "stack ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers && passwd stack

Deactivate the firewall:

service iptables save
ufw disable

Lets do some naming of the nodes

vim /etc/hosts

Add all your nodes here … maybe also future nodes:

192.168.29.1      openstack01 openstack01.local.net
192.168.29.2      openstack02 openstack02.local.net
192.168.29.3      openstack03 openstack03.local.net
192.168.29.4      openstack04 openstack04.local.net
192.168.29.5      openstack05 openstack05.local.net
192.168.29.6      openstack06 openstack06.local.net

Login as user stack, which we created before and generate an ssh key (copy the pubkey afterwards):

mkdir ~/.ssh; chmod 700 ~/.ssh

ssh-keygen -t rsa 

cat ~/.ssh/id_rsa.pub

On the >>Compute<< nodes insert the copied key from above

echo "ssh-rsa YOUR_KEY_FROM_CONTROLLER_NODE_HERE stack@openstack01" > ~/.ssh/authorized_keys

Start checking out the devstack repository:

git clone https://git.openstack.org/openstack-dev/devstack

Change the Folder

cd devstack

I assume we have interface em1 or eth0 with the VLAN 29 configured, we have then em2 or eth1 with VLAN 28 (not yet configured) and VLAN 27 on interface em3 or eth2

Lets configure the network interfaces:

sudo vim /etc/network/interfaces 

Add a section at the end:

auto em2
iface em2 inet manual
        up ifconfig $IFACE 0.0.0.0 up
        down ifconfig $IFACE 0.0.0.0 down

Bring the interface up:

sudo ifup em3

Now we are creating a bridge in Openvswitch and adding the interface em2 with vlan tag 28 to it.

sudo ovs-vsctl add-br br-ex
sudo ovs-vsctl add-port br-ex em3 tag=28
sudo ovs-vsctl show

Since it is to boring to all the time do a sudo bla …
Now we are building up a second routing table so have for each network the correct routing.

sudo su
cd /etc/iproute2
cp rt_tables rt_tables.backup
ip rule show
echo "1 openstack28" >> /etc/iproute2/rt_tables
ifconfig br-ex 192.168.28.1 netmask 255.255.255.0
ip route add 192.168.28.0/24 dev br-ex src 192.168.28.1 table openstack28
ip route add default via 192.168.28.254 dev br-ex table openstack28
ip rule show
ip rule add from 192.168.28.0/24 table openstack28
ip rule add to 192.168.28.0/24 table openstack28
ip route flush cache
ip rule show

Logout as user root, so that you are again user stack.

Prepare the config for devstack on the >>CONTROLLER<< Node

vim local.conf
[[local|localrc]]

HOST_IP=192.168.29.1
SERVICE_HOST=192.168.29.1
MYSQL_HOST=192.168.29.1
SERVICE_HOST=192.168.29.1
MYSQL_HOST=192.168.29.1
RABBIT_HOST=192.168.29.1
GLANCE_HOSTPORT=192.168.29.1:9292
PUBLIC_INTERFACE=em2

ADMIN_PASSWORD=secure
MYSQL_PASSWORD=secure
RABBIT_PASSWORD=secure
SERVICE_PASSWORD=secure
SERVICE_TOKEN=secure

## Neutron options
Q_USE_SECGROUP=True
ENABLE_TENANT_VLANS=True
TENANT_VLAN_RANGE=27:28
PHYSICAL_NETWORK=default
OVS_PHYSICAL_BRIDGE=br-ex

Q_USE_PROVIDER_NETWORKING=True
Q_L3_ENABLED=False

# Do not use Nova-Network
disable_service n-net

# IMAGES directly downloaded :)
IMAGE_URLS+=",http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2"
IMAGE_URLS+=",http://cloud.centos.org/centos/6/images/CentOS-6-x86_64-GenericCloud.qcow2"

# Neutron
ENABLED_SERVICES+=,q-svc,q-dhcp,q-meta,q-agt

## Neutron Networking options used to create Neutron Subnets
IP_VERSION=4
FIXED_RANGE="192.168.28.0/24"
NETWORK_GATEWAY=192.168.28.254
PROVIDER_SUBNET_NAME="provider_net"
PROVIDER_NETWORK_TYPE="vlan"
SEGMENTATION_ID=28

Now we can start building the first node:

./stack.sh

Now we free up first 20 IPs for compute nodes and add dns for the subnets, but first source admin keystone credentials

. openrc admin admin
neutron subnet-update provider_net --allocation-pools type=dict list=true start=192.168.28.20,end=192.168.28.253
neutron subnet-update provider_net --dns-nameserver 8.8.8.6 --dns-nameserver 8.8.8.8

End for the >>CONTROLLER<< Node Thats it… for the first Node, you can already try to fire up an instance. The second node is just compute, so lets go. Repeat all general steps, above and change ip address for routing e.g. 192.168.28.1 -> 192.168.28.2
>>HINT<< don’t forget to add ICMP allowed to your Security group ! otherwise you can not ping the instance !

I assume you are now in /home/stack/devstack as user stack on the first compute node:

vim local.conf
[[local|localrc]]
HOST_IP=192.168.29.2
SERVICE_HOST=192.168.29.1
MYSQL_HOST=192.168.29.1
SERVICE_HOST=192.168.29.1
MYSQL_HOST=192.168.29.1
RABBIT_HOST=192.168.29.1
GLANCE_HOSTPORT=192.168.29.1:9292
ADMIN_PASSWORD=secure
MYSQL_PASSWORD=secure
RABBIT_PASSWORD=secure
SERVICE_PASSWORD=secure
SERVICE_TOKEN=secure

# Services that a compute node runs
ENABLED_SERVICES=n-cpu,rabbit,q-agt

## Neutron options
PHYSICAL_NETWORK=default
OVS_PHYSICAL_BRIDGE=br-ex
PUBLIC_INTERFACE=em2
Q_USE_PROVIDER_NETWORKING=True
Q_L3_ENABLED=False

Thats it here … this is just a dumb compute node 🙂

./stack

Continue adding your nodes….
Now lets say, you want to have a second vlan as provider network, you have em3 or eth2 connected to a trunk interface.
What we need is a second bridge with the tag of vlan 27, and we have to add this as well to the ml2 config of openstack, create a new routing table and reload openstack afterwards (we could have done it while building openstack but it’s a good learning …

On each node we have to add the interface:
(as root /sudo)

vim /etc/network/interfaces

add this:

auto em3
iface em3 inet manual
        up ifconfig $IFACE 0.0.0.0 up
        down ifconfig $IFACE 0.0.0.0 down

ifup em3

Add a routing table

cd /etc/iproute2
cp rt_tables rt_tables_1vlan # backup
ip rule show
echo "2 openstack27" >> /etc/iproute2/rt_tables

Create a bridge (lets name it with the vlan tag)

ovs-vsctl add-br br-ex27
ovs-vsctl add-port br-ex27 em3 tag=27

Now we have to add this to the ml2 plugin config:

vim /etc/neutron/plugins/ml2/ml2_conf.ini

Change the following lines in the sections, to add the „physical“ Networks and mapp them to the bridges.

[ml2_type_flat]
flat_networks = default,vlan27

[ml2_type_vlan]
network_vlan_ranges = default:28:28,vlan27:27:27

[ovs]
bridge_mappings = default:br-ex,vlan27:br-ex27

Now we add the routing for this network in the added routing table (Change IP addresses to your need on each node)

ifconfig br-ex27 192.168.27.1 netmask 255.255.255.0
ip route add 192.168.27.0/24 dev br-ex27 src 192.168.27.1 table openstack27
ip route add default via 192.168.27.254 dev br-ex27 table openstack27
ip rule show
ip rule add from 192.168.27.1/24 table openstack27
ip rule add to 192.168.27.1/24 table openstack27
ip route flush cache
ip rule show

Now you should restart the Devstack on each node ( ./rejoin-stack.sh CTRL+C / CTRL+D until all screen sessions are closed, afterwards just do it once more ./rejoin-stack.sh )

Now we create the network in openstack:

. openrc admin admin 
neutron net-create vlan27 --provider:network_type=vlan --provider:segmentation_id=27 --provider:physical_network=vlan27 --shared 
neutron subnet-create vlan27 192.168.27.0/24 --name provider_net27 --dns-nameserver 192.168.10.16 --dns-nameserver 10.49.2.2 --dns-nameserver 8.8.8.8 --gateway 192.168.27.254 --allocation-pool start=192.168.27.1,end=192.168.27.253

neutron subnet-update provider_net27 --allocation-pools type=dict list=true start=192.168.27.20,end=192.168.27.253

Thats it…. wasn’t so hard right ?

If you have to restart the nodes … we should build a script to recreate the routing:

#!/bin/bash
# created by torbatux on 20151109
# this script brings up vlans on openvswitch bridges (which are mapped 
# in Openstack against "physical networks"
# and adds routes in separate routing tables

HOSTIP=`hostname | grep -o [0-9][0-9] | sed s'/^0//'`
declare -A vlans
vlans=( ["28"]="br-ex,192.168.28.,openstack" ["27"]="br-ex27,192.168.27.,openstack27" )

for vlan in "${!vlans[@]}";
do
  #echo "$vlan - ${vlans["$vlan"]}"
  bridge=`echo ${vlans["$vlan"]} | cut -d"," -f1`
  bareip=`echo ${vlans["$vlan"]} | cut -d"," -f2`
  bridgeip=`echo "$bareip""$HOSTIP"`
  gateip=`echo "$bareip"254`
  table=`echo ${vlans["$vlan"]} | cut -d"," -f3`
  echo create currently:
  echo Vlan tag = $vlan 
  echo Bridge = $bridge 
  echo Bridge IP = $bridgeip 
  echo Gateway = $gateip
  echo Routingtable = $table
  echo bring up bridge
  sudo ifconfig $bridge $bridgeip netmask 255.255.255.0
  echo add routing
  sudo ip route add $bridgeip/24 dev $bridge src $bridgeip table $table
  sudo ip route add default via $gateip dev $bridge table $table
  sudo ip rule add from "$bridgeip"/24 table $table
  sudo ip rule add to "$bridgeip"/24 table $table
  sudo ip route flush cache
  sudo ip rule show
done

cd ~/devstack
./rejoin-stack.sh

Now you just have to go out the screen session with CTRL+A+D

Lets Build a script to automatically fill openstack with users / credentials /
tenants and tenant-networks so that we have a ready to test Openstack.

Login on your controller node as user stack (you can put your pubkey there as well, than it gets added as well, just change mypersonalcloud.key.pub to your key name:

cd devstack
vim create_initial_setup.sh
#!/bin/bash
# created by torbatux on 20151110
# this script fills a fresh devstack install with admin users,
# test-tenants, tenant-networks and flavors  

# Get admin creds
. openrc admin admin

# List existing projects
openstack project list

# List existing users
openstack user list

# Add users and projects
# ANAME is Admin users on Tenant
ANAME=( "user1" "user2" "user3" "user4" "user5" "user6" )
PASSWORD=secure
PROJECTS=( "main" "project1" "project2" "project3" "project4" )

# flavors are buildes by the numbers behind the dot .1_2_10 means 1 CPU, 2 GB RAM, 10 GB HDD
FLAVORS=( "main.1_1_10" "main.1_1_20"  "main.1_2_20" "main.1_4_20" "main.2_2_20" "main.2_4_20" "main.2_8_20" "main.2_12_20" "main.2_8_40" "main.4_4_50" "main.4_6_20" "main.4_6_50" "main.4_8_100" "main.4_16_200" )
SUBNETCOUNT=1

for NAME in "${ANAME[@]}"
do
  openstack user create $NAME --password=$PASSWORD
done

for PROJECT in "${PROJECTS[@]}"
do
  openstack project create $PROJECT
  openstack role add admin --user admin --project $PROJECT
  openstack --os-project-name $PROJECT security group create --description "all is allowed" "$PROJECT"-any-to-any
  openstack --os-project-name $PROJECT security group rule create --proto icmp "$PROJECT"-any-to-any
  openstack --os-project-name $PROJECT security group rule create --proto udp --dst-port 1:65535 "$PROJECT"-any-to-any
  openstack --os-project-name $PROJECT security group rule create --proto tcp --dst-port 1:65535 "$PROJECT"-any-to-any
  openstack --os-project-name $PROJECT network create "$PROJECT"-intnet
  neutron subnet-create --name "$PROJECT"-subnet "$PROJECT"-intnet 192.168."$SUBNETCOUNT".0/24
  TENANTID=`openstack project list | grep $PROJECT | cut -d"|" -f 2 | tr -d " "`
  neutron router-create --tenant-id $TENANTID "$PROJECT"-intnet-router
  neutron router-interface-add "$PROJECT"-intnet-router "$PROJECT"-subnet
  let SUBNETCOUNT=SUBNETCOUNT+1
  for NAME in "${ANAME[@]}"
  do
    # grant admin user rights on the tenant
    openstack role add admin --user $NAME --project $PROJECT
  done
done

for FLAVOR in "${FLAVORS[@]}"
do
  RAM=`echo $FLAVOR | cut -d"_" -f2`
  RAM=`echo "$RAM * 1024" | bc`
  CPU=`echo $FLAVOR | cut -d"_" -f1 | cut -d"." -f2`
  DISK=`echo $FLAVOR | cut -d"_" -f3`
  openstack --os-project-name main flavor create --ram $RAM --disk $DISK --vcpus $CPU $FLAVOR
done

# add the personal key
openstack keypair create --public-key mypersonalcloud.key.pub my_personal_cloud_key

# for tests in the admin tenant, create a sec group there as well
openstack --os-project-name admin security group create --description "all is allowed" admin-any-to-any
openstack --os-project-name admin security group rule create --proto icmp admin-any-to-any
openstack --os-project-name admin security group rule create --proto udp --dst-port 1:65535 admin-any-to-any
openstack --os-project-name admin security group rule create --proto tcp --dst-port 1:65535 admin-any-to-any
chmod 775 create_initial_setup.sh

Now you can run all the time after a fresh install: ./create_initial_setup.sh

Over all I hope, this small HowTo helped you a little bit. Enjoy Openstack 😉

Schreibe einen Kommentar

Diese Website verwendet Akismet, um Spam zu reduzieren. Erfahre mehr darüber, wie deine Kommentardaten verarbeitet werden.