Openvswitch / KVM / Libvirt / Ubuntu / VLANs the right way

July 7, 2014    networking openvswitch ubuntu vlan

There are a lot of old blog posts out there to getting KVM guests to use different vlans via openvswitch. There are a lot that tell you to create fake bridges or create the ports via ovs-vsctrl and add tell libvirt to use that created interface or portgroup. Then there are almost no blogs that really say, when you setup openvswitch, this is how you make the interface settings stick. The correct way to do it is this basic flow

  • Create a bridge via ovs-vsctrl
  • Add your working interface to the bridge via ovs-vsctrl
  • Set your ip info on the new bridge
  • Create a libvirt network 5) Select the port group you want to use from your new network on the guest xml via libvirt

When the guest starts if the interface for the vlan isn’t created it will auto create it in openvswitch for you. So this works with Ubuntu 14.04 This also assumes bonding is setup via LACP on the host. It works the same if you just have a single interface like eth0. Just remove all the bond options. So my starting ifconfig for my bond0 device looks something like

bond0     Link encap:Ethernet  HWaddr 00:25:90:ed:dc:f0
          inet addr:10.128.7.121  Bcast:10.128.7.255  Mask:255.255.255.0
          inet6 addr: fe80::f1:41ff:fe72:a331/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:713943 errors:0 dropped:0 overruns:0 frame:0
          TX packets:390750 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:49037015 (49.0 MB)  TX bytes:674651803 (674.6 MB)

So the first thing we want to do is install openvswitch-switch apt-get install openvswitch-switch Now we need to create a bridge in openvswitch ovs-vsctrl add-br br0 Now we need to add our working interface to the bridge. THIS WILL CAUSE YOUR CONNECTION TO DROP. Do not run this command if you don’t have remote KVM access or on the console. ovs-vsctrl add-port br0 bond0 Now that we have a bridge setup we need to give it IP information

ifconfig bond0 0
ifconfig br0 10.128.7.121 netmask 255.255.255.0
route add default gw 10.128.7.1

So now your bridge interface is up and it uses bond0 still. We gave it the same IP information. Now lets setup your the following file so the system reboots correctly

# The loopback network interface
auto lo
iface lo inet loopback

auto p1p1
iface p1p1 inet manual
  bond-master bond0

auto p1p2
iface p1p2 inet manual
  bond-master bond0

auto bond0
allow-br0 bond0
iface bond0 inet manual
  bond-mode 4
  bond-miimon 100
  bond-lcap-rate 1
  xmit_hash_policy layer3+4
  bond-slaves none
  ovs_bridge br0
  ovs_type OVSPort
  pre-up ifconfig $IFACE up
  post-down ifconfig $IFACE down
  address 0.0.0.0

auto br0
allow-ovs br0
iface br0 inet static
  address 10.128.7.121
  netmask 255.255.255.0
  gateway 10.128.7.1
  dns-nameservers 10.128.7.4 10.128.7.5
  ovs_type OVSBridge
  ovs_ports br0
  pre-up ifconfig $IFACE up
  post-down ifconfig $IFACE down

The big things to add/change are as follows

  • allow-br0 bond0  This tells ovs to use bond0
  • Make sure your bond0 interface is set to manual
  • Also add the pre-up/post-up lines and address line to make sure the interface comes up ok
  • ovs_bridge br0 tells the system bond0 is apart of the ovs bridge br0
  • ovs_type OVSPort tells the system that this is a port in ovs
  • allow-ovs br0 tells the system this is for ovs
  • ovs_type OVSBridge tells the system this is a bridge
  • ovs_ports br0

Now that’s all set you can run reboot and the bridge should come up just fine Now lets create a network. Here is my sample network file. It creates a network with an un-tagged port and 2 other ports that get tagged with vlans 2 and 3

<network>
 <name>vlans</name>
 <uuid>4216c8df-349d-4a32-a6ae-533135a9d682</uuid>
 <forward mode='bridge'/>
 <bridge name='br0'/>
 <virtualport type='openvswitch'/>
 <portgroup name='vlan-01' default='yes'>
 </portgroup>
 <portgroup name='vlan-02'>
   <vlan>
     <tag id='2'/>
   </vlan>
 </portgroup>
 <portgroup name='vlan-03'>
   <vlan>
     <tag id='3'/>
   </vlan>
 </portgroup>
</network>

So you’ll want to change the name of the network group and also the vlan info. My first vlan is un-tagged. and the next two are tagged. So create a file called vlans.xml and put that in now we can load it in libvirt

virsh net-define ./vlans.xml
virsh net-start vlans
virsh net-autostart vlans

Once that is all setup you can define an interface like

<interface type='network'>
 <source network='vlans' portgroup='vlan-02'/>
</interface>

So my example if I show my running set looks like

root@vmnode2:~# ovs-vsctl show
19655270-bcee-4b57-b2d5-5a180da422a8
    Bridge "br0"
        Port "vnet1"
            tag: 3
            Interface "vnet1"
        Port "br0"
            Interface "br0"
                type: internal
        Port "bond0"
            Interface "bond0"
        Port "vnet0"
            tag: 2
            Interface "vnet0"
    ovs_version: "2.0.1"

This way we don’t have to tell the guests to tag their traffic going out and we just have openvswitch tag the traffic. One gotcha might be your hardware switch has to know about the vlan ids even if you trunk the port the KVM host is connected to. In cisco that is like

vlan 2
name WebVlan
exit

Simple as that.



comments powered by Disqus