Dynamic haproxy config in SaltStack

If you are anything like me, you like to over utilize haproxy. Salt stack has a nice feature for files that allow you to add to them in different state files.

Say you want to have a stock haproxy config but sick and tired of having to maintain 5 different versions of a haproxy config.

So our basic formula layout looks like this

– packages/haproxy.sls
– web/init.sls

The haproxy.sls stores our basic haproxy install, template and makes sure the service is running

The web/init.sls is what we might apply to our app servers that want access to mysql and kafka.

So pretty much you have have as many of the file.accumulated states as you wish as long as they all have the same name unless you want to have multiple loops in your template. As long as you include the packages/haproxy.sls file in any other state file this will work.

Monitor Salt with Monit

Sometimes salt has the tendency to crash. So we can use monit to fix that problem

This assumes you already have the EPEL repo installed

yum install monit

Now with monit installed we can edit the following config


With the following contents

set daemon 5
  with start delay 5
set logfile /var/log/monit.log
set idfile /var/lib/monit.id
set statefile /var/run/monit.state
set mailserver
  localhost port 25
  with timeout 30 seconds
set mail-format {
      from: monit@hostname.domain.com
   subject: $SERVICE $EVENT at $DATE
   message:     Monit $ACTION $SERVICE at $DATE on $HOST: $DESCRIPTION.
    Yours sincerely,

set alert techops@manage.com
  only on { nonexist, timeout, resource, icmp, connection}
set eventqueue
  basedir /var/tmp
  slots 100
set httpd port 2812 and
  use address
  allow localhost
include /etc/monit.d/*.conf

Make sure to change the use address so it won’t mind to all ips or just se the allow subnet/mask to a valid one.

Then edit the following file


It will look like this

check process salt
  with pidfile /var/run/salt-minion.pid
  group system
  start "/etc/init.d/salt-minion start"
  stop "/etc/init.d/salt-minion stop"
  if 3 restarts within 5 cycles
    then unmonitor

That should check if salt is up and running.

You can use this process for almost any process that sets a local pid to check.

guestfish problems with virt-filesystems

I was trying to use guestfish to increase a qcow2 partition without booting live and fdisking and all that mess. So I tried to run it and was getting

# virt-filesystems --long --parts --blkdevs -h -a disk.qcow2
libguestfs: error: /usr/bin/supermin-helper exited with error status 1.
To see full error messages you may need to enable debugging.
See http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs

Scratching my head a bit and figured out you need to update the guestfs appliance packages


Now the command works!

# virt-filesystems --long --parts --blkdevs -h -a disk.qcow2
Name       Type       MBR  Size  Parent
/dev/sda1  partition  83   3.8G  /dev/sda
/dev/sda2  partition  82   976M  /dev/sda
/dev/sda   device     -    5.0G  -

Openvswitch / KVM / Libvirt / Ubuntu / VLANs the right way

There are a lot of old blog posts out there to getting KVM guests to use different vlans via openvswitch. There are a lot that tell you to create fake bridges or create the ports via ovs-vsctrl and add tell libvirt to use that created interface or portgroup. Then there are almost no blogs that really say, when you setup openvswitch, this is how you make the interface settings stick. The correct way to do it is this basic flow 1) Create a bridge via ovs-vsctrl 2) Add your working interface to the bridge via ovs-vsctrl 3) Set your ip info on the new bridge 4) Create a libvirt network 5) Select the port group you want to use from your new network on the guest xml via libvirt 6) When the guest starts if the interface for the vlan isn’t created it will auto create it in openvswitch for you. So this works with Ubuntu 14.04 This also assumes bonding is setup via LACP on the host. It works the same if you just have a single interface like eth0. Just remove all the bond options. So my starting ifconfig for my bond0 device looks something like

bond0     Link encap:Ethernet  HWaddr 00:25:90:ed:dc:f0
          inet addr:  Bcast:  Mask:
          inet6 addr: fe80::f1:41ff:fe72:a331/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:713943 errors:0 dropped:0 overruns:0 frame:0
          TX packets:390750 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:49037015 (49.0 MB)  TX bytes:674651803 (674.6 MB)

So the first thing we want to do is install openvswitch-switch apt-get install openvswitch-switch Now we need to create a bridge in openvswitch ovs-vsctrl add-br br0 Now we need to add our working interface to the bridge. THIS WILL CAUSE YOUR CONNECTION TO DROP. Do not run this command if you don’t have remote KVM access or on the console. ovs-vsctrl add-port br0 bond0 Now that we have a bridge setup we need to give it IP information

ifconfig bond0 0
ifconfig br0 netmask
route add default gw

So now your bridge interface is up and it uses bond0 still. We gave it the same IP information. Now lets setup your the following file so the system reboots correctly

# The loopback network interface
auto lo
iface lo inet loopback

auto p1p1
iface p1p1 inet manual
  bond-master bond0

auto p1p2
iface p1p2 inet manual
  bond-master bond0

auto bond0
allow-br0 bond0
iface bond0 inet manual
  bond-mode 4
  bond-miimon 100
  bond-lcap-rate 1
  xmit_hash_policy layer3+4
  bond-slaves none
  ovs_bridge br0
  ovs_type OVSPort
  pre-up ifconfig $IFACE up
  post-down ifconfig $IFACE down

auto br0
allow-ovs br0
iface br0 inet static
  ovs_type OVSBridge
  ovs_ports br0
  pre-up ifconfig $IFACE up
  post-down ifconfig $IFACE down

The big things to add/change are as follows

  • allow-br0 bond0  This tells ovs to use bond0
  • Make sure your bond0 interface is set to manual
  • Also add the pre-up/post-up lines and address line to make sure the interface comes up ok
  • ovs_bridge br0 tells the system bond0 is apart of the ovs bridge br0
  • ovs_type OVSPort tells the system that this is a port in ovs
  • allow-ovs br0 tells the system this is for ovs
  • ovs_type OVSBridge tells the system this is a bridge
  • ovs_ports br0

Now that’s all set you can run reboot and the bridge should come up just fine Now lets create a network. Here is my sample network file. It creates a network with an un-tagged port and 2 other ports that get tagged with vlans 2 and 3

 <forward mode='bridge'/>
 <bridge name='br0'/>
 <virtualport type='openvswitch'/>
 <portgroup name='vlan-01' default='yes'>
 <portgroup name='vlan-02'>
     <tag id='2'/>
 <portgroup name='vlan-03'>
     <tag id='3'/>

So you’ll want to change the name of the network group and also the vlan info. My first vlan is un-tagged. and the next two are tagged. So create a file called vlans.xml and put that in now we can load it in libvirt

virsh net-define ./vlans.xml
virsh net-start vlans
virsh net-autostart vlans

Once that is all setup you can define an interface like

<interface type='network'>
 <source network='vlans' portgroup='vlan-02'/>

So my example if I show my running set looks like

root@vmnode2:~# ovs-vsctl show
    Bridge "br0"
        Port "vnet1"
            tag: 3
            Interface "vnet1"
        Port "br0"
            Interface "br0"
                type: internal
        Port "bond0"
            Interface "bond0"
        Port "vnet0"
            tag: 2
            Interface "vnet0"
    ovs_version: "2.0.1"

This way we don’t have to tell the guests to tag their traffic going out and we just have openvswitch tag the traffic. One gotcha might be your hardware switch has to know about the vlan ids even if you trunk the port the KVM host is connected to. In cisco that is like

vlan 2
name WebVlan

Simple as that.

Get Mandos working in Ubuntu

I’ve been doing a lot of playing around with full dis encryption. Now there’s one big problem when you do full disk encryption is when the server reboots you are left at a prompt to enter your password to mount the drive. This is solved by a tool call mandos. This is a client/server tool that the mandos client is loaded into the initrd image on the server and on boot will query the server and if the server will send back the encryption key to the client to use.

So the issue is the packages just don’t work in ubuntu 12.04 and even 14.04. I have a patch you can apply to your source if you want to rebuild the packaged versions to make debs of your own.

Below is the patch. This works for 14.04 but is basically the same for 12.04. I think the initrd script is slightly different but you can get the gist of it.

--- mandos-1.6.0.orig/initramfs-tools-hook
+++ mandos-1.6.0/initramfs-tools-hook
@@ -148,11 +148,7 @@ for hook in /etc/mandos/network-hooks.d/
 # GPGME needs /usr/bin/gpg
-if [ ! -e "${DESTDIR}/usr/bin/gpg" \
-    -a -n "`ls \"${DESTDIR}\"/usr/lib/libgpgme.so* \
-		2>/dev/null`" ]; then
-    copy_exec /usr/bin/gpg
+copy_exec /usr/bin/gpg
 # Config files
 for file in /etc/mandos/plugin-runner.conf; do
--- mandos-1.6.0.orig/mandos-keygen
+++ mandos-1.6.0/mandos-keygen
@@ -231,8 +231,12 @@ if [ "$mode" = keygen ]; then
     # Generate a new key in the key rings
     gpg --quiet --batch --no-tty --no-options --enable-dsa2 \
-	--homedir "$RINGDIR" --trust-model always \
-	--gen-key "$BATCHFILE"
+        --homedir "$RINGDIR" \
+        --import-ownertrust < /dev/null
 +    # Generate a new key in the key rings
 +    gpg --quiet --batch --no-tty --no-options --enable-dsa2 \
 +        --homedir "$RINGDIR" --trust-model always \
 +        --gen-key "$BATCHFILE"
      rm --force "$BATCHFILE"
      if tty --quiet; then

If anyone wants working packages for this let me know and I can post them for 12.04 and 14.04.

Verify user’s password on the command line

If there’s any chance you need to verify a user’s password on the command line and you are root you can use openssl with the info from /etc/shadow.

So first we want to grab the entry from /etc/shadow

cat /etc/shadow | grep mike

That will give us something that looks like


So the items we want are the $6 and the $tCFXiZHH. The $6 is important because that tells us the password is using sha512 for encryption. And the $tCFXiZHH is the salt.

So now we can run

mkpasswd -m sha-512 somePasswordHere tCFXiZHH

The output should match up with what’s above and if it is.. you have a valid password.

SPF DNS Lookup check for Nagios

Did you know that there is a max number of DNS lookups that can be done for an SPF lookup. The number is 10 and that includes any a or mx records along with includes to different hosts. I noticed this when I ran a SPF check on a domain and noticed it was 12. One of the errors is how Zendesk sets up their SPF record. For example their doc says use the following

v=spf1 include:_spf.zdsys.com ~all

Now that’s fine but if we do a

dig txt _spf.zdsys.com

That is one lookup and we get the following back

_spf.zdsys.com. 3600 IN TXT "v=spf1 include:_netblocks.zdsys.com ~all"

So they have another dns lookup. So in order to get to Zendesk’s valid servers you need to do an extra dns lookup. It adds up when you use google apps for email since they use around 5 lookups to get to all their hosts.

So I created a simple Nagios check in Python to keep tabs of our SPF record to make sure we stay under the 10 limit.

No more apt-get prompts

Sometimes you might be installing a package and don’t want it to prompt at all. Apt will prompt you a lot sometimes if you just do an apt-get install packageName. For example

  • If you want to install, if there are multiple packages
  • Accept package from a signed repo that you don’t have the key for
  • If you want to keep the old config

So you can run the following


apt-get -q -y -o DPkg::Options::=--force-confold --force-yes install packageName

grep only stderr from a command

So sometimes you have a command where you want to only grep stderr. For example I use Cronic to manage all my cronjobs. It’s really nice since it sends a nicely formatted email back to you if a command returns anything in stderr. Crontab alone will email me if anything in stdout/stderr is printed out from a command.

There is a problem with cronic though. There’s an app called s3cmd which uploads files to s3 and on large files this output can happen

WARNING: Retrying failed request: /postgresql/2013-05-05_hour-18.sql.bz2?partNumber=1&uploadId=m0gX4xcOU7IDla2B2p55xJXDfih_mm7rDx5bJvucUAmQYC10mwoHXVDjyoat_uzNJBYpedrWu7neakUpH3zGw-- ([Errno 110] Connection timed out)
WARNING: Waiting 3 sec...

This generally happens if the fule is very large. s3cmd will restart the transfer of the part and it will be uploaded just fine. So the main issue is s3cmd has no way to ignore warnings, at least as of 1.5.0-alpha2. If it does, I am overlooking it.

So in order to solve this I have decided I want to grep out the WARNING lines from stderr. Now the easy way to do this is re-direct stderr to stdout and pipe it to grep. Well this sucks cause you will lose valid error output. So the answer is something like this

cmd 2> >(grep -v "WARNING" >&2)

That will then allow you to grep just from stderr and the -v flag will set grep to ignore WARNING lines.

Remove internal hosts in postfix

A proper way to setup your network is to have a postfix relay server sitting somewhere on your network that every other server has access to via port 25. Now your relay server(s) are the only ones that should have outside access to port 25. All other servers should be firewalled off! 

So for example say your email chain looks like this for a new signup

web1.domain.com -> mailserver.domain.com -> internet -> user's mailbox

If the user views the email source they will see that the email started at web1.domain.com and it will include your internal IP address. So you are exposing internal IP addresses which isn’t very good at all.

So we can fix this in postfix very easily on the mailserver.domain.com config. For example say your internal network is

So lets remove them everything in that subnet along with So edit the following file


Then add the following line

header_checks = regexp:/etc/postfix/header_checks

Now create a new file


Then add the following in place

/^Received:.*\[127\.0\.0\.1/ IGNORE
/^Received:.*\[10\.114\..*/ IGNORE

Then restart postfix and you are good to go.

Next Page »