guestfish problems with virt-filesystems

I was trying to use guestfish to increase a qcow2 partition without booting live and fdisking and all that mess. So I tried to run it and was getting


# virt-filesystems --long --parts --blkdevs -h -a disk.qcow2
libguestfs: error: /usr/bin/supermin-helper exited with error status 1.
To see full error messages you may need to enable debugging.
See http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs

Scratching my head a bit and figured out you need to update the guestfs appliance packages

update-guestfs-appliance

Now the command works!


# virt-filesystems --long --parts --blkdevs -h -a disk.qcow2
Name       Type       MBR  Size  Parent
/dev/sda1  partition  83   3.8G  /dev/sda
/dev/sda2  partition  82   976M  /dev/sda
/dev/sda   device     -    5.0G  -

Openvswitch / KVM / Libvirt / Ubuntu / VLANs the right way

There are a lot of old blog posts out there to getting KVM guests to use different vlans via openvswitch. There are a lot that tell you to create fake bridges or create the ports via ovs-vsctrl and add tell libvirt to use that created interface or portgroup. Then there are almost no blogs that really say, when you setup openvswitch, this is how you make the interface settings stick. The correct way to do it is this basic flow 1) Create a bridge via ovs-vsctrl 2) Add your working interface to the bridge via ovs-vsctrl 3) Set your ip info on the new bridge 4) Create a libvirt network 5) Select the port group you want to use from your new network on the guest xml via libvirt 6) When the guest starts if the interface for the vlan isn’t created it will auto create it in openvswitch for you. So this works with Ubuntu 14.04 This also assumes bonding is setup via LACP on the host. It works the same if you just have a single interface like eth0. Just remove all the bond options. So my starting ifconfig for my bond0 device looks something like


bond0     Link encap:Ethernet  HWaddr 00:25:90:ed:dc:f0
          inet addr:10.128.7.121  Bcast:10.128.7.255  Mask:255.255.255.0
          inet6 addr: fe80::f1:41ff:fe72:a331/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:713943 errors:0 dropped:0 overruns:0 frame:0
          TX packets:390750 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:49037015 (49.0 MB)  TX bytes:674651803 (674.6 MB)

So the first thing we want to do is install openvswitch-switch apt-get install openvswitch-switch Now we need to create a bridge in openvswitch ovs-vsctrl add-br br0 Now we need to add our working interface to the bridge. THIS WILL CAUSE YOUR CONNECTION TO DROP. Do not run this command if you don’t have remote KVM access or on the console. ovs-vsctrl add-port br0 bond0 Now that we have a bridge setup we need to give it IP information


ifconfig bond0 0
ifconfig br0 10.128.7.121 netmask 255.255.255.0
route add default gw 10.128.7.1

So now your bridge interface is up and it uses bond0 still. We gave it the same IP information. Now lets setup your the following file so the system reboots correctly


# The loopback network interface
auto lo
iface lo inet loopback

auto p1p1
iface p1p1 inet manual
  bond-master bond0

auto p1p2
iface p1p2 inet manual
  bond-master bond0

auto bond0
allow-br0 bond0
iface bond0 inet manual
  bond-mode 4
  bond-miimon 100
  bond-lcap-rate 1
  xmit_hash_policy layer3+4
  bond-slaves none
  ovs_bridge br0
  ovs_type OVSPort
  pre-up ifconfig $IFACE up
  post-down ifconfig $IFACE down
  address 0.0.0.0

auto br0
allow-ovs br0
iface br0 inet static
  address 10.128.7.121
  netmask 255.255.255.0
  gateway 10.128.7.1
  dns-nameservers 10.128.7.4 10.128.7.5
  ovs_type OVSBridge
  ovs_ports br0
  pre-up ifconfig $IFACE up
  post-down ifconfig $IFACE down

The big things to add/change are as follows

  • allow-br0 bond0  This tells ovs to use bond0
  • Make sure your bond0 interface is set to manual
  • Also add the pre-up/post-up lines and address line to make sure the interface comes up ok
  • ovs_bridge br0 tells the system bond0 is apart of the ovs bridge br0
  • ovs_type OVSPort tells the system that this is a port in ovs
  • allow-ovs br0 tells the system this is for ovs
  • ovs_type OVSBridge tells the system this is a bridge
  • ovs_ports br0

Now that’s all set you can run reboot and the bridge should come up just fine Now lets create a network. Here is my sample network file. It creates a network with an un-tagged port and 2 other ports that get tagged with vlans 2 and 3


<network>
 <name>vlans</name>
 <uuid>4216c8df-349d-4a32-a6ae-533135a9d682</uuid>
 <forward mode='bridge'/>
 <bridge name='br0'/>
 <virtualport type='openvswitch'/>
 <portgroup name='vlan-01' default='yes'>
 </portgroup>
 <portgroup name='vlan-02'>
   <vlan>
     <tag id='2'/>
   </vlan>
 </portgroup>
 <portgroup name='vlan-03'>
   <vlan>
     <tag id='3'/>
   </vlan>
 </portgroup>
</network>

So you’ll want to change the name of the network group and also the vlan info. My first vlan is un-tagged. and the next two are tagged. So create a file called vlans.xml and put that in now we can load it in libvirt


virsh net-define ./vlans.xml
virsh net-start vlans
virsh net-autostart vlans

Once that is all setup you can define an interface like


<interface type='network'>
 <source network='vlans' portgroup='vlan-02'/>
</interface>

So my example if I show my running set looks like


root@vmnode2:~# ovs-vsctl show
19655270-bcee-4b57-b2d5-5a180da422a8
    Bridge "br0"
        Port "vnet1"
            tag: 3
            Interface "vnet1"
        Port "br0"
            Interface "br0"
                type: internal
        Port "bond0"
            Interface "bond0"
        Port "vnet0"
            tag: 2
            Interface "vnet0"
    ovs_version: "2.0.1"

This way we don’t have to tell the guests to tag their traffic going out and we just have openvswitch tag the traffic. One gotcha might be your hardware switch has to know about the vlan ids even if you trunk the port the KVM host is connected to. In cisco that is like


vlan 2
name WebVlan
exit

Simple as that.

Get Mandos working in Ubuntu

I’ve been doing a lot of playing around with full dis encryption. Now there’s one big problem when you do full disk encryption is when the server reboots you are left at a prompt to enter your password to mount the drive. This is solved by a tool call mandos. This is a client/server tool that the mandos client is loaded into the initrd image on the server and on boot will query the server and if the server will send back the encryption key to the client to use.

So the issue is the packages just don’t work in ubuntu 12.04 and even 14.04. I have a patch you can apply to your source if you want to rebuild the packaged versions to make debs of your own.

Below is the patch. This works for 14.04 but is basically the same for 12.04. I think the initrd script is slightly different but you can get the gist of it.


--- mandos-1.6.0.orig/initramfs-tools-hook
+++ mandos-1.6.0/initramfs-tools-hook
@@ -148,11 +148,7 @@ for hook in /etc/mandos/network-hooks.d/
 done
 
 # GPGME needs /usr/bin/gpg
-if [ ! -e "${DESTDIR}/usr/bin/gpg" \
-    -a -n "`ls \"${DESTDIR}\"/usr/lib/libgpgme.so* \
-		2>/dev/null`" ]; then
-    copy_exec /usr/bin/gpg
-fi
+copy_exec /usr/bin/gpg
 
 # Config files
 for file in /etc/mandos/plugin-runner.conf; do
--- mandos-1.6.0.orig/mandos-keygen
+++ mandos-1.6.0/mandos-keygen
@@ -231,8 +231,12 @@ if [ "$mode" = keygen ]; then
 
     # Generate a new key in the key rings
     gpg --quiet --batch --no-tty --no-options --enable-dsa2 \
-	--homedir "$RINGDIR" --trust-model always \
-	--gen-key "$BATCHFILE"
+        --homedir "$RINGDIR" \
+        --import-ownertrust < /dev/null
+    # Generate a new key in the key rings
+    gpg --quiet --batch --no-tty --no-options --enable-dsa2 \
+        --homedir "$RINGDIR" --trust-model always \
+        --gen-key "$BATCHFILE"
     rm --force "$BATCHFILE"

     if tty --quiet; then

If anyone wants working packages for this let me know and I can post them for 12.04 and 14.04.

Verify user’s password on the command line

If there’s any chance you need to verify a user’s password on the command line and you are root you can use openssl with the info from /etc/shadow.

So first we want to grab the entry from /etc/shadow

cat /etc/shadow | grep mike

That will give us something that looks like

mike:$6$tCFXiZHH$tFN8HZg/hXxYePSLZHVyBWuCFKlyesvKGKefwef2qR.DEKrrkvDUhewfwefuM.kU1HewfwE3HvprG/oMnizG2.:15734:0:99999:7:::

So the items we want are the $6 and the $tCFXiZHH. The $6 is important because that tells us the password is using sha512 for encryption. And the $tCFXiZHH is the salt.

So now we can run

mkpasswd -m sha-512 somePasswordHere tCFXiZHH

The output should match up with what’s above and if it is.. you have a valid password.

SPF DNS Lookup check for Nagios

Did you know that there is a max number of DNS lookups that can be done for an SPF lookup. The number is 10 and that includes any a or mx records along with includes to different hosts. I noticed this when I ran a SPF check on a domain and noticed it was 12. One of the errors is how Zendesk sets up their SPF record. For example their doc says use the following

v=spf1 include:_spf.zdsys.com ~all

Now that’s fine but if we do a

dig txt _spf.zdsys.com

That is one lookup and we get the following back

_spf.zdsys.com. 3600 IN TXT "v=spf1 include:_netblocks.zdsys.com ~all"

So they have another dns lookup. So in order to get to Zendesk’s valid servers you need to do an extra dns lookup. It adds up when you use google apps for email since they use around 5 lookups to get to all their hosts.

So I created a simple Nagios check in Python to keep tabs of our SPF record to make sure we stay under the 10 limit.

No more apt-get prompts

Sometimes you might be installing a package and don’t want it to prompt at all. Apt will prompt you a lot sometimes if you just do an apt-get install packageName. For example

  • If you want to install, if there are multiple packages
  • Accept package from a signed repo that you don’t have the key for
  • If you want to keep the old config

So you can run the following

 

apt-get -q -y -o DPkg::Options::=--force-confold --force-yes install packageName

grep only stderr from a command

So sometimes you have a command where you want to only grep stderr. For example I use Cronic to manage all my cronjobs. It’s really nice since it sends a nicely formatted email back to you if a command returns anything in stderr. Crontab alone will email me if anything in stdout/stderr is printed out from a command.

There is a problem with cronic though. There’s an app called s3cmd which uploads files to s3 and on large files this output can happen


WARNING: Retrying failed request: /postgresql/2013-05-05_hour-18.sql.bz2?partNumber=1&uploadId=m0gX4xcOU7IDla2B2p55xJXDfih_mm7rDx5bJvucUAmQYC10mwoHXVDjyoat_uzNJBYpedrWu7neakUpH3zGw-- ([Errno 110] Connection timed out)
WARNING: Waiting 3 sec...

This generally happens if the fule is very large. s3cmd will restart the transfer of the part and it will be uploaded just fine. So the main issue is s3cmd has no way to ignore warnings, at least as of 1.5.0-alpha2. If it does, I am overlooking it.

So in order to solve this I have decided I want to grep out the WARNING lines from stderr. Now the easy way to do this is re-direct stderr to stdout and pipe it to grep. Well this sucks cause you will lose valid error output. So the answer is something like this

cmd 2> >(grep -v "WARNING" >&2)

That will then allow you to grep just from stderr and the -v flag will set grep to ignore WARNING lines.

Remove internal hosts in postfix

A proper way to setup your network is to have a postfix relay server sitting somewhere on your network that every other server has access to via port 25. Now your relay server(s) are the only ones that should have outside access to port 25. All other servers should be firewalled off! 

So for example say your email chain looks like this for a new signup

web1.domain.com -> mailserver.domain.com -> internet -> user's mailbox

If the user views the email source they will see that the email started at web1.domain.com and it will include your internal IP address. So you are exposing internal IP addresses which isn’t very good at all.

So we can fix this in postfix very easily on the mailserver.domain.com config. For example say your internal network is

10.114.0.0/16

So lets remove them everything in that subnet along with 127.0.0.1. So edit the following file

/etc/postfix/main.cf

Then add the following line

header_checks = regexp:/etc/postfix/header_checks

Now create a new file

/etc/postfix/header_checks

Then add the following in place

/^Received:.*\[127\.0\.0\.1/ IGNORE
/^Received:.*\[10\.114\..*/ IGNORE

Then restart postfix and you are good to go.

How many bits per point in a whisper file in graphite

This is an easy one. You do an ls over a whisper file and get something like

1.1M -rwxr-xr-x  1 root root 1.1M Aug  9 11:06 Active.wsp

Now you are planning to save more metrics in your graphite server or want more points in your retention so you want to plan how much space you’ll need given an estimated guess of your metric count out 6 months from now. So you can run the following over that whisper file to get info on it.

$ whisper-info.py Active.wsp
maxRetention: 31536000
xFilesFactor: 0.5
aggregationMethod: average
fileSize: 1062772

Archive 0
retention: 604800
secondsPerPoint: 60
points: 10080
size: 120960
offset: 52

So you see for Archive 0 we have a certain size and number of points. So from here it’s easy math

120960 / 10080 = 12

So for each whisper data point you save it will allocate 12 bits on your filesystem. So you can plan our growth since carbon pre-allocates the space needed for whisper files so your disk isn’t doing any random seeks while pulling up data for your graphs.

 

Google Authenticator with OpenVPN for 2 factor auth

This post will describe how to get it working with Ubuntu 12.04. It should work with other versions along with other Linux distros. The only difference is the package names might change.

Install the packages

Run the following command

apt-get -y install openvpn libpam-google-authenticator

Setup OpenVPN

Place the cert key helps

mkdir /etc/openvpn/easy-rsa/
cp -R /usr/share/doc/openvpn/examples/easy-rsa/2.0/* /etc/openvpn/easy-rsa/

Then you need to edit the following file

/etc/openvpn/easy-rsa/vars

Scroll all the way down and find the following section and change it to your needs

export KEY_COUNTRY=US
export KEY_PROVINCE=NY
export KEY_CITY=NewYork
export KEY_ORG=”Company VPN”
export KEY_EMAIL=”sysops@company.com”

Set the permissions

cd /etc/openvpn/easy-rsa/
sudo chown -R root:admin .
sudo chmod g+w .

Source the vars file to pull in needed environment variables

source ./vars

Run the scripts to build your keys

./clean-all
./build-dh
./pkitool –initca
./pkitool –server server
cd keys
openvpn –genkey –secret ta.key
sudo cp server.crt server.key ca.crt dh1024.pem ta.key ../../

Create the following file

/etc/openvpn/up.sh

With the contents of

#!/bin/sh

BR=$1
DEV=$2
MTU=$3

/sbin/ip link set “$DEV” up promisc on mtu “$MTU”
/sbin/brctl addif $BR $DEV

Then create another file called

/etc/openvpn/down.sh

With the contents of

#!/bin/sh

BR=$1
DEV=$2

/sbin/brctl delif $BR $DEV
/sbin/ip link set “$DEV” down

Now set permissions on this files

chmod +x /etc/openvpn/up.sh /etc/openvpn/down.sh

Then you want to add the following to /etc/sysctl.conf

net.ipv4.ip_forward = 1

That will set it on reboot but you want to set it now run the following

echo 1 > /proc/sys/net/ipv4/ip_forward

Now setup pam to work for openvpn. Create the following file

/etc/pam.d/openvpn

Then put the following in

auth requisite pam_google_authenticator.so forward_pass
auth required pam_unix.so use_first_pass

Now you need to edit your openvpn config. I called mine server.conf in /etc/openvpn and you want to add the following

plugin /usr/lib/openvpn/openvpn-auth-pam.so openvpn

Now once openvpn is started you can run the following as the user you are setting up VPN for. They will need a shell account and they can do this themselves if needed. So run the following command

google-authenticator

Then choose the following options in this order.

Do you want authentication tokens to be time-based (y/n) y
Do you want me to update your “/home/bla/.google_authenticator” file (y/n) y

Do you want to disallow multiple uses of the same authentication
token? This restricts you to one login about every 30s, but it increases
your chances to notice or even prevent man-in-the-middle attacks (y/n) y

By default, tokens are good for 30 seconds and in order to compensate for
possible time-skew between the client and the server, we allow an extra
token before and after the current time. If you experience problems with poor
time synchronization, you can increase the window from its default
size of 1:30min to about 4min. Do you want to do so (y/n) y

If the computer that you are logging into isn’t hardened against brute-force
login attempts, you can enable rate-limiting for the authentication module.
By default, this limits attackers to no more than 3 login attempts every 30s.
Do you want to enable rate-limiting (y/n) y

You notice this command might show a QR code at the top which you can download the Google Authenticator app on your phone to scan that code and it will setup your account automatically. If it doesn’t show a QR code it will show a link that will display the code.

You can now run openvpn and connect and the username is your shell user and if you set the password to test123 and google gives you a code of 847324, your password would be test123847324

 

Next Page »