Setup DRBD for master/master

Posted by on August 23, 2010 · 2 Comments 

At my job at OpenSky we have a common NFS share that is failed over to a slave nfs node if the master dies. It was setup so that the slave would just rsync from the master. This was done to save time. Not that I have some time, I decided to finally setup drbd on the two nodes to keep the files always in sync.

So below are my nodes with hostnames/ips and drive information

Node1
host: nfs1.domain.com
ip: 192.168.1.46
partition: /dev/sdb1

Node2
host: nfs2.domain.com
ip: 192.168.1.47
partition: /dev/sdb1

So once drbd is installed setup the following file

/etc/drbd.conf

To look something like this. You will want to change a few things in red. Put this file on both nodes.

global {
  usage-count yes;
}
common {
  protocol C;
}
resource nfs {
  meta-disk internal;
  device     /dev/drbd1;
  syncer {
    verify-alg sha1;
  }
  net {
    allow-two-primaries;
  }
  on nfs1.domain.com {
    disk       /dev/sdb1;
    address    192.168.1.46:7789;
  }
  on nfs2.domain.com {
    disk       /dev/sdb1;
    address    192.168.1.47:7789;
  }
}

I choose to use protocol C. You can see the following protocols below. Also the on nfs1.domain.com has to match the output of a hostname command

http://www.drbd.org/users-guide/s-replication-protocols.html

Now on both nodes you want to init the drbd partition. Make sure this is a blank partition. It will wipe out everything that is one it. I called my resource nfs. If you changed it the commands will be a bit different based on what name you gave your resource

drbdadm create-md nfs

You should get an output like

initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.
success

Now make sure drbd can be run on startup and start it up now

chkconfig drbd on
service drbd start

Once you start drbd on both nodes it will connect to both. If you just start one and not the other.. the other will wait for the second node to come alive.

You can now make a filesystem. Make sure it is a filesystem for clustering. I recommend ocfs2 but gfs2 also works.

Filed under Sys Admin · Tagged with , , ,

Setup MongoDB master/slave replication

Posted by on June 1, 2010 · 3 Comments 

I am recently getting into MongoDB. Thanks to my company (OpenSky) sending me to MongoNYC event. I am also tasked with setting up infrastructure to run MongoDB. So we are starting off with a simple master/slave MongoDB setup.

I always thought MySQL was easy to setup replication on but MongoDB proves it just gets easier. This will expect you are using the official MongoDB provided RPMs from their repository.

So the first thing we want to do is install the -stable branch on both nodes

yum install mongo-stable-server

Now that you have it installed lets edit the init.d script. As of writing this, they do not make use of a sysconfig script.

/etc/init.d/mongod

Now find the following line

OPTIONS=" -f /etc/mongod.conf"

For the master you want to make it look like

OPTIONS=" -f /etc/mongod.conf --master"

For the slave you want to do this

OPTIONS=" -f /etc/mongod.conf --slave --source 10.0.0.1"

10.0.0.1 is the IP address of the master server.

Now start mongod on both servers and run chkconfig to make sure it loads on restart

chkconfig mongod on
service mongod start

Now on the master run the following command to get into the Mongo Shell

mongo

Now in the mongo shell run

db.runCommand("ismaster");

You should get something like this if everything is setup correctly

> db.runCommand("ismaster");
{ "ismaster" : 1, "msg" : "not paired", "ok" : 1 }

Now lets insert something into the master server. By default you connect to the test database and this is not replicated so you must switch to a new database. In Mongo, you don’t have to create a database, just use it and it will create it.

> use zcentric
switched to db zcentric
> db.users.insert({'username': 'mzupan', 'full_name': 'Mike Zupan'});
> db.users.find({})
{ "_id" : ObjectId("4c05431becb442845873ec05"), "username" : "mzupan", "full_name" : "Mike Zupan" }

Now on the slave run

> use zcentric
switched to db zcentric
> db.users.find({})
{ "_id" : ObjectId("4c05431becb442845873ec05"), "username" : "mzupan", "full_name" : "Mike Zupan" }

So You see it works nicely! Have fun with Mongo.. I am

Filed under Sys Admin · Tagged with , , ,

A better Jira init.d script for Redhat/CentOS

Posted by on April 15, 2010 · 4 Comments 

I wrote a bit better init.d script for Jira for Redhat/CentOS

The first thing you want to do is create a sysconfig file. Below is the filename

/etc/sysconfig/jira

The file has the following contents in it

#
# The user Jira runs as
#
JIRA_USER=jira

#
# The home directory of Jira
#
JIRA_HOME=/opt/jira/current

Now create the init.d script

/etc/init.d/jira

It has the following contents in it.

#!/bin/sh
#
# JIRA startup script
#
# chkconfig: 2345 80 05
# description: JIRA

# Source function library.
. /etc/rc.d/init.d/functions

if [ -f /etc/sysconfig/jira ]; then
    . /etc/sysconfig/jira
fi

prog=jira
RETVAL=0

start() {
        echo -n $"Starting $prog: "
        /bin/su -m $JIRA_USER -c "cd $JIRA_HOME/logs && $JIRA_HOME/bin/startup.sh &> /dev/null"
        RETVAL=$?
        if [ $RETVAL = 0 ]
        then
                echo $! > /var/run/jira.pid
                echo_success
        else
                echo_failure
        fi

        echo

        return $RETVAL
}

stop() {
        echo -n $"Stopping $prog: "
        /bin/su -m $JIRA_USER -c "$JIRA_HOME/bin/shutdown.sh &> /dev/null"
        RETVAL=$?
        if [ $RETVAL = 0 ]
        then
                rm -f /var/run/jira.pid
                echo_success
        else
                echo_failure
        fi
        echo

        return $RETVAL
}

case "$1" in
        start)
                start
                ;;
        stop)
                stop
                ;;
        restart)
                stop
                sleep 5
                start
                ;;
        *)
                echo $"Usage: /etc/init.d/$prog {start|restart|stop}"
                exit 1
                ;;
esac

exit $RETVAL

Now make permissions correct

chmod +x /etc/init.d/jira

Now you can chkconfig it

chkconfig jira on

This will make it look a bit more Redhat/CentOS like.

Filed under Sys Admin · Tagged with , , , ,

Install fabric on Redhat/CentOS 5

Posted by on March 12, 2010 · Leave a Comment 

Fabric is a great tool for helping deploy applications. It is written in Python and still in it’s early stages. The big issue is that fabric will not run on python v2.4. That is a problem since Redhat/CentOS v5 ships with v2.4 so we have to make it work.

So the first thing to do is install python 2.5. I found great rpms here. I am going to mirror the RPMS on my blog just in case. You also need to install tkinter25 which I provide for you also. These are only for 64bit. You can do some googling to find the 32 bit versions I am sure. You also need to install libtk and libTix.

Install some dependancies

yum install tk tix

Now lets install python and tkinter

mkdir python
cd python
wget http://zcentric.com/files/python25-2.5.1-bashton1.x86_64.rpm
wget http://zcentric.com/files/python25-devel-2.5.1-bashton1.x86_64.rpm
wget http://zcentric.com/files/python25-libs-2.5.1-bashton1.x86_64.rpm
wget http://zcentric.com/files/python25-test-2.5.1-bashton1.x86_64.rpm
wget http://zcentric.com/files/python25-tools-2.5.1-bashton1.x86_64.rpm
wget http://zcentric.com/files/tkinter25-2.5.1-bashton1.x86_64.rpm
rpm -ivh *.rpm

Remove all the python25 rpms

rm -rf *.rpm

Now you have v2.4 and v2.5 installed. Now lets download the needed fabric dependancies and fabric. I built these to run for python25

wget http://zcentric.com/files/fabric-0.9-0.1.b1.noarch.rpm
wget http://zcentric.com/files/python25-setuptools-0.6c5-2.noarch.rpm
wget http://zcentric.com/files/python25-paramiko-1.7.6-1.noarch.rpm
wget http://zcentric.com/files/python25-crypto-2.0-1.rf.x86_64.rpm
rpm -ivh *.rpm
Now we just need to edit the fab file to change the python version it will run as by default
vi /usr/bin/fab
Change
#!/usr/bin/python
To
#!/usr/bin/python25
Now you can run fab and you should get something like
[root@dev1 ~]# fab
Fatal error: Couldn’t find any fabfiles!
Aborting.
More blog posts to come on how to use fabric

Filed under Python, Sys Admin · Tagged with , , , ,

Bash Command Logger with Curl Support

Posted by on March 9, 2010 · 9 Comments 

There is a great project called Bash Paranoia. Right now their site is busted so I can’t link to it. Its a patch that applies to bash that allows commands to be logged to syslog. I basically took this one step further and added curl support.

The bash paranoia patch and my curl addition can be found on my GitHub project page

http://github.com/mzupan/bash-paranoia-curl

Below is my patch I wrote. Right now it will only work with 64bit systems. It should be easy to make it work with 32bit systems if you edit the patch file at the bottom where I patch Makefile.in. Change the lib64 to lib and you should be good to go

Now if you want to install these patches you would run the following commands. My curl patch needs the base paranoia patch to work. I don’t even think it will apply alone.

wget http://zcentric.com/files/bash-paranoia.patch
wget http://zcentric.com/files/bash-paranoia-curl.patch
tar zxf bash-3.2.tar.gz
cd bash-3.2
patch -p0 < ../bash-paranoia.patch
patch -p1 < ../bash-paranoia-curl.patch
./configure ––enable-paranoia #you can include  other configure flags here
make
make install

That will get you going and the next time you login (if bash if your default shell) you will see the following in your logs (for redhat is is /var/log/messages)

Mar  9 15:24:02 263724-mgmt1 bash: user: mzupan as root from ip: 192.168.71.154:pts/0 execs: ‘cat /var/log/messages’

There you go a nice little command logger that will tell you most of what you need to do to keep tabs on users.

Now if you want to also append this to a db somewhere then curl and a web endpoint is the best solution. So my database look like

CREATE TABLE `commandlog` (
`id` int(11) NOT NULL auto_increment,
`server` varchar(100) NOT NULL,
`user_login` varchar(100) NOT NULL,
`user_run` varchar(100) NOT NULL,
`ip` varchar(100) NOT NULL,
`session` varchar(100) NOT NULL,
`command` longtext NOT NULL,
`ts` datetime NOT NULL,
PRIMARY KEY  (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=1 DEFAULT CHARSET=latin1;
So now here is a simple php app to save it into a db. THERE IS NO SANITY CHECKING HERE. THIS IS JUST A SAMPLE!
I am writing this without checking any code.. so it might be broken.
<?php
$server = $_SERVER['REMOTE_ADDR'];
$user_login = $_POST['user_login'];
$user_run =  $_POST['user_run'];
$ip =  $_POST['ip'];
$session =  $_POST['session'];
$command =  $_POST['command'];
$ts = time();
$sql = “INSERT INTO commandlog(server,user_login,user_run,ip,session,command,ts) VALUES(‘$server’,'$user_login’,'$user_run’,'$ip’,'$session’,'$command’,'$ts’)”;
// place into sql now.. too lazy to do this for you
?>
Now if you want to enable the curl post on the server you edit the following file
/etc/bash.conf
Here is an example
URL=http://1.1.1.1/endpoint/
Have fun!
If you want a spec file that will work for Redhat/CentOS 4/5 64bit you can download the following file

Filed under Sys Admin · Tagged with , , , , , ,