Postfix has an option where you can setup a relayhost to send all mail to another mail server. It looks something like

relayhost = [mail.domain.com]:25

So that will send all mail being sent out from the server to mail.domain.com on port 25. So what happens if you want to send mail that is sent to company.com out that server and not through another relay. You would do the following in main.cf

transport_maps = hash:/etc/postfix/transport

Then your /etc/postfix/transport would look something like

company.com :
* smtp:[mail.domain.com]:25

Then you want to run

postmap /etc/postfix/transport

Then reload postfix

service postfix reload

Now all mail for company.com will send out that server while all other mail will go through mail.domain.com:25

File this under you learn something new every day.

Ever do a ps aux only to get something like this to happen?

xymon 26931 0.0 0.0 38944 1396 ? S 05:44 0:00 sh -c vmstat 300 2 1>/usr/lib64/xymon/client/tmp/hob

So the output of ps aux gets cut off when it hits the end of your terminal. And you have to expand your width a lot to see it. It always bothered me till I decided to read the man page.

Now I learned you can do

ps aux --width 1000

That will line wrap your output but cut the output at 1000 characters. You can also pass w’s into the command like

ps auxww

That will ignore your terminals width settings

So now we can add the following in our .bashrc or for system wide /etc/profile so we never have to remember to add a ww or –width again

alias ps='ps ww'

So that alias means you can type ps aux and it will be like your typed ps auxww

I recently wrote a quick little fabric function to create a fabric shell. This works with a stock fabric install, you just might have to change where you import it from. It is at my Github page below.

https://github.com/mzupan/fabric_shell

If you place shell.py in the same directory as fabfile.py you can use it like

from fabric.api import *
from fabric.decorators import hosts

import shell

env.hosts = [
    'host1.domain.com',
    'host2.domain.com'
]

@hosts('')
def cmd():
    shell.shell()

Then you can use like like

$ fab cmd
[] Executing task 'cmd'
fabric::> id
[host1.domain.com] run: id
[host1.domain.com] out: uid=1020(mzupan) gid=1020(mzupan) groups=1020(mzupan)
[host1.domain.com] out:
[host2.domain.com] run: id
[host2.domain.com] out: uid=1020(mzupan) gid=1020(mzupan) groups=1020(mzupan)
[host2.domain.com] out:
fabric::> .sudo id
[host1.domain.com] sudo:  id
[host1.domain.com] out: uid=0(root) gid=0(root) groups=0(root)
[host1.domain.com] out:
[host2.domain.com] sudo:  id
[host2.domain.com] out: uid=0(root) gid=0(root) groups=0(root)
[host2.domain.com] out:
fabric::> 

Right now any command that is issued in the fabric shell gets run on all hosts in env.hosts and if you add .sudo in front of the command it will run them as root.

This is just another reason I find Django so easy.

For example.. Say you are making a monitoring application. Now you have a bunch of servers and you have certain groups of servers. A server can belong in more then one group also. So that is pretty easy and all with Django.

Now in your template you have a few pages. One page that displays the group information like all the servers in the group and another that displays the server information and will list all the groups that server is in. So below is my basic model layout

class Group(models.Model):
    parent = models.ForeignKey("Group", null=True, blank=True)
    user = models.ForeignKey(User)

    name = models.CharField(max_length=100)
    about = models.TextField(blank=True, null=True)

    created = models.DateTimeField(auto_now_add=True)
    updated = models.DateTimeField(auto_now=True)

    def __unicode__(self):
        return self.name

class Server(models.Model):
    hostname = models.CharField(max_length=300, db_index=True)
    about = models.TextField(blank=True, null=True)

    created = models.DateTimeField(auto_now_add=True)
    updated = models.DateTimeField(auto_now=True)

    groups = models.ManyToManyField(Group)

    def __unicode__(self):
        return self.hostname

So to get all the groups my server is a member of is pretty easy.

s = Server.objects.get(id=1)
s.groups.all()

Now to get all the servers in a group is also easy

g = Group.objects.get(id=1)
g.server_set.all()

Recently ran into this problem. We had a crontab entry that looked something like this

0 0 * * * /usr/bin/sleep $((RANDOM\%90)); cmdHere

What that attempted was to run a command a random amount of seconds once midnight hit. Now running the command on console causes it to error out

$  /usr/bin/sleep $((RANDOM\%90))
-bash: RANDOM\%90: syntax error: invalid arithmetic operator (error token is "\%90")

So what you might be saying is well the \ is messing it up so remove it. Well the problem is % is a reserved character in crontab land. It pretty much means ignore everything after it so you escape it. But its not working as expected. If you remove the \ from the shell command it works just fine.

So the real answer here was use expr. Once that was in all the crontab errors went away

0 0 * * * /bin/sleep `/usr/bin/expr $RANDOM \% 90`; cmdHere

A common deploy strategy for a lot of web sites is as follows

v1.0.10
v1.0.11
v1.1.0
current -> v1.1.0

So in this directory we have three directories and one symlink. The directories contain our web application and are tagged with the source control version they were tagged with by the developers. Now its a common practice when deploying a new version to upload the new release which will be v1.1.1 and then just change the symlink to point to the new location. That really isn’t a great idea.

Sure it will work for 99.9% of your traffic but if you have a busy website it could really effect users as your do the upgrade. You are probably saying well I’m forcing the new symlink with the following command

ln -snf v1.1.1 current

Did you ever look at the system calls using the -f flag for ln does? If not lets take a look

# strace ln -snf v1.1.1 current 2>&1 | grep link
symlink("v1.1.1", "current")            = -1 EEXIST (File exists)
unlink("current")                       = 0
symlink("v1.1.1", "current")            = 0

So you can see it first tries to create the link and finds the symlink is there so it unlinks curent and makes the new symlink. The man page even says this is the case!

-f If the target file already exists, then unlink it so that the link may occur. (The -f option overrides any previous -i options.)

So how do we make this more atomic so that our users do not even notice and everyone is happy? Well its very simple.

ln -s v1.1.1 current.new && mv -Tf current.new current

That will do an atomic update. If you want to learn more about the -T flag you can find it here

It might not be 100% atomic but it’s really the best I can get right now. POSIX says its atomic so right now I’ll trust it.

I’ve been a big fan of the fabric project for a long time now. It has made my life very easy in the past and gets better with each new release. Well a few weeks ago I got introduced to a fork of the project by Tav. It provides many new feature additions to fabric and makes fabric a little bit better. So the best way to get it started is via the following

git clone git://github.com/tav/pylibs.git
cd pylibs
python setup.py
export PYTHONPATH=$PYTHONPATH:`pwd`

Once installed, you have to create a fab app

vi /usr/bin/fab

Then you want to add the following contents to it

#!/usr/bin/env python

from fabric.main import main
main()

Then you want to give it the correct permissions

chmod +x /usr/bin/fab

Now create a directory and place a fabfile.py file in it.

Now create a config.yml file and use the following as a simple template

default:
  shell: /bin/bash -l -c

servers:
  directory: /dir
  hosts:
    - host1.domain.com
    - host2.domain.com

Then your fab file could look something like

from fabric.api import *

env.config_file = 'config.yml'
env.user = 'mzupan'

@task('servers')
def command():
    env().multirun('uname -a')

Normally fabric runs in serial, but Tav’s port adds a multirun() function that runs the commands in parallel. BONUS!

Feel free to play around with it and check out his blog post on it to get more commands.

http://tav.espians.com/fabric-python-with-cleaner-api-and-parallel-deployment-support.html

You can see all the questions I have compiled here.

Question

You are already logged into a server and a user runs a fork bomb script. How can you stop the fork bomb without restarting the server or bringing any services offline that are currently running?

You also know the name of the running fork bomb script.

Reason for the this question

This will tell you how well the canidate knows about the signals Linux uses with processes. I would think most people would say the easy thing

killall -9 scriptName

This will not work due the nature of a fork bomb. The reason is the killall does not hold a lock on the process table so each one that is killed a new one takes its place in the process table.

Also you will not be able to run a killall due to the shell forking off another shell to run the killall

This question also tells you if the admin knows about Linux internals. That just running a killall will fork a new process buy running exec killall will run the killall in the current process and not fork out a new one

Answer

So for this I will use a fork bomb script that is written in C. Below is an example

#include 

int main(void)
{
  for(;;)
    fork();
  return 0;
}

We can compile it like

gcc -o fork fork.c

So the way to stop it is sending a SIGSTOP signal to each fork bomb process and once they are all stopped you can send a SIGKILL to each process.

So if you run a

killall -STOP fork

You will get a error message like resources not avaliable so you cannot run that. You can run it with exec.

exec killall -STOP fork
exec killall -9 fork

That will stop the fork bomb.

So I’m starting a bit of a series on my blog. I’ve found there are no good senior level Linux admin questions out there to ask. A lot of companies with a sys admin team have their default questions they ask but when you are hiring your first sys admin and want a senior guy there is no real way to tell how good he is if no one on your team knows Linux all that well.

So maybe some of these will help some companies figure it all out.

Question

The server is running Apache and by mistake one of the log files gets deleted via

rm domain.com-access_log

Without installing any 3rd party recovery tools how can you recover this file that your boss needs. Also assume that as soon as the log file was deleted that Apache was not restarted.

Reason for this question

This is a really tough question to answer. I would think a lot of senior guys might not even know the answer to this. I certainly did not when I got it asked by Google five years ago.

If the person gets it right I would say he has a really great grasp on how Linux /proc system works and I would say he is a pretty solid admin if he answered all your other questions correctly

Answer

So the simple answer is the deleted file is still held open by Apache so it can still be recovered in the /proc filesystem.

So here is how to get to it. For example I have a really small access_log

  4 -rw-r--r--   1 root root   2262 Jan 13 12:32 access_log

So lets remove it

rm -f access_log

So the file has been deleted now. So lets find the process number for the main apache process. It will be owned by root

[root@laptop httpd]# ps aux | grep httpd
root      8070  0.0  0.3  38468 10948 ?        Ss   12:31   0:00 /usr/sbin/httpd
apache    8072  0.0  0.1  38388  5844 ?        S    12:31   0:00 /usr/sbin/httpd
apache    8073  0.0  0.3  40752 11532 ?        S    12:31   0:00 /usr/sbin/httpd

So we see that the pid for the main apache process is 8070. So now lets list the file descriptors

ls -lsa /proc/8070/fd

We get something like

0 l-wx------ 1 root root 64 Jan 13 12:34 11 -> /var/log/httpd/access_log (deleted)

As you can see that is our file and its marked as deleted.. so we can do

cp /proc/8070/fd/11 /tmp/access_log

Then from there you can stop apache.. move the access_log in its correct place and start apache back up

I use to hate this question. I was reading a post on Reddit about some guy that went through the programming interview at Facebook and it made me remember the simple interview question I always hated answering. It always went something like this

How many usable IPs in a /24

Ok that is an easy one.. 256-2=254 not a big deal.. next!

Then they would ask

How many usable IPs in a /28

That is what usually made me say “Networking is not really my strong point” Well got that one wrong.. So I went out to fix this many many years ago. Well there’s an easy step you can do in your head.

All subnets going up are half of the one below it. So your /24 would have 256 IPs in it. So it would go like this

/25 = 128
/26 = 64
/27 = 32
/28 = 16

So there was my answer 16 IPs and 14 are usable. Its really that easy to figure out. Anything between /24 and /32 is easy to figure out if you can easily do the math in your head like that. You can do the oppsite if they give you a /29

/32 = 1
/31 = 2
/30 = 4
/29 = 8

Now there is also a little math formula you can use also.

So back to our question.. If we want to know how many usably IPs in a /28. We would go like

2^(32-28) - 2

Which would be

(2^4) - 2 = 14

That broken down is like saying

(2*2*2*2) - 2 = 14

When you toss exponents in it gets a bit rough but its easier if they tell you.. So how many in a /20

(2^(32-20)) = 4096

We won’t worry about usable in this case because it depends how you will subnet it off.