Mapping /proc/meminfo to output of free command
So an example output of free is
total used free shared buffers cached
Mem: 12333980 12151544 182436 0 233128 11197680
-/+ buffers/cache: 720820 11613160
Swap: 487416 3536 483880And /proc/meminfo
MemTotal: 12333980 kB
MemFree: 182436 kB
Buffers: 233128 kB
Cached: 11197680 kB
SwapCached: 996 kB
Active: 7296040 kB
Inactive: 4231216 kB
Active(anon): 73848 kB
Inactive(anon): 22892 kB
Active(file): 7222192 kB
Inactive(file): 4208324 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 487416 kB
SwapFree: 483880 kB
Dirty: 268 kB
Writeback: 0 kB
AnonPages: 95640 kB
Mapped: 7686976 kB
Shmem: 292 kB
Slab: 346296 kB
SReclaimable: 335352 kB
SUnreclaim: 10944 kB
KernelStack: 1920 kB
PageTables: 23844 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 6654404 kB
Committed_AS: 613716 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 309052 kB
VmallocChunk: 34359422748 kB
HardwareCorrupted: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 10240 kB
DirectMap2M: 12572672 kBI’ve done my best to match up the two outputs via colors. Below you will find the math used if there is any.
Total Memory
No math needed.. follow the colors
Used Memory
So for this we want to find the math used to get the used memory on the +/- buffers line. This is the true RAM used by the system.
So this math is based on output of the cat /proc/meminfo
MemTotal - (Buffers + Cached + MemFree)
So our example would be
12333980 - (182352 + 233128 + 11197680) = 720820
Free Memory
So for this we want to find the math used to get the free memory on the +/- buffers line. This is the true RAM that can be used by applications that request it. The operating system will remove data from here to give to applications on demand.
So this math is based on output of the cat /proc/meminfo
Cached + Buffers + MemFree
So our example would be
182352 + 233128 + 11197680 = 11613160
Total Swap
No math needed.. follow the colors
Used Swap
So to find out what free is outputting is pretty simple just do the math from the meminfo output
SwapTotal - SwapFree
So that would look like
487416 - 483880 = 3536
Free Swap
No math needed.. follow the colors
Varnish ACL with X-Forwarded-For Header
So I did a setup like this once
nginx -> varnish -> backend apaches
I did the nginx in front of varnish to handle SSL termination since varnish doesn’t do SSL. So the issue is you can do this for subnet checking in your varnish config
acl vpn {
"192.168.0.0"/16;
}sub vcl_recv {
if (client.ip ~ vpn) {
# something here
}
return(pass);
}So the issue with this is varnish thinks the client.ip is 127.0.0.1 which is correct since the connection is coming from nginx. If varnish was out in front of nginx we wouldn’t have this problem and the example above would just work. So you might be thinking why not just replace client.ip with something like req.http.x-forwarded-for and be done with it all. Well that is a string in varnish and client.ip is an object I believe so you can’t do that. So we have to do some C hacking in the config to get around this.
I found the following example a bit ago on another blog in the comments and if I remember I’ll give credit.
So we need to add the following to make it work
C{
#include <netinet/in.h>
#include <string.h>
#include <sys/socket.h>
#include <arpa/inet.h>
}C
acl vpn {
"192.168.0.0"/16;
}
sub vcl_recv {
C{
//
// This is a hack from Igor Gariev (gariev hotmail com):
// Copy IP address from "X-Forwarded-For" header
// into Varnish's client_ip structure.
// This works with Varnish 3.0.1; test with other versions
//
// Trusted "X-Forwarded-For" header is a must!
// No commas are allowed. If your load balancer something other
// than a single IP, then use a regsub() to fix it.
//
struct sockaddr_storage *client_ip_ss = VRT_r_client_ip(sp);
struct sockaddr_in *client_ip_si = (struct sockaddr_in *) client_ip_ss;
struct in_addr *client_ip_ia = &(client_ip_si->sin_addr);
char *xff_ip = VRT_GetHdr(sp, HDR_REQ, "\020X-Forwarded-For:");
if (xff_ip != NULL) {
// Copy the ip address into the struct's sin_addr.
inet_pton(AF_INET, xff_ip, client_ip_ia);
}
}C
if (client.ip ~ vpn) {
# do something here
}
return(pass);
}Now client.ip is set with the value of x-forwarded-for
Postfix ignore catchall relayhost
Postfix has an option where you can setup a relayhost to send all mail to another mail server. It looks something like
relayhost = [mail.domain.com]:25
So that will send all mail being sent out from the server to mail.domain.com on port 25. So what happens if you want to send mail that is sent to company.com out that server and not through another relay. You would do the following in main.cf
transport_maps = hash:/etc/postfix/transport
Then your /etc/postfix/transport would look something like
company.com :
* smtp:[mail.domain.com]:25Then you want to run
postmap /etc/postfix/transport
Then reload postfix
service postfix reload
Now all mail for company.com will send out that server while all other mail will go through mail.domain.com:25
ps aux with line wrap
File this under you learn something new every day.
Ever do a ps aux only to get something like this to happen?
xymon 26931 0.0 0.0 38944 1396 ? S 05:44 0:00 sh -c vmstat 300 2 1>/usr/lib64/xymon/client/tmp/hob
So the output of ps aux gets cut off when it hits the end of your terminal. And you have to expand your width a lot to see it. It always bothered me till I decided to read the man page.
Now I learned you can do
ps aux --width 1000
That will line wrap your output but cut the output at 1000 characters. You can also pass w’s into the command like
ps auxww
That will ignore your terminals width settings
So now we can add the following in our .bashrc or for system wide /etc/profile so we never have to remember to add a ww or –width again
alias ps='ps ww'
So that alias means you can type ps aux and it will be like your typed ps auxww
Fabric Shell
I recently wrote a quick little fabric function to create a fabric shell. This works with a stock fabric install, you just might have to change where you import it from. It is at my Github page below.
https://github.com/mzupan/fabric_shell
If you place shell.py in the same directory as fabfile.py you can use it like
from fabric.api import *
from fabric.decorators import hosts
import shell
env.hosts = [
'host1.domain.com',
'host2.domain.com'
]
@hosts('')
def cmd():
shell.shell()Then you can use like like
$ fab cmd
[] Executing task 'cmd'
fabric::> id
[host1.domain.com] run: id
[host1.domain.com] out: uid=1020(mzupan) gid=1020(mzupan) groups=1020(mzupan)
[host1.domain.com] out:
[host2.domain.com] run: id
[host2.domain.com] out: uid=1020(mzupan) gid=1020(mzupan) groups=1020(mzupan)
[host2.domain.com] out:
fabric::> .sudo id
[host1.domain.com] sudo: id
[host1.domain.com] out: uid=0(root) gid=0(root) groups=0(root)
[host1.domain.com] out:
[host2.domain.com] sudo: id
[host2.domain.com] out: uid=0(root) gid=0(root) groups=0(root)
[host2.domain.com] out:
fabric::> Right now any command that is issued in the fabric shell gets run on all hosts in env.hosts and if you add .sudo in front of the command it will run them as root.
Fun with Django ManyToMany Fields
This is just another reason I find Django so easy.
For example.. Say you are making a monitoring application. Now you have a bunch of servers and you have certain groups of servers. A server can belong in more then one group also. So that is pretty easy and all with Django.
Now in your template you have a few pages. One page that displays the group information like all the servers in the group and another that displays the server information and will list all the groups that server is in. So below is my basic model layout
class Group(models.Model):
parent = models.ForeignKey("Group", null=True, blank=True)
user = models.ForeignKey(User)
name = models.CharField(max_length=100)
about = models.TextField(blank=True, null=True)
created = models.DateTimeField(auto_now_add=True)
updated = models.DateTimeField(auto_now=True)
def __unicode__(self):
return self.name
class Server(models.Model):
hostname = models.CharField(max_length=300, db_index=True)
about = models.TextField(blank=True, null=True)
created = models.DateTimeField(auto_now_add=True)
updated = models.DateTimeField(auto_now=True)
groups = models.ManyToManyField(Group)
def __unicode__(self):
return self.hostnameSo to get all the groups my server is a member of is pretty easy.
s = Server.objects.get(id=1)
s.groups.all()Now to get all the servers in a group is also easy
g = Group.objects.get(id=1)
g.server_set.all()Crontab with a random sleep
Recently ran into this problem. We had a crontab entry that looked something like this
0 0 * * * /usr/bin/sleep $((RANDOM\%90)); cmdHere
What that attempted was to run a command a random amount of seconds once midnight hit. Now running the command on console causes it to error out
$ /usr/bin/sleep $((RANDOM\%90))
-bash: RANDOM\%90: syntax error: invalid arithmetic operator (error token is "\%90")So what you might be saying is well the \ is messing it up so remove it. Well the problem is % is a reserved character in crontab land. It pretty much means ignore everything after it so you escape it. But its not working as expected. If you remove the \ from the shell command it works just fine.
So the real answer here was use expr. Once that was in all the crontab errors went away
0 0 * * * /bin/sleep `/usr/bin/expr $RANDOM \% 90`; cmdHere
Atomic Symlink Changes
A common deploy strategy for a lot of web sites is as follows
v1.0.10
v1.0.11
v1.1.0
current -> v1.1.0So in this directory we have three directories and one symlink. The directories contain our web application and are tagged with the source control version they were tagged with by the developers. Now its a common practice when deploying a new version to upload the new release which will be v1.1.1 and then just change the symlink to point to the new location. That really isn’t a great idea.
Sure it will work for 99.9% of your traffic but if you have a busy website it could really effect users as your do the upgrade. You are probably saying well I’m forcing the new symlink with the following command
ln -snf v1.1.1 current
Did you ever look at the system calls using the -f flag for ln does? If not lets take a look
# strace ln -snf v1.1.1 current 2>&1 | grep link
symlink("v1.1.1", "current") = -1 EEXIST (File exists)
unlink("current") = 0
symlink("v1.1.1", "current") = 0So you can see it first tries to create the link and finds the symlink is there so it unlinks curent and makes the new symlink. The man page even says this is the case!
-f If the target file already exists, then unlink it so that the link may occur. (The -f option overrides any previous -i options.)
So how do we make this more atomic so that our users do not even notice and everyone is happy? Well its very simple.
ln -s v1.1.1 current.new && mv -Tf current.new current
That will do an atomic update. If you want to learn more about the -T flag you can find it here
It might not be 100% atomic but it’s really the best I can get right now. POSIX says its atomic so right now I’ll trust it.
Tav’s Fabric Fork Quick Start
I’ve been a big fan of the fabric project for a long time now. It has made my life very easy in the past and gets better with each new release. Well a few weeks ago I got introduced to a fork of the project by Tav. It provides many new feature additions to fabric and makes fabric a little bit better. So the best way to get it started is via the following
git clone git://github.com/tav/pylibs.git
cd pylibs
python setup.py
export PYTHONPATH=$PYTHONPATH:`pwd`Once installed, you have to create a fab app
vi /usr/bin/fab
Then you want to add the following contents to it
#!/usr/bin/env python
from fabric.main import main
main()Then you want to give it the correct permissions
chmod +x /usr/bin/fab
Now create a directory and place a fabfile.py file in it.
Now create a config.yml file and use the following as a simple template
default:
shell: /bin/bash -l -c
servers:
directory: /dir
hosts:
- host1.domain.com
- host2.domain.comThen your fab file could look something like
from fabric.api import *
env.config_file = 'config.yml'
env.user = 'mzupan'
@task('servers')
def command():
env().multirun('uname -a')Normally fabric runs in serial, but Tav’s port adds a multirun() function that runs the commands in parallel. BONUS!
Feel free to play around with it and check out his blog post on it to get more commands.
http://tav.espians.com/fabric-python-with-cleaner-api-and-parallel-deployment-support.html
Senior Linux Admin Interview Question #2
You can see all the questions I have compiled here.
Question
You are already logged into a server and a user runs a fork bomb script. How can you stop the fork bomb without restarting the server or bringing any services offline that are currently running?
You also know the name of the running fork bomb script.
Reason for the this question
This will tell you how well the canidate knows about the signals Linux uses with processes. I would think most people would say the easy thing
killall -9 scriptName
This will not work due the nature of a fork bomb. The reason is the killall does not hold a lock on the process table so each one that is killed a new one takes its place in the process table.
Also you will not be able to run a killall due to the shell forking off another shell to run the killall
This question also tells you if the admin knows about Linux internals. That just running a killall will fork a new process buy running exec killall will run the killall in the current process and not fork out a new one
Answer
So for this I will use a fork bomb script that is written in C. Below is an example
#include
int main(void)
{
for(;;)
fork();
return 0;
} We can compile it like
gcc -o fork fork.c
So the way to stop it is sending a SIGSTOP signal to each fork bomb process and once they are all stopped you can send a SIGKILL to each process.
So if you run a
killall -STOP fork
You will get a error message like resources not avaliable so you cannot run that. You can run it with exec.
exec killall -STOP fork
exec killall -9 forkThat will stop the fork bomb.
