grep only stderr from a command

Posted by on May 6, 2013 · Leave a Comment 

So sometimes you have a command where you want to only grep stderr. For example I use Cronic to manage all my cronjobs. It’s really nice since it sends a nicely formatted email back to you if a command returns anything in stderr. Crontab alone will email me if anything in stdout/stderr is printed out from a command.

There is a problem with cronic though. There’s an app called s3cmd which uploads files to s3 and on large files this output can happen


WARNING: Retrying failed request: /postgresql/2013-05-05_hour-18.sql.bz2?partNumber=1&uploadId=m0gX4xcOU7IDla2B2p55xJXDfih_mm7rDx5bJvucUAmQYC10mwoHXVDjyoat_uzNJBYpedrWu7neakUpH3zGw-- ([Errno 110] Connection timed out)
WARNING: Waiting 3 sec...

This generally happens if the fule is very large. s3cmd will restart the transfer of the part and it will be uploaded just fine. So the main issue is s3cmd has no way to ignore warnings, at least as of 1.5.0-alpha2. If it does, I am overlooking it.

So in order to solve this I have decided I want to grep out the WARNING lines from stderr. Now the easy way to do this is re-direct stderr to stdout and pipe it to grep. Well this sucks cause you will lose valid error output. So the answer is something like this

cmd 2> >(grep -v "WARNING" >&2)

That will then allow you to grep just from stderr and the -v flag will set grep to ignore WARNING lines.

Filed under Sys Admin · Tagged with

Remove internal hosts in postfix

Posted by on April 15, 2013 · Leave a Comment 

A proper way to setup your network is to have a postfix relay server sitting somewhere on your network that every other server has access to via port 25. Now your relay server(s) are the only ones that should have outside access to port 25. All other servers should be firewalled off! 

So for example say your email chain looks like this for a new signup

web1.domain.com -> mailserver.domain.com -> internet -> user's mailbox

If the user views the email source they will see that the email started at web1.domain.com and it will include your internal IP address. So you are exposing internal IP addresses which isn’t very good at all.

So we can fix this in postfix very easily on the mailserver.domain.com config. For example say your internal network is

10.114.0.0/16

So lets remove them everything in that subnet along with 127.0.0.1. So edit the following file

/etc/postfix/main.cf

Then add the following line

header_checks = regexp:/etc/postfix/header_checks

Now create a new file

/etc/postfix/header_checks

Then add the following in place

/^Received:.*\[127\.0\.0\.1/ IGNORE
/^Received:.*\[10\.114\..*/ IGNORE

Then restart postfix and you are good to go.

Filed under Sys Admin · Tagged with

How many bits per point in a whisper file in graphite

Posted by on October 19, 2012 · Leave a Comment 

This is an easy one. You do an ls over a whisper file and get something like

1.1M -rwxr-xr-x  1 root root 1.1M Aug  9 11:06 Active.wsp

Now you are planning to save more metrics in your graphite server or want more points in your retention so you want to plan how much space you’ll need given an estimated guess of your metric count out 6 months from now. So you can run the following over that whisper file to get info on it.

$ whisper-info.py Active.wsp
maxRetention: 31536000
xFilesFactor: 0.5
aggregationMethod: average
fileSize: 1062772

Archive 0
retention: 604800
secondsPerPoint: 60
points: 10080
size: 120960
offset: 52

So you see for Archive 0 we have a certain size and number of points. So from here it’s easy math

120960 / 10080 = 12

So for each whisper data point you save it will allocate 12 bits on your filesystem. So you can plan our growth since carbon pre-allocates the space needed for whisper files so your disk isn’t doing any random seeks while pulling up data for your graphs.

 

Filed under graphite, Sys Admin · Tagged with

Google Authenticator with OpenVPN for 2 factor auth

Posted by on October 9, 2012 · 7 Comments 

This post will describe how to get it working with Ubuntu 12.04. It should work with other versions along with other Linux distros. The only difference is the package names might change.

Install the packages

Run the following command

apt-get -y install openvpn libpam-google-authenticator

Setup OpenVPN

Place the cert key helps

mkdir /etc/openvpn/easy-rsa/
cp -R /usr/share/doc/openvpn/examples/easy-rsa/2.0/* /etc/openvpn/easy-rsa/

Then you need to edit the following file

/etc/openvpn/easy-rsa/vars

Scroll all the way down and find the following section and change it to your needs

export KEY_COUNTRY=US
export KEY_PROVINCE=NY
export KEY_CITY=NewYork
export KEY_ORG=”Company VPN”
export KEY_EMAIL=”sysops@company.com”

Set the permissions

cd /etc/openvpn/easy-rsa/
sudo chown -R root:admin .
sudo chmod g+w .

Source the vars file to pull in needed environment variables

source ./vars

Run the scripts to build your keys

./clean-all
./build-dh
./pkitool –initca
./pkitool –server server
cd keys
openvpn –genkey –secret ta.key
sudo cp server.crt server.key ca.crt dh1024.pem ta.key ../../

Create the following file

/etc/openvpn/up.sh

With the contents of

#!/bin/sh

BR=$1
DEV=$2
MTU=$3

/sbin/ip link set “$DEV” up promisc on mtu “$MTU”
/sbin/brctl addif $BR $DEV

Then create another file called

/etc/openvpn/down.sh

With the contents of

#!/bin/sh

BR=$1
DEV=$2

/sbin/brctl delif $BR $DEV
/sbin/ip link set “$DEV” down

Now set permissions on this files

chmod +x /etc/openvpn/up.sh /etc/openvpn/down.sh

Then you want to add the following to /etc/sysctl.conf

net.ipv4.ip_forward = 1

That will set it on reboot but you want to set it now run the following

echo 1 > /proc/sys/net/ipv4/ip_forward

Now setup pam to work for openvpn. Create the following file

/etc/pam.d/openvpn

Then put the following in

auth requisite pam_google_authenticator.so forward_pass
auth required pam_unix.so use_first_pass

Now you need to edit your openvpn config. I called mine server.conf in /etc/openvpn and you want to add the following

plugin /usr/lib/openvpn/openvpn-auth-pam.so openvpn

Now once openvpn is started you can run the following as the user you are setting up VPN for. They will need a shell account and they can do this themselves if needed. So run the following command

google-authenticator

Then choose the following options in this order.

Do you want authentication tokens to be time-based (y/n) y
Do you want me to update your “/home/bla/.google_authenticator” file (y/n) y

Do you want to disallow multiple uses of the same authentication
token? This restricts you to one login about every 30s, but it increases
your chances to notice or even prevent man-in-the-middle attacks (y/n) y

By default, tokens are good for 30 seconds and in order to compensate for
possible time-skew between the client and the server, we allow an extra
token before and after the current time. If you experience problems with poor
time synchronization, you can increase the window from its default
size of 1:30min to about 4min. Do you want to do so (y/n) y

If the computer that you are logging into isn’t hardened against brute-force
login attempts, you can enable rate-limiting for the authentication module.
By default, this limits attackers to no more than 3 login attempts every 30s.
Do you want to enable rate-limiting (y/n) y

You notice this command might show a QR code at the top which you can download the Google Authenticator app on your phone to scan that code and it will setup your account automatically. If it doesn’t show a QR code it will show a link that will display the code.

You can now run openvpn and connect and the username is your shell user and if you set the password to test123 and google gives you a code of 847324, your password would be test123847324

 

Filed under Sys Admin · Tagged with

Simple script to selectively remove emails from postfix’s queue

Posted by on June 20, 2012 · Leave a Comment 

If you send out a lot of emails, the queue in postfix might be full of junk emails from people putting like bla@example.net as their email address. Now there could be other valid emails in there that you don’t want to remove due to their mail server down and postfix keeps trying for a set amount of time to re-send that email. So below is a simple script that you have to run as root that you can enter a search string as a paramater to remove emails from the queue

Filed under Python, Sys Admin · Tagged with

Nagios check for monit

Posted by on June 11, 2012 · Leave a Comment 

Here is a simple nagios check I just created to check the monitor status for monit. The background is as simple as this.. If you have a service that is flapping and monit is set to timeout after a number of failed attempts to start the service, it will un-monitor the service. Without m/monit running, you might never notice it is set as un-monitored. So I created a little python script to check for this

Filed under Python, Sys Admin · Tagged with

Mapping /proc/meminfo to output of free command

Posted by on May 29, 2012 · Leave a Comment 

So an example output of free is

                             total             used             free                shared    buffers     cached
Mem:                     12333980   12151544    182436          0            233128    11197680
-/+ buffers/cache:                     720820        11613160
Swap:                    487416       3536             483880

And /proc/meminfo

MemTotal:       12333980 kB
MemFree:          182436 kB
Buffers:          233128 kB
Cached:         11197680 kB
SwapCached:          996 kB
Active:          7296040 kB
Inactive:        4231216 kB
Active(anon):      73848 kB
Inactive(anon):    22892 kB
Active(file):    7222192 kB
Inactive(file):  4208324 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:        487416 kB
SwapFree:         483880 kB
Dirty:               268 kB
Writeback:             0 kB
AnonPages:         95640 kB
Mapped:          7686976 kB
Shmem:               292 kB
Slab:             346296 kB
SReclaimable:     335352 kB
SUnreclaim:        10944 kB
KernelStack:        1920 kB
PageTables:        23844 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     6654404 kB
Committed_AS:     613716 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      309052 kB
VmallocChunk:   34359422748 kB
HardwareCorrupted:     0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       10240 kB
DirectMap2M:    12572672 kB

I’ve done my best to match up the two outputs via colors. Below you will find the math used if there is any.

Total Memory

No math needed.. follow the colors

Used Memory

So for this we want to find the math used to get the used memory on the +/- buffers line. This is the true RAM used by the system.

So this math is based on output of the cat /proc/meminfo

MemTotal - (Buffers + Cached + MemFree)

So our example would be

12333980 - (182352 + 233128 + 11197680) = 720820

Free Memory

So for this we want to find the math used to get the free memory on the +/- buffers line. This is the true RAM that can be used by applications that request it. The operating system will remove data from here to give to applications on demand.

So this math is based on output of the cat /proc/meminfo

Cached + Buffers + MemFree

So our example would be

182352 + 233128 + 11197680 = 11613160

Total Swap

No math needed.. follow the colors

Used Swap

So to find out what free is outputting is pretty simple just do the math from the meminfo output

SwapTotal - SwapFree

So that would look like

487416 - 483880 = 3536

Free Swap

No math needed.. follow the colors

Filed under Sys Admin · Tagged with

Varnish ACL with X-Forwarded-For Header

Posted by on March 16, 2012 · 4 Comments 

So I did a setup like this once

nginx -> varnish -> backend apaches

I did the nginx in front of varnish to handle SSL termination since varnish doesn’t do SSL. So the issue is you can do this for subnet checking in your varnish config

acl vpn {
  "192.168.0.0"/16;
}
sub vcl_recv {
  if (client.ip ~ vpn) {
    # something here
  }
  return(pass);
}

So the issue with this is varnish thinks the client.ip is 127.0.0.1 which is correct since the connection is coming from nginx. If varnish was out in front of nginx we wouldn’t have this problem and the example above would just work. So you might be thinking why not just replace client.ip with something like req.http.x-forwarded-for and be done with it all. Well that is a string in varnish and client.ip is an object I believe so you can’t do that. So we have to do some C hacking in the config to get around this.

I found the following example a bit ago on another blog in the comments and if I remember I’ll give credit.

So we need to add the following to make it work

C{
#include <netinet/in.h>
#include <string.h>
#include <sys/socket.h>
#include <arpa/inet.h>
}C

acl vpn {
  "192.168.0.0"/16;
}

sub vcl_recv {

C{
//
// This is a hack from Igor Gariev (gariev hotmail com):
// Copy IP address from "X-Forwarded-For" header
// into Varnish's client_ip structure.
// This works with Varnish 3.0.1; test with other versions
//
// Trusted "X-Forwarded-For" header is a must!
// No commas are allowed. If your load balancer something other
// than a single IP, then use a regsub() to fix it.
//
struct sockaddr_storage *client_ip_ss = VRT_r_client_ip(sp);
struct sockaddr_in *client_ip_si = (struct sockaddr_in *) client_ip_ss;
struct in_addr *client_ip_ia = &(client_ip_si->sin_addr);
char *xff_ip = VRT_GetHdr(sp, HDR_REQ, "\020X-Forwarded-For:");

if (xff_ip != NULL) {
// Copy the ip address into the struct's sin_addr.
inet_pton(AF_INET, xff_ip, client_ip_ia);
}
}C

  if (client.ip ~ vpn) {
    # do something here
  }

  return(pass);
}

Now client.ip is set with the value of x-forwarded-for

Filed under Sys Admin · Tagged with , ,

Postfix ignore catchall relayhost

Posted by on October 13, 2011 · Leave a Comment 

Postfix has an option where you can setup a relayhost to send all mail to another mail server. It looks something like

relayhost = [mail.domain.com]:25

So that will send all mail being sent out from the server to mail.domain.com on port 25. So what happens if you want to send mail that is sent to company.com out that server and not through another relay. You would do the following in main.cf

transport_maps = hash:/etc/postfix/transport

Then your /etc/postfix/transport would look something like

company.com :
* smtp:[mail.domain.com]:25

Then you want to run

postmap /etc/postfix/transport

Then reload postfix

service postfix reload

Now all mail for company.com will send out that server while all other mail will go through mail.domain.com:25

Filed under Sys Admin · Tagged with , ,

ps aux with line wrap

Posted by on August 22, 2011 · 2 Comments 

File this under you learn something new every day.

Ever do a ps aux only to get something like this to happen?

xymon 26931 0.0 0.0 38944 1396 ? S 05:44 0:00 sh -c vmstat 300 2 1>/usr/lib64/xymon/client/tmp/hob

So the output of ps aux gets cut off when it hits the end of your terminal. And you have to expand your width a lot to see it. It always bothered me till I decided to read the man page.

Now I learned you can do

ps aux --width 1000

That will line wrap your output but cut the output at 1000 characters. You can also pass w’s into the command like

ps auxww

That will ignore your terminals width settings

So now we can add the following in our .bashrc or for system wide /etc/profile so we never have to remember to add a ww or –width again

alias ps='ps ww'

So that alias means you can type ps aux and it will be like your typed ps auxww

Filed under Sys Admin · Tagged with

Next Page »