Compile and install python with mysql for users

To run custom python version for a useraccount:

download the latest version of python

$ wget http://www.python.org/ftp/python/2.6.5/Python-2.6.5.tgz
$ tar xvzf Python-2.6.5.tgz
$ cd Python-2.6.5
$ ./configure --prefix=/home/username/python-2.6.5
$ make
$ make install

Install setuptools
as root:

# ln -s /home/username/python-2.6.5/bin/python2.6 /usr/bin/ *this is required for setuptools

as normal user:

$ wget http://pypi.python.org/packages/source/s/setuptools/setuptools-0.6c11.tar.gz#md5=7df2a529a074f613b509fb44feefe74e
$ tar xvzf setuptools-0.6c11.tar.gz
$ sh setuptools-0.6c11-py2.6.egg --prefix=~/python-2.6.5/

download mysql-python

$ wget http://downloads.sourceforge.net/project/mysql-python/mysql-python-test/1.2.3c1/MySQL-python-1.2.3c1.tar.gz?use_mirror=citylan
$ tar xvzf MySQL-python-1.2.3c1.tar.gz
$ cd MySQL-python-1.2.3c1
$ /home/username/python-2.6.5/bin/python setup.py build
$ /home/username/python-2.6.5/bin/python setup.py install

Done:

sh-3.00$ python2.6
Python 2.6.5 (r265:79063, May 23 2010, 14:40:28)
>>> import MySQLdb
>>>

./arun

KVM image on LVM

Convert qcow2/raw images to LVM logical volume to use with KVM:

– Convert the qcow2 image to raw format (if it is in qcow2)
$ qemu-img convert image.qcow2 -O raw image.raw

– Create the physical volume for LVM
# pvcreate /dev/sdb
(replace the device with correspond to the system)

– Create the volume group
# vgcreate pool1 /dev/sdb
(replace pool1 with the name as required)

– Create Logical volume with same size as the image
# lvcreate -n justaname --size 50G pool1
(replace justaname and size as per the requirements)
Use lvresize incase you required the change the volume size

– dd the raw image to lvm logical volume
# dd if=image.raw of=/dev/pool1/justaname bs=8M
(Change the block size according to the requirements.

Edit the kvm xml configuration for the corresponding virutal machine to use the logical volume

< disk type='block' device='disk' >
< source dev='/dev/pool1/justaname'/ >
< /code >

./arun

Virtualization with KVM under Redhat Linux, Migrate VMware virtual images to KVM

KVM (Kernel Based Virtual Machine) – http://www.linux-kvm.org/ , is one of the best choice to do virtualization under linux, and especially without extra licensing cost.

Install KVM
To install KVM on redhat enterprise linux:
– Install the machine with 64 bit version of EL5
– Register the machine with redhat (rhn_register)
– enable virtualization entitlement for the system in RHN
– Install KVM package:
# yum install kvm
# yum install virt-manager libvirt libvirt-python python-virtinst

Migration VMware virtual machines to KVM:
– Login to the vmware server
– make single vmdk image with vmware-diskmanager
eg:
# vmware-vdiskmanager -r path_to_vmware_virtualmachine.vmdk -t 0 destination_file_vmware.vmdk
Creating disk ‘destination_file_vmware.vmdk’
Convert: 100% done.
Virtual disk conversion successful.

– Copy the image to KVM server
– Convert the image to KVM supported format with qemu-img
# qemu-img convert destination_file_vmware.vmdk -O qcow2 kvm_supported.img

Create bridge interface to to share the network card.
* This section assumes that you have two nic in your server and would need to have bonding along with bridging and you have static ip required for virtual machines. incase you using dhcp and single network interface create the bridge interface accordingly.

– Create bridge interface:
$ cat /etc/sysconfig/network-scripts/ifcfg-br0

DEVICE=br0
ONBOOT=yes
TYPE=Bridge
IPADDR=11.11.11.11
NETMASK=255.0.0.0
GATEWAY=1.1.1.1

– Configure the bond interface:
$ cat /etc/sysconfig/network-scripts/ifcfg-bond0

DEVICE=bond0
BRIDGE=br0
ONBOOT=yes

– Configure eth0 and eth1
$ cat /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0
MASTER=bond0
SLAVE=yes
ONBOOT=yes

– Change bonding to active-backup , i have faced some issues with xor – might be silly to fix
# cat /etc/modprobe.conf

options bond0 miimon=100 mode=active-backup

– Restart network interface and check the bridge status
# brctl show , it will show bond0 as an enabled interface.

Create KVM virtual machine:
– it can be done from the command line or with virt-manager
– open virt-manager application
– click create new, and select qemu hypervisor
– during disk selection, choose the converted vmware image path
– done, just start it.

Register the virtual machine with Redhat, save some license 😉

– enabled network tools entitlement in RHN
– install the package rhn-virtualization-host on the core machine
# yum install rhn-virtualization-host
– enable virtualization under the properties of host in RHN
– execute the following commands on host machine
# rhn_check
# rhn-profile-sync
– login to virtual machine and use rhn_register, now it will be registered as a virtual machine under the core license.

./arun

Enable Full virtualization in HP DL servers (Intel)

You need to enable hardware virtualization in BIOS if you want to create Fully virtualized instances.

Enter BIOS (F9) –> Advanced Options –> Processor Options –> Enable intel Virtualization Technology

Now you should be able to create Fully virtualized virtual machines from XEN or similar virtualization packages without OS modifications.

./arun

Issues with zone transfer in Dual stack IPv4 / IPv6

You might face issues with zone transfer to ipv4 secondaries on a dual stack server where the bind listening on IPv4 and IPv6 address,

client ::ffff:11.11.11.11#43253: zone transfer 'example.com/IN' denied

this happens because , once the v6 is enabled on bind it just try to make ipv4 address looks like v6 address.

Solution : just add the v6 formatted v4 address to the allowed list

allow transfer { ::ffff:11.11.11.11; };

./arun

Configure Apache over IPv6

Once your network interface is configured with IPv6, it is easy to configure the webserver. No real difference with IPv4 configuration.

Configure Apache to listen the IPv6 address:

Listen ipv6_address:port
NameVirtualHost ipv6_address:port

If the apache virtual host is configured with domain name , eg < VirtualHost arunns.com:80 >, just add AAAA record for arunns.com in dns and the website will work without any extra configurations other than the previous two lines.

Also we can specifically configure it :

< VirtualHost ipv4_address:80 ipv6_address:80 >

It is possible to have different contents for ipv4 and ipv6 sites, just create two different virtual hosts with different document root one for IPv4 and other for IPv6.

< VirtualHost ipv4_address:80 >
DocumentRoot /home/123/
< /VirtualHost >
< VirtualHost ipv6_address:80 >
DocumentRoot /home/456/
< /VirtualHost >

./arun

IPV6 Tunnel from MAC/Linux

It is really easy to establish an ipv6 network tunnel from your machine directly. Make your network/system/services IPv6 ready 🙂

Create a ipv6 regular tunnel from any connection brokers: List of IPV6 tunnel brokers

I have used Hurricane Electric which is free tunnel broker.
Tunnel Broker

With the tunnel broker, you can create a tunnel by specifying your public ipv4 address in their website.

Once the tunnel is created with tunnel broker, Configure your machine with required interfaces , tunnel and routing.

    For MAC OS X:

1) Configure tunnel
$ sudo ifconfig gif0 tunnel host_ip tunnel_broker_ipv4_ip

If you are behind a natd network specify your machine private address as host_ip, otherwise mention the current public ip assigned to your machine. If you are behind a nat’d network make sure that protocol 41 is allowed in the nat’d device.
eg:
$ sudo ifconfig gif0 tunnel 192.168.1.2 216.66.xxx.xxx

2) Setup the tunnel end points

$ sudo ifconfig gif0 inet6 host_ipv6_address tunnel_broker_ipv6_address prefixlen 128

Both these ipv6 addresses are assigned by the tunnel broker.
eg:
$ sudo ifconfig gif0 inet6 2001:470:xxxx:xxxx::2 2001:470:xxxx:xxxx::1 prefixlen 128

3) Add the default route for ipv6 traffic
$ sudo route -n add -inet6 default tunnel_broker_ipv6_address

eg:
$ sudo route -n add -inet6 default 2001:470:xxxx:xxxx::1

Now you should be able to access the ipv6 networks 🙂

Incase of any issues, just make sure that ipv6 is enabled on the interface using:

$ sudo ip6 -x gif0

Test your ip6 connectivity:


$ ping6 ipv6.google.com
$ telnet ipv6.google.com 80

    For Linux:

The procedure is exactly same on linux as well:

Make sure that the ipv6 module is present in the kernel:

$ sudo modprobe ipv6

Create the tunnel
$ sudo ip tunnel add he-ipv6 mode sit remote 216.66.xx.xx local 192.168.1.2 ttl 255
* use the public ip if it is directly assigned to your machine

Activate the tunnel
$ sudo ip link set he-ipv6 up

Assign ip address to interface:

$ sudo ip addr add 2001:470:xxxx:xxxx::2/64 dev he-ipv6

Add default route for ipv6:

$ sudo ip route add ::/0 dev he-ipv6

Add protocol family identifer:

$ sudo ip -f inet6 addr

./arun

IPv6 and Linux

It is straight forward to enable IPv6 on any linux system, since the latest kernel support it very well. This document is more relevant for Redhat linux but the idea is same for all.

Make sure the ipv6 support is not disabled in kernel
Comment out the following line in /etc/modprobe.conf if existing.

#alias ipv6 off
#alias net-pf-10 off

Enable IPv6 networking:
edit /etc/sysconfig/network

NETWORKING_IPV6=yes

Configure the IPv6 address:
edit /etc/sysconfig/network-scripts/ifcfg-eth0 (or bond0 for bond interfaces, ipv6 works as expected with bond interface as well)

IPV6INIT=yes
IPV6ADDR=
IPV6ADDR_SECONDARIES=
IPV6_DEFAULTGW=
IPV6_AUTOCONF=yes/no

Just restart network and you will be able to see the IPv6 address.

Incase if the ipv6 module doesnt exist in kernel, just do a modprobe:
# modprobe -a ipv6

Almost all softwares in linux works with IPv6,

For apache add the listen address to ipv6 address and enable name virtual host for ipv6 address if required.

You can test your ipv6 connectivity by:
$ ping6 ipv6.google.com

./arun

svn+ssh with custom port number and public key authentication

To make custom configurations for svn+ssh:

Edit ~/.subversion/config

– add the ssh configuration details under [tunnels]
like:
foobar = /usr/bin/ssh -i /home/foo/.ssh/foobar.private -p 12345

Now use:
svn co svn+foobar://user@svn.test.com/home/test/repos/foobar

create svn repository and initial check in

To create svn repository login to the svn server:

$ sudo -u svnuser svnadmin create --fs-type fsfs /path/to/repository
* we can use bdb as well as db format

To make all the group members privilege to write access the repository:

$chmod g+w /path/to/repository

and add the user to svn group.

To create initial contents:
either you can check out the repository and create the file structure like:

[local_machine]$ svn co svn+ssh://user@svnhost/path/to/repos localdirectory
[local_machine]$ mkdir -p localdirectory/trunk localdirectory/tags localdirectory/branches
[local_machine]$ cd localdirectory; svn commit -m "initial repository structure"

or you can do the same from the svn server itself using file:///

Mysql one way DB replication

One way replication of mysql database:

Mysql replication help us in keeping the data replicated to one or more sites reliably with binary logs. Apart from good amount of advantages Mysql replication doesn’t help with data corruption, since the corrupted data is replicated in all slaves. It is good to have periodic backup of database apart from replication.

Replication Steps

– Create database with same name on all servers
> mysql -u db_user -p -e "CREATE DATABASE db_name"

– Create database user with replication privilege on master
> GRANT REPLICATION SLAVE ON *.* TO 'replication_user'@'replication_clients" IDENTIFIED BY 'replication_password'
This can be also supplied with particular database name with ;db_name.*’ instead of *.*

– Edit Mysql master configuration (my.cnf) to allow replication
[mysqld]
server-id = 1 # Important with replication
log-slave-updates
log-bin = /var/lib/mysql/mysql-bin
log-bin-index = /var/lib/mysql/mysql-bin.index
replicate-do-db = db_name # specify the dbs to replicate
log-warnings
innodb_flush_log_at_trx_commit=1
sync-binlog=1
innodb_safe_binlog

– Take dump of master db and put them on all replicas

use db_name;
FLUSH TABLES WITH READ LOCK;

$ mysqldump -u dbuser -p db_name > db_dump.sql
install on slaves
$ mysql -u dbuser -p db_name < db_dump.sql


use db_name;
UNLOCK TABLES;

– Edit mysql configuration on replicas with master credentials

[mysqld]
old_passwords=1
server-id=2
innodb_file_per_table
log-slave-updates
master-host = master_hostname
master-port = master_port
master-user = master_user
master-password = master_password
log-bin = /var/lib/mysql/mysql-bin
log-bin-index = /var/lib/mysql/mysql-bin.index

Restart the Mysql daemon on all servers and check the replication status:
Master: > show master status;
Replicas: > show slave status;

Hardening Apache webserver

Tips to harden apache webserver:

Disable weak SSLV2 siphers
edit ssl.conf and add
SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:!LOW:!SSLv2:+EXP

Restrict apache to giveout minimum informations
Edit httpd.conf and change
ServerTokens ProductOnly

Disable track and trace in every virtual hosts
LoadModule rewrite_module modules/mod_rewrite.so
RewriteEngine On
RewriteCond %{REQUEST_METHOD} ^TRACE
RewriteRule .* - [F]

Always better to configure apache to Loan minimal/required modules and include only necessary config files.

./arun

Mysql backup script

To backup mysql on a daily/hourly basis with time stamp and compress it after backup also it will remove the files older than x days.

#!/bin/bash
# Arun N S
# variables
DATE="$(date +"%d-%m-%Y")"
TIME="$(date +"%d-%m-%Y-%H%M")"
USER=username
PASSWORD=password
DATABASE=dbname


# Directories and dump
/bin/mkdir -p /backup/Mysql/$DATE
/usr/bin/mysqldump -l -F -u $USER --password=$PASSWORD $DATABASE > /backup/Mysql/$DATE/backup_$TIME.sql


# Compressing
/usr/bin/bzip2 /backup/Mysql/*/*.sql


#Removing files older than x days eg: 90 days
for i in `/usr/bin/find /backup/Mysql/ -maxdepth 1 -type d -mtime +90 -print`; do
/bin/echo -e "Deleting old directories $i"; /bin/rm -rf $i; done

Fix for – – mixing * ports and non-* ports with a NameVirtualHost address is not supported

If you happend to see this error in apache logs,

– mixing * ports and non-* ports with a NameVirtualHost address is not supported, proceeding with undefined results

Check your apache configuration, to make sure that there is no VirtualHost defined without port numbers like : <VirtualHost 127.0.0.1>, if existing fix it with port number.

Mysql Queries and Tips

Some useful database queries:

login to mysql database:

$ mysql $database_name -u $user_name -h $host_name -p

Take a dump by locking transactions

$ mysqldump -l -F $database_name -u $user_name -h $host_name -p > file.sql
*need lock table and read privilege from the host you trying

Take backup of only some tables
$ mysqldump -l -F $database_name -u $user_name -h $host_name --tables $tables_name -p > file.sql

Take backup of only database structure , without data
$ mysqldump -l -F -d $database_name> -u $username -h $hostname -p > file.sql

List the permissions assigned for a user
login to mysql:
> show grants for 'user'@'hostname';

Find Tips

Copy/Move files with find

find <path> -name "filename" -exec cp -prf {} /destination/{} ;
find /var/log/ -name "m*" -exec cp -prf {} /tmp/message/{} ;

This will create the same directory structure under /tmp/message, incase you want all subdirectory files under /tmp/message/ remove the {} .

Remove files older than certain days (using find/mtime)
find -name "" -mtime +N -exec rm -r {} ;

Eg : find /var/log/ -name "*.log" -mtime +5 -exec rm -r {} ;
This will remove the *.log files older than 5 days in directory /var/log/

Find with file type

directories : find / -type d -print0
files: find / -type f -print0

Remove held messages from mailman queue

If you have too many mails/spams tend to pending moderator requests in mailman queue it can be removed by:

$ cd ~mailman
$ bin/discard /var/lib/mailman/data/heldmsg--*

And if the no. of argument list exceeds you can try:

$ find /var/lib/mailman/data -name heldmsg--* -print | xargs bin/discard

* the path of mailman depends on your implementation anyway.