I will divide it on the 3 little parts:

1. Structure of puppet modules
2. Modules manifests
3. Files
4. Temlates

1. Structure of puppet modules

Puppet modules represents itself as a standalone directory inside the directory specified by modulepath variable at /etc/puppet/puppet.conf:

modulepath = /etc/puppet/modules

The directory tree has next structure:

my_module
        |- files
        |      |
        |      \- some_config.conf
        |- manifests
        |         |
        |         \- init.pp
        \- templates
                  |
                  \- some_template.erb

The module directory name should be the name of module and should match class name.

So there are three basic directories files, manifests, template.

The main directory is manifests and it should be exists as well as manifest file init.pp inside it.

files – contains configuration or other files which should be shared between other hosts. Like resolv.conf

templates – contains templates.

2. Modules manifests

Let’s look inside init.pp file. It have to contain at least one class with name matching madule name:

class my_module {
}

It also can inherits some other module/class

class my_module inherits other_module  {
}

It can be useful if some resource from third modules should be involved in your module’s manifest.

For example if you change php.ini in php module then after deployment you need to reload httpd daemon to make changes to take effect. Here is the part of httpd modules describes reload:

class httpd {
...
  exec { "httpd-reload":
      command => "/etc/init.d/httpd reload",
      refreshonly => true,
      require => [Package['httpd'], Exec["httpd-test-config"] ],
  }

  exec { "httpd-test-config":
      command => "/usr/sbin/apachectl configtest",
      refreshonly => true,
  }

...
}

and part used in php module:

class php inherits httpd {
...
  file { "/etc/php.ini":
         ensure => present,
         owner  => root,
         group  => root,
         mode   => 644,
         source => "puppet:///php/php.init",
         notify => Service["httpd"],
  }
...
}

The class can have next objects:

• package
• service
• file
• exec

Examples of each of them:

package { package_name:
        ensure => present|latest,
    }

service { "service_name":
    ensure => stopped|running,
    enable => false|true,
}

exec { "exec_name":
    command => "some command to reload",
}

3. Files

There are no any special requirement to share a file between system. To deploy file to systems with activated module create new object in manifest:

file { "/where/to/deploy/filename":
  ensure => link|directory|present|absent,
  target => "/path/to/deployed/file",
  owner  => "root",
  group  => "root",
  mode   => "777",
  recurse => true|false,
  ignore => [".svn", "some_file"], # applied to directories
  source => "puppet:///module_name/file_to_deploy/",
  notify => Service["my_service"],
}

Most interesting parameters are notify and source. First parameter triggers some command my_service(described earlier in this or in inherited class) shortly afterwards file changed. And second parameter is specified path to file at puppet side. Value puppet:///module_name/php.d/ means that file should be located at /path/to/puppet/modules/modules_name/file/file_to_deploy. So if your need to deploy php.ini from php class to /etc/php.ini on client node put file php.ini into /etc/puppet/modules/php/files/php.ini(it’s default module path).

4. Templates

Template allows describe content of file depends on user’s and system’s variables using loops and conditions. To use file in class as well as file – describe it as content instead of source

file { '/test.file':
   content => template("my-modules/test.file.erb"),
}

Here are couple examples of using templates.
/etc/resolv.conf:

define variables at manifest:

$domainname = example.com
$nameservers = ['192.168.0.1', '192.168.0.2']

resolv.conf.erb:

domain
search

nameserver

On client side we’ll get:

domain example.com
nameserver 192.168.0.1
nameserver 192.168.0.2

Example #2 sent different content depends on node name:

   content for node host1
   where fqdm stands for fully qualified domain name

   content for all node who hostname match 'host2*' regexp

Template file should be located at /etc/puppet/modules/my-module/manifests/ directory.

I download siege to make benchmarks:

apt-get install siege

Then prepared list of urls, actually I took it from my sitemap.xml and normalized it in next format:

root@laptop:~# head url.txt

http://andriigrytsenko.net

http://andriigrytsenko.net/2011/07/apache-vs-nginx-vs-lighttpd-and-static-content

http://andriigrytsenko.net/2011/07/ldap-mss-replication

http://andriigrytsenko.net/2011/07/pacemaker-and-apache

Then run siege with/without caching plugin with next arguments:

siege -t1M -i -A 'Chrome/12.0.742.112 Safari/534.30' -f url.txt

where
-t1M – means run siege during one minute
-i – download links randomly from the list
-A – user-agent description
-f – path to file contains urls.

And get next results:

If you blog doesn’t have a lot of updates like my does, then apparently it is a good idea to install and active caching plugin.

I take 3 more popular unix http server and make some little statistic for them. First I will create 4 files(test,test2,test3,test4) with appropriate size 2GB,700MB,100Mb and 2MB.

dd if=/dev/zero of=test count=1024 bs=2000000
dd if=/dev/zero of=test2 count=1024 bs=700000
dd if=/dev/zero of=test3 count=1024 bs=100000
dd if=/dev/zero of=test4 count=1024 bs=2000

here they are:

andrii@agrytsenko:~/tmp/test/web$ ls -lh
total 2.7G
-rw-r--r-- 1 andrii vboxusers 2.0G May 13 11:15 test
-rw-r--r-- 1 andrii vboxusers 684M May 13 12:14 test2
-rw-r--r-- 1 andrii vboxusers  98M May 13 12:15 test3
-rw-r--r-- 1 andrii vboxusers 2.0M May 13 12:15 test4

Next step downloading and compilation of servers. If will show only configuration flags and skip all the rest of it.
APACHE

./configure --prefix=~/tmp/test/http --disable-include --disable-filter \
--disable-charset-lite --disable-log-config --disable-env --disable-version \
--disable-mime --disable-status --disable-autoindex --disable-asis --disable-cgid \
--disable-cgi --disable-negotiation --disable-dir --disable-actions --disable-userdir \
--disable-alias --without-pcre --without-ssl

NGINX

./configure --prefix=~/tmp/test/http2 --without-http_ssi_module \
--without-http_access_module --without-http_userid_module --without-http_geo_module \
--without-http_map_module --without-http_rewrite_module \
--without-http_proxy_module --without-http_fastcgi_module \
--without-http_limit_zone_module --without-http_browser_module \
--without-pcre --without-http-cache

LIGHTTPD

./configure --prefix=/home/sheva/tmp/test/http3 --disable-silent-rules\
  --disable-dependency-tracking  --disable-libtool-lock --disable-ipv6 \
--without-pcre --without-bzip2 --without-mysql --without-ldap  \
--without-openssl --without-zlib --without-kerberos5

All of them uses port 1234 to listen. It will prevent all servers work simultaneously and it by turn prevent from distorted results.

I used default configuration so result can be far away from reality. I’ve made 2 tests make result more realistic and fetch files with wget through local TCP/IP stack.

This is result of first one:

Second attempt with next results:

This is final average results table:

Here the scheme illustrates the distributed LDAP system between severals networks located in different part of world:

This can be used for centralized authentication system shared between different data centers or like LDAP fail-over as well .

MASTER

Create ldap user replication in directory People.

Append three lines at the end of your slapd.conf:

overlay syncprov
syncprov-checkpoint 1000 60
syncprov-sessionlog 100

overlay syncprov – to make ldap instance act as a provider
syncprov-checkpoint ‘ops’ ‘minutes’ – if number of operations or time passed more then specified then new checkpoint is set up.
syncprov-sessionlog ‘time’ – is the maximum number of session log entries the session log can record.

SLAVES

The configuration for slave are almost the same. Except rid parameter they are must be unique for each node.

SLAVE #1

syncrepl rid=102
    provider=ldap://10.2.2.1:389
    type=refreshOnly
    interval=00:00:01:00
    searchbase="dc=cloudstores,dc=com"
    filter="(objectclass=*)"
    attrs="*,+"
    scope=sub
    schemachecking=off
    bindmethod=simple
    binddn="cn=replication,ou=People,dc=cloudstores,dc=com"
    credentials=replication_password

SLAVE #2

syncrepl rid=103
    provider=ldap://10.2.2.1:389
    type=refreshOnly
    interval=00:00:01:00
    searchbase="dc=cloudstores,dc=com"
    filter="(objectclass=*)"
    attrs="*,+"
    scope=sub
    schemachecking=off
    bindmethod=simple
    binddn="cn=replication,ou=People,dc=cloudstores,dc=com"
    credentials=replication_password

rid – node identification number should be unique for every node.
provider=ldap[s]://hostname[:port] – path to ldap master server.
[type='refreshOnly|refreshAndPersist'] – can be refreshOnly and refreshAndPersist. If set refreshOnly replication will be run with interval specified at interval directive.
[interval=dd:hh:mm:ss] – interval time for replication.
[searchbase='base DN'] – root DN.
[filter='filter str'] – filter for replication
[attrs='attr list'] -sort returned entries based on attributes. The zero-length attributes sorts entries by DN.
[scope='sub|one|base'] – specify a base object, one-level, or subtree search. The default is sub.
[schemachecking='on|off'] – default value is off. If turned on slapd will be checking every entry on compliance with schema definition.
[bindmethod='simple|sasl'] – simple or sasl. Describe replication authentication method
[binddn='DN'] – dn name of replication user.
[credentials='passwd'] – password of replication user.

Find more about syncrepl

This is how it’s should be looks like at the end:

First, install all needed packages for pacemaker:

yum install heartbeat corosync pacemaker

Start service and make it bootable during the system startup:

/etc/init.d/corosync start

After installation generate corosync key for nodes:

corosync-keygen

Copy corosync key to second node:

scp /etc/corosync/authkey 192.168.0.2:/etc/corosync/authkey

create /etc/corosync/service.d/pcmk on nodes with content :

service {
        name: pacemaker
        ver:  0
}

Edit configuration files on both nodes:

mv /etc/corosync/corosync.conf.example /etc/corosync/corosync.conf

Set bindnetaddr to network address others directives can be left without the changes:

bindnetaddr: 192.168.0.0

Now configure Virtual IP resource for this nodes. Nodes have to check each other every 20 seconds:

 crm configure primitive P_IP ocf:heartbeat:IPaddr2 \
        params ip="192.168.0.3" cidr_netmask="255.255.255.0" \
        op monitor interval="20s"

And go up to next level. Httpd server resource:

 crm configure primitive P_APACHE ocf:heartbeat:apache \
        params configfile="/etc/httpd/conf/httpd.conf" statusurl="http://localhost/server-status" \
        op monitor interval="40s"

where
P_APACHE – resource name
configfile – path to apache configuration file
statusurl – url to status page( below how to configures one)
interval – time between checks

To prevent situation when resource apache migrate to node002 and resource IP stays at node001(It happens when apache at node001 hung but network stack works well) we need to make colocation

 crm configure colocation WEB_SITE inf: P_APACHE P_IP

To make pacemaker start up apache only after IP is set up. In other words describe start up order run:

 crm configure order START_ORDER inf: P_IP P_APACHE

Describe location priority:

location L_IP_NODE001 P_IP 100: node001.example.com
location L_IP_NODE002 P_IP 100: node002.example.com

Now set priority threshold. Value 110 is enough to prevent resource migration back. It could be happens when next scenario occurs:
1. node001 fails and resources are moved to node002.
2. then node001 is going online
3. resources are migrated to node001

To stick resources to node002 and prevent from further migration add resource-stickiness:

 crm configure rsc_defaults resource-stickiness="110"

# crm configure show
node node001.example.com
node node002.example.com
primitive P_APACHE ocf:heartbeat:apache \
        params configfile="/etc/httpd/conf/httpd.conf" statusurl="http://localhost/server-status" \
        op monitor interval="40s"
primitive P_IP ocf:heartbeat:IPaddr2 \
        params ip="10.22.48.138" cidr_netmask="255.255.255.240" \
        op monitor interval="20s"
colocation WEB_SITE inf: P_APACHE P_IP
order START_ORDER inf: P_IP P_APACHE
property $id="cib-bootstrap-options" \
        dc-version="1.0.10-da7075976b5ff0bee71074385f8fd02f296ec8a3" \
        cluster-infrastructure="openais" \
        expected-quorum-votes="2" \
        stonith-enabled="false" \
        no-quorum-policy="ignore"

Keep in mind that server-status page should be described at apache configuration files on both nodes as next:

<VirtualHost 127.0.0.1:80>
    ServerAdmin webmaster@dummy-host.example.com
    ServerName localhost
    ErrorLog logs/dummy-host.example.com-error_log
    CustomLog logs/dummy-host.example.com-access_log common
    <Location /server-status>
        SetHandler server-status
        Order deny,allow
        Deny from all
        Allow from 127.0.0.1
    </Location>
</VirtualHost>

Finally check you pacemaker status:

crm_mon
============
Last updated: Wed Jul  6 15:17:46 2011
Stack: openais
Current DC: node001.example.com - partition with quorum
Version: 1.0.10-da7075976b5ff0bee71074385f8fd02f296ec8a3
2 Nodes configured, 2 expected votes
2 Resources configured.
============

Online: [ node001.example.com node001.example.com ]

P_APACHE        (ocf::heartbeat:apache):        Started node001.example.com
P_IP        (ocf::heartbeat:IPaddr2):       Started node001.example.com

That means that two resources started at node001 where node002 in stand-by mode.
There are couples ways to move resources to other node:
1. Set active node to standby:

crm node standby node001.example.com

2. Or directly move resource

crm resource [resource_name] [node_name]

To enable module you need to compile nginx with –with-http_perl_module for CentOS build this possability has being already built inside. When you done with it you need inculude you lib to nginx configuration:

perl_modules        /root/libs;
perl_require        test.pm;

where
perl_modules – path to your module(s)
perl_require – the filename of your module

Then describe the location where scripts where run:

location /statistics {
perl test::runcmd;
}

Means if you go to http://your_site/statistics script function runcmd from test.pm file will be executed.

package test;

use nginx;

sub runcmd {
     my $r = shift;

    $r->send_http_header("text/html");
    return OK if $r->header_only;

    my %commands = (
        'netstat' => 'netstat -lnp |',
        'ls' => 'ls -la /var/log/nginx |',
        'pass' => 'cat /etc/passwd |'
    );

    my $file = $r->filename;

    my $cmd;
    $file =~ /^.*\/([^\/]+$)/;
    my $meth = $1;
    $r->print("Command is $commands{$meth}<br><br>");

    open(NETSTAT, "$commands{$meth}");
    my $cont;
    $cont .= "$_<br>" while (<NETSTAT>);
    close NETSTAT;
    $r->print("$cont");
    return OK;
}

1;
__END__

If you open http://your_site/statistics/netstat script run netstat -lnp command and redirect output to your browser.

There is also possibility to run perl function from SSI:

<!- # perl sub="module::function" arg="parameter1" arg="parameter2"... >

And dont forget to reload nginx after changing configuration:

/etc/init.d/nginx reload

For more details go to official souce.

And btw I still didn’t find any practical appliance yet:(.

PHP-FPM

If you are going to use PHP 5.3.3 or higher you don’t need to patch your php sources by PHP-FPM patch anymore.
Install all required debian packages:

 apt-get install libxml2-dev \
libxml2 libssl-dev libevent-dev libbz2-dev libcurl4-openssl-dev \
libjpeg62-dev libpng12-dev libxpm-dev libfreetype6-dev libt1-dev \
libmcrypt-dev libmysql++-dev libmysqld-dev libmysqlclient-dev \
libxslt1-dev autoconf 

Then download and unzip your php sources and go inside the dir and run buildconf:

./buildconf --force

if no errors occur run configure:

./configure \
--prefix=/opt/php5 \
--with-config-file-path=/opt/php5/etc \
--with-curl \
--with-pear \
--with-gd \
--with-jpeg-dir \
--with-png-dir \
--with-zlib \
--with-xpm-dir \
--with-freetype-dir \
--with-t1lib \
--with-mcrypt \
--with-mhash \
--with-mysql \
--with-mysqli \
--with-pdo-mysql \
--with-openssl \
--with-xmlrpc \
--with-xsl \
--with-bz2 \
--with-gettext \
--with-fpm-user=www-data \
--with-fpm-group=www-data \
--enable-fpm \
--enable-exif \
--enable-wddx \
--enable-zip \
--enable-bcmath \
--enable-calendar \
--enable-ftp \
--enable-mbstring \
--enable-soap \
--enable-sockets \
--enable-sysvmsg \
--enable-sysvsem \
--enable-sysvshm \
--with-libevent-dir=/usr/lib/

After finish without errors proceed and compile and install it:

make && make install

The place where you can find all php-fpm related files and default prefix is /opt/php5. Now customize your configuration at /opt/php5/etc/php-fpm.conf

[global]
pid = /opt/php5/var/run/php-fpm.pid
error_log = /opt/php5/var/log/php-fpm.log
log_level = warning
daemonize = yes # set yes to make php work as a daemon

[www]
listen = 127.0.0.1:9000 # the ip and port for php to listen; the port which nginx are going to work with
user = www-data # set user of new php instances(children)
group = www-data
pm = dynamic
pm.max_children = 50 # the maximum concurrent user's request which can be served by php
pm.min_spare_servers = 5 # the minimum idle processes
pm.max_spare_servers = 35

More details about how to children and servers work you can get here pm.max_children, pm.max_spare_servers, pm.min_spare_servers. Its works for apache just in the same way as for php-fpm.

The next step is making init scripts… Okay I’ve done it for you. Just enjoy:

#!/bin/bash

### BEGIN INIT INFO
# Provides:          php-fpm
# Required-Start:    $local_fs $remote_fs $network $syslog
# Required-Stop:     $local_fs $remote_fs $network $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: starts php-fpm daemon
# Description:       starts php-fpm daemon
### END INIT INFO

function start {
    $DAEMON -c $CONFIG &> /dev/null
}

function stop {
#    killall php-fpm &> /dev/null
    kill `cat /opt/php5/var/run/php-fpm.pid 2>/dev/null` &> /dev/null
}

. /lib/lsb/init-functions

CONFIG=/opt/php5/etc/php-fpm.conf
DAEMON=/opt/php5/sbin/php-fpm
DESC="php-fpm daemon"
PID=/opt/php5/var/run/php-fpm.pid

test -x $DAEMON || exit 0
test -d $CONFIG_DIR || exit 0

kill -0 `cat /opt/php5/var/run/php-fpm.pid 2>/dev/null` &> /dev/null
running=$?
case $1 in
start)
    if [ 0 -eq $running ]; then
        echo "Process php-fpm is already running... Nothing to do..."
        exit
    fi
    start && echo "Process php-fpm was successfully started ... "
;;
stop)
    if [ 0 -ne $running ]; then
        echo "Process php-fpm is not running..."
        exit
    fi
    stop && echo "Process php-fpm was successfully stoped ... "
;;
restart)
    stop && start;
    echo "Process php-fpm was successfully restarted ... "
;;
status)
    if [ 0 -eq $running ]; then
        echo "Process php-fpm is running..."
    else
        echo "Process php-fpm is not running..."
    fi
;;
*)
    echo "$0 [start|stop|restart|status]"
;;
esac

Then make the stuff work after reboot add it to runlevels:

update-rc.d php-fpm defaults

NGINX

With nginx installation there is easier way. Simply install it with apt-get package manager:

apt-get install nginx 

I’m not going to describe all configuration just mention about php part. To make nginx work with php let it know about. Go to server section and add:

location ~ \.php$ {
    fastcgi_pass   127.0.0.1:9000;
    fastcgi_index  index.php;
    fastcgi_param  SCRIPT_FILENAME  /var/www/nginx$fastcgi_script_name;
    include        fastcgi_params;
    }

The fastcgi_pass should be the similar to section listen in php-fpm.conf.
Directive fastcgi_param you need to specify path to your dir where php files are located(i.e. for me its /var/www/nginx)

VERIFICATION

To verify that everything work well. Use an old method with phpinfo function. Create php file in you htroot direcory with next content:

<?php
phpinfo();
?>

Then give the name like info.php or whatever(but should with .php extension anyway). And try to open in your browser. If works it would be looks like that.

PROFIT…

My keyboards Fn key was invented for a long time. It was driving me crazy all the time when I used console. So finally I decided to find a solution it’s takes couple minutes but I have checked a lot of links before find useful. So I wanna share this with you.

To make Fn key work properly restart you computer and go to the BIOS installation(press F10 as soon as computer will start) then go to System Configuration -> Action Keys Mode set as Disable.

So that’s all.

There are different relationship type of replication which can be applied. Here I gonna set up MASTER<->SLAVE scheme.

MASTER

These actions should be taken at master side. Go to /etc/mysql:

 cd /etc/mysql 

And edit your mysql configuration file my.cfg:

server-id = 1
log_bin = /var/log/mysql/mysql-bin.log
binlog_do_db = monitoring
binlog_do_db = roundcube
binlog_do_db = information_schema

Here I defined 3 databases for replication: monitoring, roundcube and information_schema.
Where directive server-id should be always be 1 for master host and >1 for any other slave. Also to define more than one database for replication use directive binlog_do_db for each DB. Never ever try to put all database in one line and separate by comma or whatever(It is a trap I spent a lot of time doing this stuff:( ).

To replicate all databases from master skip variable binlog_do_db.

Restart your mysql when done:

/etc/init.d/mysql restart

Connect to your mysql master server:

mysql -u root -p

and run these mysql statements:

mysql> CREATE USER 'repl'@'192.168.7.177' IDENTIFIED BY 'slavepass';
mysql> GRANT REPLICATION SLAVE ON *.* TO 'repl'@'192.168.7.177';

This two lines create new user repl with password slavepass allow it connects only from ip address 192.168.7.177(set ‘%’ instead of 192.168.7.177 if you want to open access for any host). And grand permission for that user to do replication operations.

mysql> FLUSH TABLES WITH READ LOCK;
mysql> SHOW MASTER STATUS;

+———————-+————+———————————————————-+————————-+
| File                          | Position | Binlog_Do_DB                                                  | Binlog_Ignore_DB |
+———————-+————+———————————————————-+————————-+
| mysql-bin.000001 | 98            | monitoring,roundcube,information_schema |                                    |
+———————-+————+———————————————————-+————————-+

From these statements you need keep in mind value of Position it will be useful when you will run slave.

Before you run mysqlump you need to lock database(don’t close the session before backup done):

mysql>LOCK TABLES;

And run mysqldump in another console window

mysqldump --lock-all-tables -u root -p --databases monitoring, roundcube, informantion_scheman >dbdump.db

In case you have decided that you going to replicate all dbs:

mysqldump --lock-all-tables -u root -p --all-databases >dbdump.db

Now you can unlock the database. To do it you can either close mysql session where you run LOCK TABLES (then tables will be automatically unlocked) or run:

mysql> UNLOCK TABLES;

Then you are ready to transfer snapshot to slave host:

scp dbdump.db user@slave:/tmp

SLAVE:

The same actions with mysql config in /etc/mysqldirectory but slave side:

server-id = 2
replicate-do-db = monitoring
replicate-do-db = roundcube
replicate-do-db = information_schema

If you want replicate all databases skip variables replicate-do-db.
Run mysql client and set up connection with master:

mysql> CHANGE MASTER TO
         MASTER_HOST='192.168.7.5',
         MASTER_USER='repl',
         MASTER_PASSWORD='slavepass',
         MASTER_LOG_FILE='mysql-bin.000001',
         MASTER_LOG_POS=98;

Where,
MASTER_HOST – ip or domain address of your master host.
MASTER_USER – the name of user you created above.
MASTER_PASSWORD- the password
MASTER_LOG_FILE – field file from SHOW MASTER STATUS statement above.
MASTER_LOG_POS the value from you Position field at SHOW MASTER STATUS.

And restart mysql:

/etc/init.d/mysql restart

TESTING

To check replication status run on slave:

mysql> show slave status \G 

And you get something like that:

*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: xxx.xxx.xxx.xxx
                  Master_User: repl
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: bin-log.000463
          Read_Master_Log_Pos: 694544815
               Relay_Log_File: relay-log.001607
                Relay_Log_Pos: 130280711
        Relay_Master_Log_File: bin-log.000463
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 694544815
              Relay_Log_Space: 130280860
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
1 row in set (0.00 sec)

Most important position you got to check is Slave_IO_Running  and Slave_SQL_Running if one of then is No something is definetly wrong with your replication and you need to check variables Last_IO_Error and Last_SQL_Error for more details.

If both Slave_IO_Running  and Slave_SQL_Running are Yes than make some test changes(little data modification) at master in replicated database and value in Read_Master_Log_Pos should be increased in some way. If so than it works well.

To make local https server visible to outside world. You can make PAT(Port Address Translation) but what if 443 port is already binded? Than you probably want to make a reverse proxy like I did. So to configure apache’s virtual domain to work with ssl http do several easy steps.

First, you need to load mod_proxy’s modules into apache and make them available after the restart(for debian there is kind of exotic way to do that):

cd /etc/apache2/mods-enabled
ln -s ../mods-available/proxy.conf proxy.conf
ln -s ../mods-available/proxy.load proxy.load
ln -s ../mods-available/proxy_connect.load proxy_connect.load
ln -s ../mods-available/proxy_http.load proxy_http.load

The idea that contents of each file put in mods-enabled becomes a part of main configuration after restart or configuration re-read.

For example, if file contains line like that

LoadModule proxy_module /usr/lib/apache2/modules/mod_proxy.so

It is the same if we put this line directly in apache2.conf or httpd.conf of whatever you have.

Hope it is pretty clear. Than let’s move on and create new virtual host configuration file:

cd /etc/apache2/sites-available
touch ssl-proxy

Here the example how it’s should be looks like:

<VirtualHost *:443>

  ServerName mail.center.ua
  ProxyRequests Off
  <Proxy *>
    Order deny,allow
    Allow from all
  </Proxy>

  SSLProxyEngine On
  ProxyPass / https://192.168.1.3/
  ProxyPassReverse / https://192.168.1.3/

  # logging section
  LogLevel info
  ErrorLog /var/log/apache2/ssl-proxy.log
  CustomLog /var/log/apache2/ssl-proxy.log combined

  SSLEngine on
  SSLProtocol all
  SSLCertificateFile /etc/ssl/apache/server.crt
  SSLCertificateKeyFile /etc/ssl/apache/server.key
</VirtualHost>

About SSLCertificateFile and SSLCertificateKeyFile you need to generate keys. Use google to figure how to do it.

The mechanism of virtual host is similar to mods mechanism. So create new link in sites-enabled:

ln -s ../sites-available/ssl-proxy ssl-proxy

And restart apache:

/etc/init.d/apache2 restart

Good look!

INSTALLATION

Installation takes just a few easy steps. As I installed it on Debian box so used apt-get for installation:
apt-get install nagios3

After nagios was installed I created new monitoring user root with password root123 to have access to web page:
htpasswd -cb /etc/nagios3/htpasswd.users root root123

And start new service:
/etc/init.d/nagios3 start

SETTING UP

All Almost all configuration are stored at /etc/nagios3/conf.d.

I have one administrator and one contact group of course. So I remained example configuration and change only my mail address at contacts_nagios2.cfg:
email my@address.com

I’m gonna monitoring 3 linux hosts plus one router with next services:
1. Master

• dns
• httpd
• http + ssl
• samba
• nis
• ftp
• openvpn  ( don’t have corresponding script. Probably make my own.)
• ping
• smtp
• imap
• mysql
• ssh
• processes
• disk usage
• swap utilization
• users
• uptime
• current load

2. Slave

• processes
• swap
• disk usage
• users
• http
• ssh
• current load
• ping
• uptime

3. Test

• ldap ( was unable  to configure script check through  the tcp_check)
• ping
• ssh
• samba

4. Router

• ping

First of all you need to define all hosts in localhost_nagios2.cfg:
define host{
use generic-host ; Name of host template to use
host_name secondary
alias Secondary
address 192.168.7.177
}

Where, use refers to generic-host in generic-host_nagios2.cfg. I left everything as was:
define host{
name generic-host ; The name of this host template
notifications_enabled 1 ; Host notifications are enabled
event_handler_enabled 1 ; Host event handler is enabled
flap_detection_enabled 1 ; Flap detection is enabled
failure_prediction_enabled 1 ; Failure prediction is enabled
process_perf_data 1 ; Process performance data
retain_status_information 1 ; Retain status information across program restarts
retain_nonstatus_information 1 ; Retain non-status information across program restarts
check_command check-host-alive
max_check_attempts 10
notification_interval 0
notification_period 24x7
notification_options d,u,r
contact_groups admins
register 0 ; DONT REGISTER THIS DEFINITION - ITS NOT A REAL HOST, JUST A TEMPLATE!
}


You can customize it – set group who will be notified in case of emergency and time-frame for notification.

Also you can make some host groups if you want. It’s not necessary but it can simplify service management if you have a lot of hosts. To define host group go to hostgroups_nagios2.cfg:

define hostgroup {
hostgroup_name all
alias All Servers
members *
}define hostgroup {
hostgroup_name debian-servers
alias Debian GNU/Linux Servers
members secondary, master
}

Here we created two groups of hosts: all and debian-servers. All contains all defined hosts in configuration and debian-servers just a secondary and master hosts.

And at the end most interesting and most resource-intensive part – services. For example if you need to monitor HTTP server on host master put next lines in services_nagios2.cfg:

define service {
host_name master
service_description HTTP
check_command check_http
use generic-service
notification_interval 0
}

Next example if you want to just ping router put:
define service {
host_name router
service_description PING
check_command check_ping!100.0,20%!500.0,60%
use generic-service
notification_interval 0
}

If you need sent arguments to monitoring script separate it by ‘!’.
To monitor group of servers use hostgroup_name intead of host_name, like:
hostgroup_name all
- to monitoring all servers described in hostgroup all.

To monitoring such stuff like disk usage, process, users, swap and memory utilization I followed direction from official site.

PROBLEMS

The mail notifications wasn’t work for me. I suggest because there is no mail server on box. I tried to described it in the configuration but haven’t found anything could help me. So I wrote little script to send a mail directly to another smtp server:

#!/usr/bin/perluse strict;
use MIME::Lite;
use Getopt::Long;

Please define variables smtp and to by yourself. And remember that perl modules MIME::Lite and Getopt::Long have to be preinstalled in the system.

And change lines at /etc/nagios3/commands.cfg appropriately:
command_line /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\nHost: $HOSTNAME$\nState: $HOSTSTATE$\nAddress: $HOSTADDRESS$\nInfo: $HOSTOUTPUT$\n\nDate/Time: $LONGDATETIME$\n" | /usr/bin/mail -s "** $NOTIFICATIONTYPE$ Host Alert: $HOSTNAME$ is $HOSTSTATE$ **" $CONTACTEMAIL$
and
command_line /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$" | /root/scripts/mail.pl -s "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" -t $CONTACTEMAIL$

To those ones:
command_line /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\nHost: $HOSTNAME$\nState: $HOSTSTATE$\nAddress: $HOSTADDRESS$\nInfo: $HOSTOUTPUT$\n\nDate/Time: $LONGDATETIME$\n" | /path/to/mail.pl -s "** $NOTIFICATIONTYPE$ Host Alert: $HOSTNAME$ is $HOSTSTATE$ **" -t $CONTACTEMAIL$
and
command_line /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$" | /root/scripts/mail.pl -s "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" -t $CONTACTEMAIL$

Where, /root/scripts/mail.pl path to script mail.pl which was provided above.

Get new version of package:

wget ftp://ftp.sunfreeware.com/pub/freeware/sparc/10/lsof_1106-4.80-sol10-sparc-local.gz

Unpack it:

gunzip lsof_1106-4.80-sol10-sparc-local.gz

And run installation command as root:

pkgadd -d lsof_1106-4.80-sol10-sparc-local                                                                   

The following packages are available:
  1  SMClsof     lsof
                 (sparc) 4.80        

Select package(s) you wish to process (or 'all' to process
all packages). (default: all) [?,??,q]: all
## Verifying package  dependencies in zone 

Files that are setuid and/or setgid will be installed and/or modified
for package  on zone .                         

Do you want to continue with the installation of  [y,n,?] y
..............skipped................
Do you want to install these as setuid/setgid files [y,n,?,q] y

..............skipped................
Installation of  on zone  was successful.

And finally check it out:

lsof -v
....skipped....
revision: 4.80

Done.

There will be two scenario:

1. Delete partition by fdisk and restart the computer.
2. Delete and create empty partition by fdisk and restart the computer.
3. Delete whole storage device from the system and restart computer.
4. Delete and create whole storage device from the system and restart computer.

1. Delete partition by fdisk and restart the computer.

First, I gotta create the raid device:

[root@node1 ~]# mdadm -C /dev/md0 -n 2 -l 1 /dev/hdd1 /dev/hdb8
mdadm: largest drive (/dev/hdb8) exceed size (476160K) by more than 1%
Continue creating array? y
mdadm: array /dev/md0 started.

and build ext3 file system on the device:

[root@node1 ~]# mke2fs -j /dev/md0

Edit /etc/fstab according to the changes:

/dev/md0                /test                   ext3    defaults        0 0

and mount everything is contained at /etc/fstab:

[root@node1 ~]#mount -a

Copy some files to /test(to be able to check files integrity in the future):

[root@node1 ~]# cp -rfv /var/* /test/

Now, delete /dev/hdd1 through fdisk:

[root@node1 ~]# fdisk  /dev/hdd

The number of cylinders for this disk is set to 8322.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): d
Selected partition 1

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

and restart computer:

[root@node1 ~]# reboot

During the system booting I got next error(the same I got when try to manually mount md0):

mount: wrong fs type, bad option, bad superblock on /dev/md0,
       missing codepage or other error
       (could this be the IDE device where you in fact use
       ide-scsi so that sr0 or sda or so is needed?)
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

/proc/mdstat doesn’t see any local raid devices:

[root@node1 ~]# cat /proc/mdstat
Personalities :
unused devices:

Let’s create new mdadm.conf :

[root@node1 ~]#  mdadm --examine --scan  /dev/hdb8 > /etc/mdadm.conf

[root@node1 ~]# cat /etc/mdadm.conf
ARRAY /dev/md0 level=raid1 num-devices=2 UUID=c1ce0c10:035aa2a3:829450b6:84b7a236

And active everything from mdadm.conf:

[root@node1 ~]#  mdadm -A -s
mdadm: /dev/md0 has been started with 1 drive (out of 2).

The md0 was activated, but only with one disk:

[root@node1 ~]# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 hdb8[1]
      476160 blocks [2/1] [_U]

Conclusion: If it was a root partition your system wouldn’t boot at all. And you would have to dance with boot disk to make you system alive.

2. Delete and create empty partition by fdisk and restart the computer.

The same actions as above and additional create partition /dev/hdd1 in fdisk.
After system booted. None of raid’s are active :

[root@node1 ~]# cat /proc/mdstat
Personalities :
unused devices:

But after restore procedure system is going to work in full-fledged mode(with 2 disks):

[root@node1 ~]#  mdadm --examine --scan  /dev/hdb8 > /etc/mdadm.conf
[root@node1 ~]# mdadm -A -s
mdadm: /dev/md0 has been started with 2 drives.
[root@node1 ~]# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 hdd1[0] hdb8[1]
475136 blocks [2/2] [UU]

The new device has no influence on system behavior at all. Result is similar to previous one.

3. Delete whole storage device from the system and restart computer.

During this test the md0 was restored and mounted during system boot without any problem or delays. Also next notice message were generated and put into the syslog facility kernel:

kernel: raid1: raid set md0 active with 1 out of 2 mirrors

4. Delete and create whole storage device from the system and restart computer.

During boot raid wasn’t rebuilt by itself:

[root@node1 ~]# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 hdb8[1]
      475136 blocks [2/1] [_U]

And has to be re-built manually :

[root@node1 ~]# mdadm -a /dev/md0 /dev/hdd1
mdadm: re-added /dev/hdd1

Still recovering :) :

[root@node1 ~]# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 hdd1[0] hdb8[1]
      475136 blocks [2/1] [_U]
      [====>................]  recovery = 21.5% (103296/475136) finish=0.4min speed=12912K/sec

NOTE: partition type for new partition should be set to ‘Linux raid autodetect’ (FD in a hex code). Frankly speaking I didn’t test whether it works without that, that’s why it may be or may _not_ useless procedure. My advice is: just do it!

[root@node1 ~]# fdisk -l /dev/hdd

Disk /dev/hdd: 4294 MB, 4294967296 bytes
16 heads, 63 sectors/track, 8322 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/hdd1               1         943      475240+  fd  Linux raid autodetect

P.S. System information:

Tests were performed at virtual machine:
virtualbox-2.1.4
virtualbox-ose-guest-modules-2.6.26-1-686
Software installed inside VM:
CentOS(kernel-2.6.18-128.el5)
mdadm-2.6.9-2.el5

PRE-REQUIREMENTS:

oowriter has to be already installed. It is provided by Open Office system(for Debian its openoffice.org-writer package).

Unfortunately, I can’t remember the source where this macros is came from. Anyway here my script:

#!/bin/bash
# Written by Andrii Grytsenko
# pp: AndriiGrytsenko.net

if [ $# -lt 1 ]; then
    echo "Example: pdf2doc [doc_file] [output_dir]"
    echo "in case output_dir is missing generate into current dir"
fi

DOCDIR=$(dirname $1)
DOCFILE=$(basename $1)
PDFFILE=$(echo $DOCFILE | sed -e 's/\.doc$/\.pdf/g')
CURRENT_DIR=$(pwd)
OUT_DIR=$2

# if output_dir is missing set current
if [ -z $OUT_DIR ]; then
    OUT_DIR=$CURRENT_DIR
fi

if  echo $DOCDIR | grep -E "^\.$" > /dev/null; then
        DOCDIR=$CURRENT_DIR
elif ! echo $DOCDIR | grep -E "^/" > /dev/null; then
        DOCDIR=$CURRENT_DIR/$DOCDIR
fi

cd $DOCDIR
/usr/bin/oowriter -invisible "macro:///Standard.Module1.ConvertWordToPDF($DOCDIR/$DOCFILE)"
sleep 2 # for big files generation can take more time
mv $PDFFILE $OUT_DIR/
cd $CURRENT_DIR

Put file into the catalog from PATH env variable, into the /usr/bin for example. And call doc2pdf.

To generate question.pdf from question.doc and put it into the /tmp, run this:

doc2pdf ~/tmp2/questions.doc /tmp/

PROFIT:

andrii@agrytsenko:~$ ls -l /tmp/questions.pdf
-rw-r--r-- 1 andrii vboxusers 53226 Jul 16 14:17 /tmp/questions.pdf

I’m not gonna describe how to make HTML form and how to parse input data from html. I leave these as you your homework ;). As storage for HTTP sessions I chose database(specifically MYSQL).

Here is my table with sessions :

desc mc_sessions;
+-----------------+--------------+------+-----+---------+-------+
| Field           | Type         | Null | Key | Default | Extra |
+-----------------+--------------+------+-----+---------+-------+
| user_name       | varchar(50)  | YES  |     | NULL    |       |
| session_id      | varchar(255) | NO   | PRI | NULL    |       |
| expiration_time | int(11)      | YES  |     | NULL    |       |
+-----------------+--------------+------+-----+---------+-------+

use CGI qw/:standard/;
use CGI::Cookie;

sub set_cookies {.
    my ($db,$user_name,$user_passwd) = @_;

    my $exp_time = time() + 259200; #(3 days)
    my $session_id = md5_hex($user_id.''.$user_name.''.$user_passwd.''.$exp_time);

    my $sth = $dbh->prepare("INSERT INTO $table (user_name,session_id,expiration_time) VALUES (?,?,?)");
    $sth->execute($user_name,$session_id,$exp_time);

    my $cookie1 = new CGI::Cookie(-name=>'SessionID',-value=>"$session_id");
    my $cookie2 = new CGI::Cookie(-name=>'ExpirationTime',-value=>"$exp_time");

    print header(-cookie=>[$cookie1,$cookie2]);

    return 1;
}

sub check_auth {
    my ($dbh) = @_;

    my %cookies = fetch CGI::Cookie;
    my %cook_hash;

    foreach my $key (sort keys %cookies) {
        $cookies{$key} =~ /=(.*);/;
        $cook_hash{$key} = $1;
    }

    return -1 if (!defined($cook_hash{SessionID}));

    my $user_name = is_session_valid($dbh,$cook_hash{SessionID});

    return -1 if ($user_name == -1);

    return $user_name;
}

sub is_session_valid {
    my ($dbh,$session_id) = @_;

    my ($exp_time);

    my $sth = $dbh->prepare("SELECT * FROM sessions WHERE session_id=? and expiration_time > ?");
    $sth->execute($session_id,time());

    while ( my @array = $sth->fetchrow_array ) {
            ($user_name,$session_id,$exp_time) = @array;
    }

    return $user_name if (defined($user_name) and $exp_time > time());

    return -1;
}

As result we have 3 functions.

1. set_cookies – this function generate, put in database and send cookies to user’s browser.
2. check_auth – check if user has already have some cookies from server if yes call is_session_valid for future checking.
3. is_session_valid – look about user’s  session id  and check whether this session was expired or not.

Pay attention:

Each function receives variable $dbh it’s database connection identificator and should be created before function run.

How does it use?

Edit you CGI code. And take care that all pages checks sessions id.
Put this function call after authorization by password to send cookies to user:

set_cookies($dbh,$username,$password);

And these in all remains site’s pages to prevent site from unauthorized actions:

my $user_name = check_auth($dbh);

if ( $user_id == -1 ) {
        some action[s]
}

First I decide to make some test database to do everything step-by-step. This will be companies database contains information about all employers, projects and departments.

CREATE DATABASE company

Go in:

USE company

And make 4 tables.

This table consists relation between employer and projects:

Let’s create its in DB:

CREATE TABLE employers (
employer_id int primary key auto_increment,
name varchar(15),
surname varchar(25),
hire_date date,
department_id int,
fired int);

CREATE TABLE projects (
project_id int primary key auto_increment,
name varchar(255));

CRETE TABLE departments (
department_id int primary key auto_increment,
name varchar(255));

CREATE TABLE project_involved (
employer_id int,
project_id int);

And insert some data into them. I will show you only few of them:

INSERT INTO projects (departments) VALUES ('General Project');

In this insert I use subquery:

INSERT INTO employers
(name,surname,hire_date,department_id,fired)
VALUES ('Jonh','Smith','2007-10-30',
(SELECT department_id FROM departments WHERE name='Department A'),
0);

Format of date should be the next:

YYYY-MM-DD

There is also some example of SELECT query in the UPDATE statement:

INSERT INTO projects (name) VALUES ('General Project');

INSERT INTO project_involved (employer_id,project_id)
VALUES (
(SELECT employer_id FROM employers WHERE name='Jonh' AND surname='Smith'),
(SELECT project_id FROM projects WHERE name='General Project')
);

In the ending of filling db up I’ve got next results:

SELECT * FROM employers;
+-------------+------+---------+------------+---------------+-------+
| employer_id | name | surname | hire_date  | department_id | fired |
+-------------+------+---------+------------+---------------+-------+
|           1 | Jonh | Smith   | 2007-10-30 |             1 |     0 |
|           2 | Yan  | Brown   | 2008-11-23 |             2 |     0 |
+-------------+------+---------+------------+---------------+-------+
2 rows in set (0.00 sec)

SELECT * FROM project_involved;
+-------------+------------+
| employer_id | project_id |
+-------------+------------+
|           1 |          1 |
|        NULL |          2 |
+-------------+------------+
2 rows in set (0.00 sec)

SELECT * FROM projects;
+------------+-----------------+
| project_id | name            |
+------------+-----------------+
|          1 | General Project |
|          2 | Second Project  |
+------------+-----------------+
2 rows in set (0.00 sec)

SELECT * FROM departments;
+---------------+---------------+
| department_id | name          |
+---------------+---------------+
|             1 | departament A |
|             2 | departament B |
+---------------+---------------+
2 rows in set (0.00 sec)

If you need to to remove something from your table(Department D from departments for example):

DELETE FROM departments WHERE name='Department D';

I’ve totally forgot about some column should be at employers table. Column position has to be added. To change the structure of the table use ALTER:

ALTER TABLE employers
ADD COLUMN position varchar(255) AFTER surname;

in the same way you can remove needed column from the table:

ALTER TABLE employers DROP COLUMN position;

To rename column name run:

ALTER TABLE table_name CHANGE old_name new_name INTEGER

Now update the data:

UPDATE employers SET position='system engineer'
WHERE surname='Smith' AND name='Jonh';

There are few examples with subqueries. If you need to select all data from emloyers table and resolve department name from departments table. Run:

SELECT employers.name, employers.surname, employers.hire_date, departments.name
FROM employers INNER JOIN departments USING (department_id);

To simplify a query use alias:

SELECT e.name, e.surname, e.hire_date, d.name FROM employers e
INNER JOIN departments d USING (department_id);
+------+---------+------------+---------------+
| name | surname | hire_date  | name          |
+------+---------+------------+---------------+
| Jonh | Smith   | 2007-10-30 | departament A |
| Yan  | Brown   | 2008-11-23 | departament C |
+------+---------+------------+---------------+
2 rows in set (0.00 sec)

Here is more complex query which is involves all 4 tables:

SELECT e.name, e.surname, e.position, e.hire_date, d.name department, p.name project
FROM projects p
INNER JOIN project_involved USING (project_id)
INNER JOIN employers e USING (employer_id)
INNER JOIN departments d USING (department_id);
+------+---------+-----------------+------------+---------------+-----------------+
| name | surname | position        | hire_date  | department    | project         |
+------+---------+-----------------+------------+---------------+-----------------+
| Jonh | Smith   | system engineer | 2007-10-30 | departament A | General Project |
| Jonh | Smith   | system engineer | 2007-10-30 | departament A | Second Project  |
+------+---------+-----------------+------------+---------------+-----------------+
2 rows in set (0.00 sec)

To remove database or table use DROP:

DROP [database|table]

For example:

DROP employers;

To see database or table structure use SHOW CREATE:

SHOW CREATE TABLE table;

SHOW CREATE DATABASE company;

SHOW CREATE DATABASE company;
+----------+--------------------------------------------------------------------+
| Database | Create Database                                                    |
+----------+--------------------------------------------------------------------+
| company  | CREATE DATABASE `company` /*!40100 DEFAULT CHARACTER SET latin1 */ |
+----------+--------------------------------------------------------------------+

BACKUPS

To make backups of your database use mysqldump :

mysqldump -u root -p --database company > company.sql

To restores from backups :

mysql -u root -p -e 'source company.sql' company

CREATE USER

GRANT [privileges] ON [resource] TO [identity]

In example I create user company_admin which has all privileges on all tables in database company from host localhost with password secret-password:

GRANT ALL PRIVILEGES ON company.* TO company_admin@localhost
IDENTIFIED BY 'secret_password';

There are some of the privileges, almost all of them is self-explained:

ALTER
CREATE
DELETE
DROP
GRANT
INSERT
SELECT
UPDATE

For more details concerning GRANT go to ofdocs.

MYISAM VS INNODB

InnoDB is set by default. And it’s speed and mature way to store simple data.

Advantages of InnoDB:

1. transaction support
2. row-level locking
3. foreign key constraints

To disadvantages we can attribute the next:

1. don’t support full-text searching
2. some troubles with AUTO_INCREMENT columns

To change of the table engine run:

ALTER TABLE table ENGINE=[engine]

ALTER TABLE employers ENGINE=InnoDB

FOREIGN KEY

This feature help you to prevent your data from human mistake and keep it integrity.
First, you need to convert your tables into InnoDB:

ALTER TABLE table ENGINE=InnoDB;

and set dependency:

ALTER TABLE employers ADD CONSTRAINT dep_id_cons
FOREIGN KEY (department_id) REFERENCES departments (department_id);

Now when we try to set employers.department_id which is not at departments.department_id. As result changes is not committed and we get an error:

INSERT INTO employers (name,surname,department_id)
VALUES ('test','test','123');
ERROR 1452 (23000): Cannot add or update a child row:
a foreign key constraint fails (`company/employers`,
CONSTRAINT `dep_id_cons` FOREIGN KEY (`department_id`)
REFERENCES `departments` (`department_id`))

To delete foreign key run:

ALTER TABLE employers DROP FOREIGN KEY dep_id_cons;

Speaking about UA-IX, from wikipedia we know:

The Ukrainian traffic exchange network (UA-IX) (Ukrainian: Українська мережа обміну інтернет-трафіком) is an Internet exchange point that was established in Kiev, the capital of Ukraine, on July 2000 and is the daughter company of Ukrainian Internet Association. Companies that established Ukrainian traffic exchange network are major Ukrainian businesses that work expanding Ukrainian Internet market.

Here the list of all networks which belongs to UA-IX net. Download it to your machine and save as ua-list.txt.

Content of file. We have some networks list here:

$ head -3 ua-list.txt
8.8.4.0/24
8.8.8.0/24
62.16.0.0/19

Here is the script:

$cat uaix.pl
#!/usr/local/bin/perl
# Written by Andrii Grytsenko 2010
# PP: http://andriigrytsenko.net     

use POSIX;
use strict;

sub dec2bin {
   my $hex = shift;
   my $pack = unpack("B32",pack("N",$hex));
   my $pack = substr($pack,-8);
   return $pack;
}

my $bin_ip;
$bin_ip .= dec2bin($_) foreach (split(/\./,$ARGV[0]));

open(IP, "<$ARGV[1]");
while(){
    chomp();
    my ($ip,$net_mask) = split(/\//,$_);
    $net_mask = 24 if (!defined($net_mask));
    my $bin_network;
    $bin_network .= dec2bin($_) foreach (split(/\./,$ip));
    my $net_net_part = substr($bin_network,0,$net_mask);
    my $net_ip_part  = substr($bin_ip,0,$net_mask);
    if ( $net_net_part == $net_ip_part) {
        print "IP $ARGV[0] belong to network $_\n";
        exit;
    }
}

print "IP $ARGV[0] is not part of UA-IX\n";
close(IP);

To run the script:

$ perl uaix.pl [ip_addr] [path/to/ua-list.txt]

For example:

$ perl uaix.pl 72.14.228.156 ua-list.txt
IP 72.14.228.156 belong to network 72.14.192.0/18

There is also some ways to use script on your own… You can use your list of networks to detect ;)

There is some listing of my script I hope this example help you to understand some fundamental points in IP math.

#!/usr/local/bin/perl
# Written by Andrii Grytsenko 2010
# PP: http://andriigrytsenko.net    

use POSIX;

sub hex2bin {
   my $hex = shift;
   my $pack = unpack("B32",pack("N",$hex));
   my $pack = substr($pack,-8);
   return $pack;
}                                          

sub bin2hex {
   my $bin = shift;
   return unpack("N", pack("B32", substr("0" x 32 . $bin, -32)))
}                                                               

sub bin2ip {
    my $bin = shift;
    my $type = shift;
    my $ip;
    $ip .= bin2hex(substr($bin,0,8)) ."\.";
    $ip .= bin2hex(substr($bin,8,8)) ."\.";
    $ip .= bin2hex(substr($bin,16,8)) ."\.";
    if ( $type eq 'first') {
        $ip .= bin2hex(substr($bin,24,8)) + 1 ;
    } elsif ($type eq 'broad') {
        $ip .= bin2hex(substr($bin,24,8)) - 1 ;
    } else {
        $ip .= bin2hex(substr($bin,24,8));
    }
    return $ip;
}

sub normalization {
    my $line = shift;
    my ($ip,$mask) = split(/\//,$line);
    return($ip,$mask);
}

my ($ip,$net_mask) = normalization(@ARGV[0]);

my $host_mask = 32-$net_mask;
my $max_host = POSIX::pow(2,$host_mask)-2;
my $bin_mask = "1" x $net_mask . "0" x $host_mask;

my $bin_ip ;
$bin_ip .= hex2bin($_) foreach (split(/\./,$ip));

my $min = $bin_ip & $bin_mask;
my $max_mask = "0" x $net_mask . "1" x $host_mask;
my $max = $bin_ip  | $max_mask;

my $min_ip = bin2ip($min);
my $min_host_ip = bin2ip($min,'first');
my $max_ip = bin2ip($max);
my $max_host_ip = bin2ip($max,'broad');

print \<\< EOF
IP ADDRESS:\t $ip\t $bin_ip
NETWORK MASK:\t \/$net_mask\t\t $bin_mask
WILDCARD:\t \/$host_mask\t\t $max_mask
NETWORK IP:\t $min_ip\/$net_mask\t $min
BROADCAST ADDR:\t $max_ip\t $max
FIRST ADDR:\t $min_host_ip
LAST ADDR:\t $max_host_ip
MAX HOSTS\t $max_host
EOF

Run scripts with one argument ip/netmask… Unfortunately, you can use only short network mask type :

./ipcalc 192.157.9.3/21

the output looks like this:

IP ADDRESS:      192.157.9.3     11000000100111010000100100000011
NETWORK MASK:    /21             11111111111111111111100000000000
WILDCARD:        /11             00000000000000000000011111111111
NETWORK IP:      192.157.8.0/21  11000000100111010000100000000000
BROADCAST ADDR:  192.157.15.255  11000000100111010000111111111111
FIRST ADDR:      192.157.8.1
LAST ADDR:       192.157.15.254
MAX HOSTS        2046

Here is short info about the host(solaris 5.9, arch sparc):

uname -a
SunOS hostname1 5.9 Generic_118558-25 sun4u sparc SUNW,Sun-Fire-V890 Solaris

To see all shared memory segment, use ipcs

user1@~>ipcs -m -a
IPC status from  as of Wed Mar 10 16:26:20 CET 2010
T         ID      KEY        MODE        OWNER    GROUP  CREATOR   CGROUP NATTCH      SEGSZ  CPID  LPID   ATIME    DTIME    CTIME
Shared Memory:
m       4608   0x5efaa728 --rw-r-----   oracle oinstall   oracle oinstall     23 3992993792  5720 18925 16:24:47 16:24:47 11:59:42
m    5246977   0x517b6233 --rw-r--r-- user1   os_int user1   os_int     32    7352048 10685 16417 16:21:26 16:21:26 16:13:35
m    1969154   0x22b80b3e --rw-r--r-- user1   os_int user1   os_int      3   72552448 10689 11443 16:14:19 16:13:51 16:13:37
m    1967619   0x1b68b6e6 --rw-r--r-- user1   os_int user1   os_int      8  438326272 10689 16417 16:21:26 16:21:26 16:13:38
m    1966084   0x5724f977 --rw-r--r-- user1   os_int user1   os_int     14  212926464 10689 11379 16:14:27 16:13:50 16:13:40
m    1966085   0x534aab2a --rw-r--r-- user1   os_int user1   os_int     10  216572928 10689 11442 16:14:38 16:13:51 16:13:42
m   11391045   0x7bfc552e --rw-r--r-- user2   os_int user2   os_int      0    7352048  8047  9445 14:43:22 14:43:40 13:41:22
m   10461766   0x63fcba9b --rw-r--r-- user2   os_int user2   os_int      0   72552448  8114  9457 14:43:22 14:43:38 13:41:25
m    9721415   0x5cad6643 --rw-r--r-- user2   os_int user2   os_int      0  583097344  8114  9459 14:43:22 14:43:38 13:41:27
m    9655368   0x1869a8d4 --rw-r--r-- user2   os_int user2   os_int      0  212926464  8114  9466 14:43:22 14:43:38 13:41:29
m    9753673   0x148f5a87 --rw-r--r-- user2   os_int user2   os_int      0  216572928  8114  9473 14:43:22 14:43:38 13:41:31

To delete/destroy/remove any segment of the shared memory, use ipcrm. You can specify segment by shmkey(3-rd field in ipcs output) or by shmid(2-nd field) use -M and -m accordingly . Like :

ipcrm -m 4608

or

ipcrm -M 0x5efaa728

To remove all segment in shared memory where owner is user1, run

ipcs -m -a | awk '{if($5 ~ /user1/ && $1 ~ /m/) print "ipcrm -m "$2;}' | bash

END.

I used perl module Benchmark to make measurements. Rules:

1. If you can use single quotes instead of double quote, do it. In most cases it will be executed more faster because of string in single quotes is not going to concatenate.

2. Use compiled regexps in case you have loops. It can brings huge performance:

use Benchmark;

sub qwe {
    my $re = shift;
    my $line = "wer";
    foreach my $reg (@$re){
        if ( $line =~ /$reg/) {
            my $l=0;
        }
    }
}

my $line = "sdfsdfsdfsdf";
my $ref = ".*";
my @arr = qw(qwe sdf wer); #plaine regexps
my @c_re;
foreach my $p_re (@arr) { # compile it and put in to the @c_re
    my $re = qr/\b$p_re\b/i ;
    push(@c_re,$re);
}

Benchmark::cmpthese(100000, {
    'Compiled' => sub { qwe(\@c_re) },
    'Not_compiled' => sub { qwe(\@arr) },
});

As result:

                 Rate Not_compiled     Compiled
Not_compiled  36232/s           --         -79%
Compiled     169492/s         368%           --

3. Extract values from string with substr if you can. There are three well knows ways to do such kind of operation: regular expressions, split and substr. Let’s compare it:

sub mksplit {
    my ($line) = @_;
    my ($fst,$snd) = split(/-/,$line);
    return($fst,$snd);
}

sub mkregexp {
    my ($line,$re) = @_;
    $line =~ m/$re/;
    return($1,$2);
}

sub mkstr {
    my ($line) = @_;
    my $pos = index($line,'-');
    my $fst = substr($line,0,$pos);
    my $snd = substr($line,$pos+1);
    return($fst,$snd);
}

my $line = "12-34";
my $n_comp_re = '(\d+)-(\d+)';
my $comp_re = qr/\b$n_comp_re\b/i ;

Benchmark::cmpthese(1000000, {
    'Compiled' => sub { mkregexp($line,$comp_re) },
    'Not_compiled' => sub { mkregexp($line,$n_comp_re) },
    'STR' => sub { mkstr($line) },
    'Split' => sub { mksplit($line) },
});

Benchmark::timethese(1000000, {
    'Compiled' => sub { mkregexp($line,$comp_re) },
    'Not_compiled' => sub { mkregexp($line,$n_comp_re) },
    'STR' => sub { mkstr($line) },
    'Split' => sub { mksplit($line) },
});

Where compiled and not_compiles we use when talk about regular expression.

                 Rate        Split     Compiled Not_compiled          STR
Split        297619/s           --          -9%         -12%         -25%
Compiled     327869/s          10%           --          -4%         -17%
Not_compiled 340136/s          14%           4%           --         -14%
STR          396825/s          33%          21%          17%           --
Benchmark: timing 1000000 iterations of Compiled, Not_compiled, STR, Split...
  Compiled:  4 wallclock secs ( 3.14 usr +  0.00 sys =  3.14 CPU) @ 318471.34/s (n=1000000)
Not_compiled:  3 wallclock secs ( 2.91 usr +  0.00 sys =  2.91 CPU) @ 343642.61/s (n=1000000)
       STR:  2 wallclock secs ( 2.53 usr +  0.00 sys =  2.53 CPU) @ 395256.92/s (n=1000000)
     Split:  3 wallclock secs ( 3.19 usr +  0.00 sys =  3.19 CPU) @ 313479.62/s (n=1000000)

As you can see from output the more faster way to exract values from string is substr, next is split and regular expressions is the most slower. I don’t know why compiled regexps is more slow than regular one in this case. I guess its because of 1 iteration.

4. Don’t use sort in hash loops if there is no necessity. The most faster way to through over the hash is keys statement:

sub mkeach {
    my $hash = shift;
    my %hash2;
    while ( my($key,$value) = each(%$hash)){
        $hash2{$key} = $value;
    }
    return(\%hash2);
}

sub mkkeys {
    my $hash = shift;
    my %hash2;
    for my $key (keys(%$hash)){
        $hash2{$key} = $hash->{$key};
    }
    return(\%hash2);
}

sub mksort {
    my $hash = shift;
    my %hash2;
    foreach my $key (sort keys %$hash) {
        $hash2{$key} = $hash->{$key};
    }
    return(\%hash2);
}

my %hash = (name => 'name1', surname => 'surname1', address => 'address', var => 'var1', var2 => 'var2');

Benchmark::cmpthese(1000000, {
    'Keys' => sub { mkkeys(\%hash) },
    'Each' => sub { mkeach(\%hash) },
    'Sort' => sub { mksort(\%hash) },
});

Benchmark::timethese(1000000, {
    'Keys' => sub { mkkeys(\%hash) },
    'Each' => sub { mkeach(\%hash) },
    'Sort' => sub { mksort(\%hash) },
});

        Rate Each Sort Keys
Each 66800/s   --  -3% -14%
Sort 68681/s   3%   -- -11%
Keys 77459/s  16%  13%   --
Benchmark: timing 1000000 iterations of Each, Keys, Sort...
      Each: 16 wallclock secs (14.92 usr +  0.00 sys = 14.92 CPU) @ 67024.13/s (n=1000000)
      Keys: 12 wallclock secs (12.91 usr +  0.00 sys = 12.91 CPU) @ 77459.33/s (n=1000000)
      Sort: 15 wallclock secs (14.62 usr +  0.00 sys = 14.62 CPU) @ 68399.45/s (n=1000000)

That’s all for now, but I’ll continue to optimize my code so I think there will be an updates soon.