Aug
15
2017
Nagios with PNP4Nagios on CentOS 6.x

nagios_logo.png

In many companies, nagios is the de-facto monitoring tool. Even with new modern alternatives solutions, this opensource project, still, has a large amount of implementations in place. This guide is based on a “clean/fresh” CentOS 6.9 virtual machine.

Epel

An official nagios repository exist in this address: https://repo.nagios.com/
I prefer to install nagios via the EPEL repository:

# yum -y install http://fedora-mirror01.rbc.ru/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

# yum info nagios | grep Version
Version     : 4.3.2

# yum -y install nagios

Selinux

Every online manual, suggest to disable selinux with nagios. There is a reason for that ! but I will try my best to provide info on how to keep selinux enforced. To write our own nagios selinux policies the easy way, we need one more package:

# yum -y install policycoreutils-python

Starting nagios:

# /etc/init.d/nagios restart

will show us some initial errors in /var/log/audit/audit.log selinux log file

Filtering the results:

# egrep denied /var/log/audit/audit.log | audit2allow

will display something like this:

#============= nagios_t ==============
allow nagios_t initrc_tmp_t:file write;
allow nagios_t self:capability chown;

To create a policy file based on your errors:

# egrep denied /var/log/audit/audit.log | audit2allow -a -M nagios_t

and to enable it:

# semodule -i nagios_t.pp

BE AWARE this is not the only problem with selinux, but I will provide more details in few moments.

Nagios

Now we are ready to start the nagios daemon:

# /etc/init.d/nagios restart

filtering the processes of our system:

# ps -e fuwww | egrep na[g]ios

nagios    2149  0.0  0.1  18528  1720 ?        Ss   19:37   0:00 /usr/sbin/nagios -d /etc/nagios/nagios.cfg
nagios    2151  0.0  0.0      0     0 ?        Z    19:37   0:00  _ [nagios] <defunct>
nagios    2152  0.0  0.0      0     0 ?        Z    19:37   0:00  _ [nagios] <defunct>
nagios    2153  0.0  0.0      0     0 ?        Z    19:37   0:00  _ [nagios] <defunct>
nagios    2154  0.0  0.0      0     0 ?        Z    19:37   0:00  _ [nagios] <defunct>
nagios    2155  0.0  0.0  18076   712 ?        S    19:37   0:00  _ /usr/sbin/nagios -d /etc/nagios/nagios.cfg

super!

Apache

Now it is time to start our web server apache:

# /etc/init.d/httpd restart

Starting httpd: httpd: apr_sockaddr_info_get() failed
httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName

This is a common error, and means that we need to define a ServerName in our apache configuration.

First, we give an name to our host file:

# vim /etc/hosts

for this guide, I ‘ll go with the centos69 but you can edit that according your needs:

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4 centos69

then we need to edit the default apache configuration file:

# vim /etc/httpd/conf/httpd.conf

#ServerName www.example.com:80
ServerName centos69

and restart the process:

# /etc/init.d/httpd restart

Stopping httpd:      [  OK  ]
Starting httpd:      [  OK  ]

We can see from the netstat command that is running:

# netstat -ntlp | grep 80

tcp        0      0 :::80                       :::*                        LISTEN      2729/httpd      

Firewall

It is time to fix our firewall and open the default http port, so that we can view the nagios from our browser.
That means, we need to fix our iptables !

# iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT

this is want we need. To a more permanent solution, we need to edit the default iptables configuration file:

# vim /etc/sysconfig/iptables

and add the below entry on INPUT chain section:

-A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT

Web Browser

We are ready to fire up our web browser and type the address of our nagios server.
Mine is on a local machine with the IP: 129.168.122.96, so

http://192.168.122.96/nagios/

User Authentication

The default user authentication credentials are:

nagiosadmin // nagiosadmin

but we can change them!

From our command line, we type something similar:

# htpasswd -sb /etc/nagios/passwd nagiosadmin e4j9gDkk6LXncCdDg

so that htpasswd will update the default nagios password entry on the /etc/nagios/passwd with something else, preferable random and difficult password.

ATTENTION: e4j9gDkk6LXncCdDg is just that, a random password that I created for this document only. Create your own and dont tell anyone!

Selinux, Part Two

at this moment and if you are tail-ing the selinux audit file, you will see some more error msgs.

Below, you will see my nagios_t selinux policy file with all the things that are needed for nagios to run properly - at least at the moment.!

module nagios_t 1.0;

require {
        type nagios_t;
        type initrc_tmp_t;
        type nagios_spool_t;
        type nagios_system_plugin_t;
        type nagios_exec_t;
        type httpd_nagios_script_t;
        class capability chown;
        class file { write read open execute_no_trans getattr };
}

#============= httpd_nagios_script_t ==============
allow httpd_nagios_script_t nagios_spool_t:file { open read getattr };

#============= nagios_t ==============
allow nagios_t initrc_tmp_t:file write;
allow nagios_t nagios_exec_t:file execute_no_trans;
allow nagios_t self:capability chown;

#============= nagios_system_plugin_t ==============
allow nagios_system_plugin_t nagios_exec_t:file getattr;

Edit your nagios_t.te file accordingly and then build your selinux policy:

# make -f /usr/share/selinux/devel/Makefile

You are ready to update the previous nagios selinux policy :

# semodule -i nagios_t.pp

Selinux - Nagios package

So … there is an rpm package with the name: nagios-selinux on Version: 4.3.2
you can install it, but does not resolve all the selinux errors in audit file ….. so ….
I think my way is better, cause you can understand some things better and have more flexibility on defining your selinux policy

Nagios Plugins

Nagios is the core process, daemon. We need the nagios plugins - the checks !
You can do something like this:

# yum install nagios-plugins-all.x86_64

but I dont recommend it.

These are the defaults :

nagios-plugins-load-2.2.1-4git.el6.x86_64
nagios-plugins-ping-2.2.1-4git.el6.x86_64
nagios-plugins-disk-2.2.1-4git.el6.x86_64
nagios-plugins-procs-2.2.1-4git.el6.x86_64
nagios-plugins-users-2.2.1-4git.el6.x86_64
nagios-plugins-http-2.2.1-4git.el6.x86_64
nagios-plugins-swap-2.2.1-4git.el6.x86_64
nagios-plugins-ssh-2.2.1-4git.el6.x86_64

# yum -y install nagios-plugins-load nagios-plugins-ping nagios-plugins-disk nagios-plugins-procs nagios-plugins-users nagios-plugins-http nagios-plugins-swap nagios-plugins-ssh

and if everything is going as planned, you will see something like this:

nagios_checks.jpg

PNP4Nagios

It is time, to add pnp4nagios a simple graphing tool and get read the nagios performance data and represent them to graphs.

# yum info pnp4nagios | grep Version
Version     : 0.6.22

# yum -y install pnp4nagios

We must not forget to restart our web server:

# /etc/init.d/httpd restart

Bulk Mode with NPCD

I’ve spent toooooo much time to understand why the default Synchronous does not work properly with nagios v4x and pnp4nagios v0.6x
In the end … this is what it works - so try not to re-invent the wheel , as I tried to do and lost so many hours.

Performance Data

We need to tell nagios to gather performance data from their check:

# vim +/process_performance_data /etc/nagios/nagios.cfg

process_performance_data=1

We also need to tell nagios, what to do with this data:

nagios.cfg

# *** the template definition differs from the one in the original nagios.cfg
#
service_perfdata_file=/var/log/pnp4nagios/service-perfdata
service_perfdata_file_template=DATATYPE::SERVICEPERFDATAtTIMET::$TIMET$tHOSTNAME::$HOSTNAME$tSERVICEDESC::$SERVICEDESC$tSERVICEPERFDATA::$SERVICEPERFDATA$tSERVICECHECKCOMMAND::$SERVICECHECKCOMMAND$tHOSTSTATE::$HOSTSTATE$tHOSTSTATETYPE::$HOSTSTATETYPE$tSERVICESTATE::$SERVICESTATE$tSERVICESTATETYPE::$SERVICESTATETYPE$
service_perfdata_file_mode=a
service_perfdata_file_processing_interval=15
service_perfdata_file_processing_command=process-service-perfdata-file

# *** the template definition differs from the one in the original nagios.cfg
#
host_perfdata_file=/var/log/pnp4nagios/host-perfdata
host_perfdata_file_template=DATATYPE::HOSTPERFDATAtTIMET::$TIMET$tHOSTNAME::$HOSTNAME$tHOSTPERFDATA::$HOSTPERFDATA$tHOSTCHECKCOMMAND::$HOSTCHECKCOMMAND$tHOSTSTATE::$HOSTSTATE$tHOSTSTATETYPE::$HOSTSTATETYPE$
host_perfdata_file_mode=a
host_perfdata_file_processing_interval=15
host_perfdata_file_processing_command=process-host-perfdata-file

Commands

In the above configuration, we introduced two new commands

service_perfdata_file_processing_command  &
host_perfdata_file_processing_command

We need to define them in the /etc/nagios/objects/commands.cfg file :

#
# Bulk with NPCD mode
#
define command {
       command_name    process-service-perfdata-file
       command_line    /bin/mv /var/log/pnp4nagios/service-perfdata /var/spool/pnp4nagios/service-perfdata.$TIMET$
}

define command {
       command_name    process-host-perfdata-file
       command_line    /bin/mv /var/log/pnp4nagios/host-perfdata /var/spool/pnp4nagios/host-perfdata.$TIMET$
}

If everything have gone right … then you will be able to see on a nagios check something like this:

nagios_perf.png

Verify

Verify your pnp4nagios setup:

# wget -c http://verify.pnp4nagios.org/verify_pnp_config

# perl verify_pnp_config -m bulk+npcd -c /etc/nagios/nagios.cfg -p /etc/pnp4nagios/ 

NPCD

The NPCD daemon (Nagios Performance C Daemon) is the daemon/process that will translate the gathered performance data to graphs, so let’s started it:

# /etc/init.d/npcd restart
Stopping npcd:                                             [FAILED]
Starting npcd:                                             [  OK  ]

You should see some warnings but not any critical errors.

Templates

Two new template definition should be created, one for the host and one for the service:

/etc/nagios/objects/templates.cfg

define host {
   name       host-pnp
   action_url /pnp4nagios/index.php/graph?host=$HOSTNAME$&srv=_HOST_' class='tips' rel='/pnp4nagios/index.php/popup?host=$HOSTNAME$&srv=_HOST_
   register   0
}

define service {
   name       srv-pnp
   action_url /pnp4nagios/graph?host=$HOSTNAME$&srv=$SERVICEDESC$' class='tips' rel='/pnp4nagios/popup?host=$HOSTNAME$&srv=$SERVICEDESC$
   register   0
}

Host Definition

Now we need to apply the host-pnp template to our system:

so this configuration: /etc/nagios/objects/localhost.cfg

define host{
        use                     linux-server            ; Name of host template to use
                                                        ; This host definition will inherit all variables that are defined
                                                        ; in (or inherited by) the linux-server host template definition.
        host_name               localhost
        alias                   localhost
        address                 127.0.0.1
        }

becomes:

define host{
        use                     linux-server,host-pnp            ; Name of host template to use
                                                        ; This host definition will inherit all variables that are defined
                                                        ; in (or inherited by) the linux-server host template definition.
        host_name               localhost
        alias                   localhost
        address                 127.0.0.1
        }

Service Definition

And we finally must append the pnp4nagios service template to our services:

srv-pnp

define service{
        use                             local-service,srv-pnp         ; Name of service template to use
        host_name                       localhost

Graphs

We should be able to see graphs like these:

nagios_ping.png

Happy Measurements!

appendix

These are some extra notes on the above article, you need to take in mind:

Services

# chkconfig httpd on
# chkconfig iptables on
# chkconfig nagios on
# chkconfig npcd on 

PHP

If you are not running the default php version on your system, it is possible to get this error msg:

Non-static method nagios_Core::SummaryLink()

There is a simply solution for that, you need to modify the index file to exclude the deprecated php error msgs:

# vim +/^error_reporting /usr/share/nagios/html/pnp4nagios/index.php   

// error_reporting(E_ALL & ~E_STRICT);
error_reporting(E_ALL & ~E_STRICT & ~E_DEPRECATED);
Jan
21
2015
Web Roles using Ldap attributes

A few days ago, I wrote a simple wiki page on how to Authenticate to a web site using LDAP backend.

There is a cool (and very simple way) to use Web Roles by matching your user’s ldap attributes to your web app.

A RFC 2255 from 1997 exists on how to implement (and use) the LDAP URL Format. The authldapurl syntax from mod_authnz_ldap explains that the “attribute” field can be used with comma to separate different attributes. Every attribute would be passed to your webserver as an AUTHENTICATED_attribute variable.

In my example:



AuthLDAPURL "ldap://ldap.domain.org/ou=web,dc=domain,dc=org?uid,MyWebAccess?one?(WebAccess=MyWebApp_Level_*)"

returns:


AUTHENTICATE_MYWEBACCESS
    MyWebApp_Level_1 
AUTHENTICATE_UID
    myusername 
Tag(s): ldap, apache
Jul
20
2014
apache Redirect permanent your web app to https

This is pretty simple to even document, but i need a reference point !



<VirtualHost 1.2.3.4:80>

        ServerName example.com
        Redirect permanent / https://example.com

</VirtualHost>

dont forget to create the https virtual host, something like that:


<VirtualHost 1.2.3.4:443>

        ServerName example.com

        ServerAdmin admin@example.com

        # Logs
        CustomLog logs/example.com.access.log combined
        ErrorLog  logs/example.com.error.log

        DocumentRoot /www/examplecom
        DirectoryIndex index.html

        <Directory "/www/examplecom">
                Order allow,deny
                Allow from all 

                AllowOverride All 

                AuthType basic
                AuthName "Enter At Your Own Risk"
                AuthUserFile /www/htpasswd_for_examplecom
                Require valid-user

        </Directory>

        # HSTS 
        Header always set Strict-Transport-Security "max-age=31536000; "

        # SSL Support
        SSLEngine on

        SSLProtocol all -SSLv2 -SSLv3
        SSLHonorCipherOrder on
        SSLCipherSuite HIGH:!aNULL:!MD5

        SSLCertificateFile      /certs/examplecom.crt
        SSLCertificateKeyFile   /certs/examplecom.key
        SSLCertificateChainFile /certs/class3.crt

</VirtualHost>
Tag(s): apache, ssl, https
May
19
2014
apache Deflate

It took me a few minutes to understand why one of my online sign files isnt the correct one.

The actual file:


[~]>  md5sum file.sign
89dd90709bbc51eb6796280123f46fe6

The file on my web server:


$  md5sum file.sign
89dd90709bbc51eb6796280123f46fe6

when download it :


[~]>  md5sum file.sign
f57846a7032a1d106799af38ab94cfb7

Say what ?

Whattttttt !


[~]> file file.sign
file.sign: gzip compressed data, from Unix

so what if i did:


gunzip -d -c file.sign | md5sum 
89dd90709bbc51eb6796280123f46fe6  -

So a light bulb lighted up upon my hackergotchi !! mod_deflate

i need to change my filters

Tag(s): deflate, apache
Dec
24
2013
Failures on update

A colleague of mine wants to add a new vhost on one of our apache web servers.

Running:


  /etc/init.d/httpd configtest 

he noticed that php_admin_flag had produced an error msg. We comment this flag out and tried to restart the web server. Unfortunately the httpd didnt came up.

Searching through logs I’ve seen these:


Dec 14 14:33:54 Erased: php-snmp
Dec 14 14:33:54 Erased: php-mbstring
Dec 14 14:33:54 Erased: php-pear
Dec 14 14:33:55 Erased: php-common
Dec 14 14:33:55 Erased: php-mcrypt
Dec 14 14:33:55 Erased: php-gd
Dec 14 14:33:55 Erased: php-mysql
Dec 14 14:33:55 Erased: php-cli
Dec 14 14:33:55 Erased: php-pgsql
Dec 14 14:33:55 Erased: php-ldap
Dec 14 14:33:55 Erased: php
Dec 14 14:33:55 Erased: php-devel
Dec 14 14:33:56 Erased: php-pdo
Dec 14 14:34:17 Installed: php53-common-5.3.3-22.el5_10.x86_64
Dec 14 14:34:17 Installed: php53-pdo-5.3.3-22.el5_10.x86_64
Dec 14 14:34:27 Installed: libc-client-2004g-2.2.1.x86_64
Dec 14 14:34:28 Installed: php53-mcrypt-5.3.3-1.el5.x86_64
Dec 14 14:34:28 Installed: php53-mysql-5.3.3-22.el5_10.x86_64
Dec 14 14:34:28 Installed: php53-ldap-5.3.3-22.el5_10.x86_64
Dec 14 14:34:28 Installed: php53-mbstring-5.3.3-22.el5_10.x86_64
Dec 14 14:34:28 Installed: php53-gd-5.3.3-22.el5_10.x86_64
Dec 14 14:34:28 Installed: php53-xml-5.3.3-22.el5_10.x86_64
Dec 14 14:34:28 Installed: php53-imap-5.3.3-22.el5_10.x86_64
Dec 14 14:34:28 Installed: php53-snmp-5.3.3-22.el5_10.x86_64
Dec 14 14:34:28 Installed: php53-pgsql-5.3.3-22.el5_10.x86_64
Dec 14 14:34:28 Installed: php53-cli-5.3.3-22.el5_10.x86_64

If you havent noticed the horror yet let me explain it to you:

There is NO php on the system!

A couple weeks ago, another colleague did a not so successfully update on this server.

blah blah blah
blah blah blah
blah blah blah

and story told short:


yum install php53.x86_64

worked it’s magic.

So keep it in mind that after yum update, you have to do manual restarts on the running services and check that everything works properly OR someone like me, will try to destroy your Christmas plans as a revenge !

Tag(s): centos, apache, php, update