Monitoring Cuckoo with Icinga2

The following instructions assume that you have both your Cuckoo instance(s) as well as the Icinga2 instance (which, preferably, runs on a separate server) running on a Debian/Ubuntu-based distribution.

Note that all commands mentioned in this document should be ran as root and that any highlighted lines feature some sort of user-specific configuration.

Installing the Icinga2 master

In this chapter we’ll install the master, also known as the node in which the results from the various clients / satellites may be monitored.

First add the apt key for Icinga2 on Ubuntu:

$ wget -O - http://packages.icinga.org/icinga.key | apt-key add -
$ echo 'deb http://packages.icinga.org/ubuntu icinga-trusty main' > /etc/apt/sources.list.d/icinga.list
$ apt-get update

Or on Debian:

$ wget -O - http://packages.icinga.org/icinga.key | apt-key add -
$ echo 'deb http://packages.icinga.org/debian icinga-jessie main' > /etc/apt/sources.list.d/icinga.list
$ apt-get update

Then we install the actual packages:

$ apt-get install icinga2 php5-json php5-gd php5-imagick php5-mysql
$ apt-get install php5-pgsql php5-intl php5-cli php5-common php5-fpm
$ echo 'date.timezone = "Europe/Amsterdam"' >> /etc/php5/fpm/php.ini
$ icinga2 feature enable command
$ service icinga2 restart

Setup the PostgreSQL database:

$ sudo -u postgres psql
postgres=# CREATE USER icingaweb WITH PASSWORD 'YOURDATABASEPASSWORD';
postgres=# CREATE DATABASE icingaweb;

Create a file, /etc/icinga2/features-enabled/ido-pgsql.conf, with the following contents. Use the Icinga2 database password you specified earlier:

library "db_ido_pgsql"

object IdoPgsqlConnection "ido-pgsql" {
    user = "icinga2",
    password = "YOURDATABASEPASSWORD",
    host = "localhost",
    database = "icinga2"
}

Install the Icinga2 PostgreSQL configuration:

$ apt-get install icinga2-ido-pgsql
# splash screen - "configure now"
# Yes, fill in password

$ icinga2 feature enable ido-pgsql
$ service icinga2 restart

Install icingaweb2:

$ apt-get install icingaweb2

# If it installs apache2, remove it, as we'll be using nginx.
$ apt-get remove apache2 --purge

Create the following nginx configuration at /etc/nginx/sites-available/icinga2, adjust where needed:

server {
    listen 0.0.0.0:80;

    server_name icinga2.yourdomain.tld;

    location ~ ^/index\.php(.*)$ {
        fastcgi_pass unix:/var/run/php5-fpm.sock;
        fastcgi_index index.php;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME /usr/share/icingaweb2/public/index.php;
        fastcgi_param ICINGAWEB_CONFIGDIR /etc/icingaweb2;
    }

    location ~ ^/(.*)? {
        allow all;

        alias /usr/share/icingaweb2/public;
        index index.php;

        rewrite ^/$ /dashboard;
        try_files $1 $uri $uri/ /index.php$is_args$args;
    }
}

Make a symbolic link to the sites-enabled directory:

$ ln -s /etc/nginx/sites-available/icinga2 /etc/nginx/sites-enabled/icinga2
$ service nginx reload

The icingaweb2 service is now accessible through http://<ip>:<port>. Setup the web interface through the website. To start the setup, a one-time token is required for authentication, create it as follows:

icingacli setup token create

As for the setup itself, take the following steps.

  • Step ‘modules’, click Next
  • Step ‘icinga web 2’, should be all green
  • Step ‘Authentication’, click Next
  • Step ‘Database Resource’, fill in PostgreSQL details
  • step ‘Authentication Backend’, click Next
  • step ‘Administration’, create an admin account
  • Next on all steps

After this is finished, login to the icinga2 web interface and notice that icinga2 is already logging the current machine.

Configuring the Icinga2 master

As this is the master node, we will have to configure it as such. We’ll use the wizard. Start as follows:

$ icinga2 node wizard
Welcome to the Icinga 2 Setup Wizard!

We'll guide you through all required configuration details.

Please specify if this is a satellite setup ('n' installs a master setup) [Y/n]: n
Starting the Master setup routine...
Please specifiy the common name (CN) [cuckoocinga2]:
Checking for existing certificates for common name 'cuckoocinga2'...
Certificates not yet generated. Running 'api setup' now.
[...]
Please specify the API bind host/port (optional):
Bind Host []: <YOUR IP ADDRESS>
Bind Port []: <YOUR PORT>
information/cli: Created backup file '/etc/icinga2/features-available/api.conf.orig'.
information/cli: Updating constants.conf.
information/cli: Created backup file '/etc/icinga2/constants.conf.orig'.
information/cli: Updating constants file '/etc/icinga2/constants.conf'.
information/cli: Updating constants file '/etc/icinga2/constants.conf'.
information/cli: Updating constants file '/etc/icinga2/constants.conf'.
Done.

Now restart your Icinga 2 daemon to finish the installation!

$ service icinga2 restart

The setup wizard will do the following:

  • Generate a local CA in /var/lib/icinga2/ca (or use existing one)
  • Generate a new CSR, sign it with the local CA and copying it into /etc/icinga2/pki
  • Generate a local zone and endpoint configuration for this master based on FQDN
  • Enabling the API feature, and setting optional bind_host and bind_port
  • Setting the NodeName and TicketSalt constants in constants.conf

Create or modify the /etc/icinga2/zones.conf file and populate it with the following configuration (please customize as needed):

object Endpoint "icinga2.yourdomain.tld" {
}

object Zone "icinga2.yourdomain.tld" {
    // This is the local master zone = "master"
    endpoints = [ "icinga2.yourdomain.tld" ]
}

Finally restart Icinga2 once again to make sure all settings are applied:

service icinga2 restart

Notifications Events

We’re almost done on the master. We’re going to configure Icinga2 to call our custom script, /etc/icinga2/scripts/notify.py whenever the services ping4, ssh, and check_cuckoo fail. It is up to the user of Cuckoo to implement the actual notify.py script though, as this is out of scope for this documentation.

First of all, on master, append the following lines to /etc/icinga2/conf.d/users.conf:

object User "sysadmin" {
    display_name = "System Administrator"
    enable_notifications = true
    states = [ Warning, Critical ]
    types = [ Problem, Recovery ]
    email = "YOUREMAILADDRESS@YOURDOMAIN.TLD"
}

template Notification "generic-notification" {
    states = [ Warning, Critical, Unknown ]
    types = [ Problem, Acknowledgement, Recovery, Custom, FlappingStart,
              FlappingEnd, DowntimeStart, DowntimeEnd, DowntimeRemoved ]
}

apply Notification "notify-sysadmin" to Service {
    import "generic-notification"

    command = "notify-cuckoo"
    users = [ "sysadmin" ]

    assign where service.name in ["check_cuckoo", "ssh", "ping4"]
}

object NotificationCommand "notify-cuckoo" {
    import "plugin-notification-command"
    command = [
        SysconfDir + "/icinga2/scripts/notify.py"
    ]

    env = {
        NOTIFICATIONTYPE = "$notification.type$"
        SERVICEDESC = "$service.name$"
        HOSTALIAS = "$host.display_name$"
        HOSTADDRESS = "$address$"
        SERVICESTATE = "$service.state$"
        LONGDATETIME = "$icinga.long_date_time$"
        SERVICEOUTPUT = "$service.output$"
        NOTIFICATIONAUTHORNAME = "$notification.author$"
        NOTIFICATIONCOMMENT = "$notification.comment$"
        HOSTDISPLAYNAME = "$host.display_name$"
        SERVICEDISPLAYNAME = "$service.display_name$"
        USEREMAIL = "$user.email$"
    }
}

Then create the file /etc/icinga2/scripts/notify.py and have some meaningful code in there. It’ll be called every time a service fails or recovers (you may want to use the ENV vars). Don’t forget to make it executable:

$ chmod +x /etc/icinga2/scripts/notify.py

Configuring a Icinga2 satellite (client)

A satellite Icinga2 node connects to the master Icinga2 node using SSL. To get started, install Icinga2 on the satellite node, i.e., a Cuckoo node.

First add the apt key for Icinga2 on Ubuntu:

$ wget -O - http://packages.icinga.org/icinga.key | apt-key add -
$ echo 'deb http://packages.icinga.org/ubuntu icinga-trusty main' > /etc/apt/sources.list.d/icinga.list
$ apt-get update

Or on Debian:

$ wget -O - http://packages.icinga.org/icinga.key | apt-key add -
$ echo 'deb http://packages.icinga.org/debian icinga-jessie main' > /etc/apt/sources.list.d/icinga.list
$ apt-get update

Then install Icinga2 itself:

$ apt-get install icinga2
$ icinga2 feature enable command
$ service icinga2 restart

To have this satellite connect to master, we once again use the wizard to properly configure it:

$ icinga2 node wizard
- Satellite setup? [Y/n]: y
- For the common name, use the master common name you supplied earlier doing the wizard for master.
- Establish a connection to master? [Y/n]: y
- Fill in connection details to master
- Leave CSR signing connection details blank
- Please specify the request ticket: run the *hint* cmd on master to acquire the ticket and use it
- Leave API blank
- Accept config from master? [y/N]: y
- Accept commands from master? [y/N]: y

As an example wizard session:

$ icinga2 node wizard
Welcome to the Icinga 2 Setup Wizard!

We'll guide you through all required configuration details.

Please specify if this is a satellite setup ('n' installs a master setup) [Y/n]: y
Please specifiy the common name (CN) [cuckoo1]:
Please specify the master endpoint(s) this node should connect to:
Master Common Name (CN from your master setup): icinga2.yourdomain.tld
Do you want to establish a connection to the master from this node? [Y/n]: y
Please fill out the master connection information:
Master endpoint host (Your master's IP address or FQDN): <YOUR IP ADDRESS>
Master endpoint port [5665]: <YOUR PORT NUMBER>
Add more master endpoints? [y/N]: n
Please specify the master connection for CSR auto-signing (defaults to master endpoint host):
Host [...]:
Port [...]:
[...]
Is this information correct? [y/N]: y
[...]
Please specify the request ticket generated on your Icinga 2 master.
(Hint: # icinga2 pki ticket --cn 'cuckoo1'): [...]
[...]
Please specify the API bind host/port (optional):
Bind Host []:
Bind Port []:
Accept config from master? [y/N]: y
Accept commands from master? [y/N]: y
[...]
Now restart your Icinga 2 daemon to finish the installation!

$ service icinga2 restart

To have master notice the newly added satellite, run the following commands on the server where the Icinga2 master is running:

$ icinga2 node update-config
$ service icinga2 restart

# Optionally you may verify the current configuration.
$ icinga2 object list --type Host

The newly added satellite should show up in the list.

Setting up the Cuckoo check service

We’ll make a custom service that checks if Cuckoo is currently working on the satellite. This code will run locally on each satellite node.

On the satellite create the following file /usr/lib/nagios/plugins/check_cuckoo with the following contents.

#!/usr/bin/python
import sys
import argparse
import requests
from math import log

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("-H", "--host", help="API server host",
                        default="localhost", action="store", required=True)
    parser.add_argument("-p", "--port", help="API server port",
                        default=8090, action="store", required=True)
    args = parser.parse_args()

def pretty_size(n, pow=0, b=1024, u='B', pre=[''] + [p + 'i' for p in 'KMGTPEZY']):
    pow, n = min(int(log(max(n * b ** pow, 1), b)), len(pre) - 1), n * b ** pow
    return "%%.%if %%s%%s" % abs(pow % (-pow - 1)) % (n / b ** float(pow), pre[pow], u)

def json_to_nagios(blob, base=""):
    def _format(key, val):
        if "diskspace" in key and isinstance(val, (int, float)):
            size = pretty_size(val, b=1024, u='B', pre=['', 'K', 'M', 'G'])
            return "%s=%s " % (key, size.replace(" ", ""))
        elif isinstance(val, (int, float)):
            return "%s=%s " % (key, str(val))
        else:
            # returning strings in nagios labels not allowed
            return ""

    rtn = ""
    if isinstance(blob, dict):
        for _k, _v in blob.iteritems():
            key = base + "_" + _k
            if isinstance(_v, dict):
                rtn += json_to_nagios(_v, key)
            else:
                rtn += _format(key, _v)
    elif isinstance(blob, list):
        #rtn += "%s=(%s) " % (base, ",".join([str(z) for z in blob]))
        pass
    else:
        rtn += _format(base, str(blob))

    return rtn

url = "http://%s:%s/cuckoo/status" % (args.host, args.port)

try:
    resp = requests.get(url, timeout=5)
    if not resp.status_code == 200:
        raise Exception("status code not 200")

    resp = resp.json()
except Exception as ex:
    print "Error - %s" % str(ex)
    sys.exit(2)

output = "Cuckoo %s OK|" % resp["version"]

for k, v in resp.iteritems():
    output += json_to_nagios(v, base=k)

print output
sys.exit(0)

Don’t forget to make it executable:

$ chmod +x /usr/lib/nagios/plugins/check_cuckoo

Now open /etc/icinga2/conf.d/services.conf, remove everything, and paste the following lines:

apply Service "ping4" {
    import "generic-service"
    check_command = "ping4"
    assign where host.address
}

apply Service "ssh" {
    import "generic-service"
    check_command = "ssh"
    assign where (host.address || host.address6) && host.vars.os == "Linux"
}

apply Service for (disk => config in host.vars.disks) {
    import "generic-service"
    check_command = "disk"
    vars += config
}

apply Service "icinga" {
    import "generic-service"
    check_command = "icinga"
    assign where host.name == NodeName
}

apply Service "load" {
    import "generic-service"
    check_command = "load"
    /* Used by the ScheduledDowntime apply rule in `downtimes.conf`. */
    vars.backup_downtime = "02:00-03:00"
    assign where host.name == NodeName
}

apply Service "swap" {
    import "generic-service"
    check_command = "swap"
    assign where host.name == NodeName
}

apply Service "check_cuckoo" {
    import "generic-service"
    check_command = "check_cuckoo"
    assign where host.name == NodeName
}

Then append the following lines to /etc/icinga2/conf.d/commands.conf:

object CheckCommand "check_cuckoo" {
    import "plugin-check-command"

    command = [ PluginDir + "/check_cuckoo" ]

    arguments = {
        "-H" = "127.0.0.1",
        "-p" = "8090"
    }
}

To finish off the installation of this satellite, run the following two commands on both the satellite and the master:

$ icinga2 node update-config
$ service icinga2 restart

The service checks for this satellite should now be visible in the Icinga2 dashboard and you should now have realtime monitoring enabled for your Cuckoo node.