2014-01-02

Monitoring the network using Icinga running on Raspberry Pi, Pushover and Nagify

The problem

My family and I live in a house with a lot of technologies and devices, for example
  • Multiple Access Point spread over the house
  • A NAS which is used as storage and backup device (8)
  • A Mac mini acting as "iTunes" content server, mostly for the Apple TV in the living room
  • A SIP based telephony solution which is based on Asterisk and SIPgate (5)
  • A firewall and internet router based on pfSense (4)
  • Multiple telephones Snom M9
  • An Elgato Netstream Sat, a satellite TV network streaming server that provides TV on the Ethernet (7)
A picture of my 19" rack in the basement with some of the mentioned devices:
The rest of the hardware is:
  • (1) Patch panel that terminates the CAT 7 cables in the house
  • (2) 24-port Gigabit Ethernet switch, the main hub for the internal network
  • (3) An old 24-port Fast-Ethernet switch for the perimeter network, currently not used
  • (4) An Alix 2D13 board in a 1HE rack case running pfSense
  • (5) Another Alix board in the same rack case running Askozia, an Asterisk distribution with a nice web front end.
  • (6) A plain old dumb DSL modem, the PPPoE is run on pfSense
  • (9) A Raspberry Pi this blog is about!
Sometimes there are problems like:
  • The balance on SIPgate gets low
  • The connection between Asterisk and SIPgate is stuck and telephony does not work at all.
  • Some of the Access Points in the house stop working correctly, mostly due to heat. This mostly happens in summer.
  • The storage on the NAS is full, so backups of the Laptops stop working.
Most of the problems can be resolved simply:
  • The connection between Asterisk and SIPgate can be fixed by resetting the firewall state tables.
  • The Access Points simply need a reboot or hard reset and continue working for months.
My family members often complained that the telephone did not work, which was simple to fix, but I wanted to monitor and fix it, before anybody else detects it. You see, I have to deliver a SLA that goes  much beyond standard business demand. So what I needed was a monitoring solution with the following requirements:
  • A monitoring solution that is able to flexibly monitor my heterogenous network
  • The possibility to push notification on my iPhone and give me the possibility to take a deeper look on the reason.
  • A separate and therefor independent infrastructure for monitoring. I did not want to add this monitoring to any of my existing devices and servers.
  • Cheap: I am not willing spend a lot of money for neither the hardware, the software and running costs (power)

The solution

To summarize the solution:
  • I have a Raspberry Pi running Raspbian as Operating System
  • It runs Icinga (a Nagios compatible monitoring solution)
  • Notifications are pushed via Pushover to my iPhone, also available for Android.
  • Nagify on the iPhone is a front end for Nagios and Icinga. I am sure that there is an alternative to Android. Nagify has some nifty features, that I will explain later.

Raspberry Pi

The Raspberry Pi is a nice piece of hardware, because it only costs about 40€ and offers a complete computer. It uses a SD-card as memory and only needs about 4W of power. I would have preferred another Alix board, which I run successfully and without any problems for the last 5 years, but with the rack casing this solution is much more expensive. The Raspberry Pi is not very fast, but absolutely ok for running Icinga (with some exceptions). The setup of Raspberry Pi is very easy:
  • Buy an SD-card with at least 8 GB capacity. I have 32 GB, but this is far more then ever needed.
  • Download and install Raspbian. There is no need to use NOOBS.
  • Enable if not already a DHCP server in the local network and connect the Pi to the local network using an Ethernet cable.
  • Stick in the SD-card
  • Power up the Pi
  • Your DHCP server should be able to show you the IP address of the last assigned IP. Make it static and give it a nice DNS entry like "icinga" or "monitor" or whatever.
  • Reset the Pi to get the new IP address
  • You should be able to login with "ssh pi@icinga" with the password "raspberry". In my case there is no more password needed, because I added my public keys for ssh.

Icinga

I evaluated multiple products
  • Nagios, the grandfather
  • Icinga, a more modern approach
  • Shinken, a very modern approach
  • Zabbix
  • Hyperic
Reasons for Icinga are:
  • It is fully compatible to Nagios so thousands of existing scripts e.g. from Nagios Exchange can be used.
  • There are already packages for Raspian
  • It has a nicer interface than Nagios
  • It has a large community
  • It is lightweight enough to run on the Pi
The basic steps for setting up Icinga are:
  • Install Icinga, MySQL, IDO2DB and Nagios-Plugins. This page offers a nice description.
  • Learn how this configure Nagios(!) objects like Hosts, Host groups, Services and Service groups. The original Nagios document is very good. Understand that there is inheritance in the object definitions that can save a lot of time. The configurations basically happen by editing the *.cfg files. There are some tools that will help you, but I learned the format and how inheritance works so I am edit the files directly by my own.
After that you should be able to login as icingaadmin on http://icinga.yourdomain/icinga/ and see the login screen. Note: Icinga offers a new web fronted called Icinga-Web. It looks nice, but on the Pi it is so slow, that it does not make fun. So stick with the old Icinga frontend which works nice and offers all we need. Here is how mine looks after I configured all:
The one warning that I currently have is the balance on my SIPgate account:
As final steps you should do the following:

  • Configure a provider for Dynamic DNS like DynDNS
  • Configure HTTPS instead or additionally to HTTP so you are able to access Icinga like https://icinga.yourdomain/icinga/
  • Configure your router/firewall to forward the port from external, so you are able to access Icinga from the internet. You ask why? For Nagify...

Nagify

Nagify is an App for the iPhone that offers you basically the same functionality like the original Icinga web frontend. As the name suggests it was originally developed for Nagios, but also supports Icinga. If you have configured HTTPS/Dynamic DNS and the Port forwarding, you should be able to configure your external URL to Nagify so you will be able to check Icinga remotely.

Overview of the hosts


List of warnings

Warning in detail with the possibility to acknowledge it

Pushover

Pushover is a nice service that offers an API for sending notifications and Apps for iPhone and Android that display those notifications. The web frontend looks like this:
As you can see, I have registered two iPhones as devices and Icinga as application. Pushover is a perfect solution where Icinga can send us notifications to the iPhone. The following steps are needed:
  • Take the script notify_by_pushover.sh and put it to your /usr/local/bin directory. I prefer adding own data to /usr/local to have a clear separation and easier backup. As you can see, I found this script elsewhere, but I modified it to contain this "ngfy:"-URL which can be used by Pushover, you will see later.
  • Add those command definitions to your configuration. Be sure to configure your correct application and ID settings, I xxx-ed mine out to avoid spam :-)
  • Configure the contacts and contact groups to actually use the Pushover commands and script to send notifications.
If everything is configured correctly, a notification looks like this:
In detail:
As you can see, the modified script contains a URL that starts with ngfy:. This URL has been defined by Nagify so tipping on this link will bring you directly to the service screen in Nagify:
This nice URL feature is described on the Nagify homepage under "FAQ" and "Launch by URL"

Conclusion

I found the solution that I was looking for. It is inexpensive and offers all I needed. Since I activated it, I received some notifications so I was able to fix them quickly. There are open points I did not mention so far:
  • There is monitoring of Icinga itself. A solution would be to have a secondary instance of Icinga just for monitoring the first one. I ordered a rack mount for my Pi which offers space for two, so maybe the second slot get a Pi for exactly this.
  • Pushover uses Apple's Pushover Notifications and Apple recommends not to use this type of notifications for business critical notifications, because Apple cannot guarantee that they will be delivered. I read into this and there are circumstances where messages can be lost, but for this business case the reliability is absolutely sufficient.
  • The Icinga installation and especially the object configuration needs to be backed up. I can imaging that running a Unix on a SD-card in read-write mode will wear it much, so there should be an easy way to re-setup Icinga on a new SD-card if the current gets worn out. I found a nice solution with my NAS. If you are interested in, contact me and I will write a blog post about it.
  • The notification itself requires that the internet connection is available. In case it should drop, the notifications will not pass through. A solution would be to use some device/old GSM phone that would be able to send SMS in such a case (or even instead of Pushover). I would take this into consideration if the internet line would fail often, but it doesn't
I did not write much about the scripts that do the actual monitoring. Tell me if you want to know more.