Monitor your cloud cluster with Munin

Monitor your cloud cluster with Munin

Munin is a great tool for giving you graphical information about the health of your servers, but deploying it in a cloud environment can be a little tricky. Unfortunately, solving that problem can sometimes be a lot easier than getting Munin setup at all. First, I’ll walk you through setting up Munin and some debugging steps in case things go wrong, then we’ll tackle getting stats from those elastic servers.

Configuration

As usual, I will be making the assumption you’re using Ubuntu. If you’re not, you will have to make adjustments accordingly.

Installation

Server config

Paste the following in to your config file.

  Save and exit

There is nothing to restart for your Munin server. Every 5 minutes it will launch, load your configuration and execute, there is no daemon process.

Static servers config

You will have some static servers, like the one that hosts your stats, or your databases. These servers will likely grow or stay static, but never decrease. You will need to create a separate config for these servers, so that it does not collide with your dynamic ones.

 Save and Exit

You can create a new grouping by changing the name of the tag before the semi-colon. You can group multiple servers this way, and it makes viewing your server list easier.

Node Config

Save and Exit

Your node will poll your server either every second, every few seconds or every minute depending on the plugin, but those stats aren’t really useful unless you have a large data set to work with so it will take 5 minutes for your Munin server to display anything meaningful — assuming it works at all!

init.d script for your dynamic graphs

I’m not too sure the reason this doesn’t ship as standard with Munin, possibly because it’s slightly different from distribution to distribution. Anyway, I found this online and set it up.

nginx

Munin comes with Apache support built in, but I don’t use Apache at all so here’s an nginx setup.

Debugging

Debugging in Munin can be a serious headache. The log files for your node can be found at /var/log/munin/munin-node.log, and it will usually be spitting some kind of vague Perl error. You’ll have to do your best to decipher what it means.

Node shows, but graphs don’t work

So, you’ve got Munin installed and you can see it in your web browser, but graphs don’t appear to be working? First things first. Swap to the munin user and try to run the update script and see if it gives you any information to work with.

Personally, when I did this I found this error.

ERROR: Cannot open ‘/etc/munin/munin.conf’: Permission denied at /usr/share/munin/munin-update line 49

Which meant that I had set my permissions wrong on the munin.conf file.

 I can’t see any nodes!

This could be for a few reasons, and it’s one of the harder ones to diagnose. The first thing you need to do is test a connection to your node. For me, the node I was trying to setup first was the local node, the one that monitors my stats server.

Check your connection

There are two things to pay attention to here. The first being Connected to localhost.localdomain. This is the name my node is representing its self as, which wasn’t the name I had told the server to look for when connecting to that address. I had to fix my /etc/hosts file, to make stats-server my primary loopback domain.

The second thing to notice is Connection closed by foreign host. This was caused by an incorrect line for allowed host, and so my node-server thought that there were no-hosts to allow so it was refusing every single connection.

This is how it should look once you have it configured correctly.

Munin & Elastic servers

You will need to replicate the Munin Node steps and have it installed and working on all of your elastic servers.

I will give you the script that I use to be able to collect stats from all my currently active servers, but it will only work if you are using Digital Ocean. I don’t use Rackspace, or EC2 because they’re not worth the money, but that’s another point for another article. So, if you’re not using Digital Ocean then you will need to make your changes accordingly.

This script will login to Digital Ocean API and retrieve any web server with the host name “web.” Then, it will produce a config file which is stored in Munin’s folder for dynamic configs.

That’s it, you’re done. Your Munin server will attempt to connect to these and retrieve information. If you want more complex things like Aggregated information, or Totals then you can generate custom graphs, but that’s a whole other World of problems which I will cover at a later date. For now, this is a good place to start!

Dave

About David Parlevliet

Dave is long time developer with a passion toward teaching. He divides his time between his wife, her cat and his projects. He recently started using twitter so make sure to follow him!