freebsd hosting

Monitoring your servers with Monit

Monitoring your servers with Monit

If you have ever had an unwanted incident on your server - cores crashing, filesystem getting full or ddos attacks- you know that finding out what went wrong can sometimes turn into one hell of a detective job, especially for those of us who don't have a degree in Computer Science. Sure there are some nifty command line monitoring tools (htop and bpytop come to mind) but when you have several machines to take care if it can get quite impractical.

As we progress in the journey to become a skilled server administrator, it becomes necessary to use a remote monitoring tool that gives us some visibility on our systems. And sure there is a myriad of paid options featuring flashy animated dashboards out there, but they mostly belong to the cash cow realm of SaaS which I purposedly try to stay away of. So today I am going to introduce you to a free monitoring tool, monit, and to its commercial dashboard, which the programmers decided to name mmonit, creating a bit of confusion in the process.

You may think that there is already enough documentation out there about tools like this and yet that's not the case. As well crafted as monit and mmonit are, the documentation is somewhat lacking, particularly when it comes to FreeBSD. So I wanted to write down my experience in getting this tool to play nice with the operating system of the connoisseurs.

Monit

Monit is a very configurable daemon that can check virtually anything on our system -bandwidth, running processes, disk usage…- or even run custom scripts that extend its capabilities, and send us an e-mail to alert when, for example, we are running out of space, or our web server crashes. Here's how an e-mail from monit looks:

Connection failed Service nginx

Your faithful employee, Monit

And here is how monit logs this event itself:

Date:        Sat, 04 Apr 2020 18:05:26
Action:      alert
Host:        mydomain.xyz
Description: failed protocol test [HTTP] at [mydomain.xyz]:443 [TCP/IP TLS] -- HTTP: Error receiving data -- Operation timed out

Let's install it on our system:

pkg install monit

And then let's get our configuration ready.

cp /usr/local/etc/monitrc.sample /usr/local/etc/monitrc
ee /usr/local/etc/monitrc

I will detail here the options we should concern ourselves about - you can leave the rest as default.

First is the mail server which we are going to use to submit mail alerts; I entered here my own mail server.

set mailserver mail.mydomain.xyz, # primary mailserver
localhost # fallback relay

Next goes the e-mail address where we want to receive the alerts:

set alert me@**mydomain.xyz** not on { instance, action }

Now we configure the web interface. Here we set a port, user and password to connect.

set httpd port 2812 and
allow **someuser**:**somepassword**

Finally we get to the substance - the monitoring rules. There are plenty of examples in the file, but I will drop my own configurations here as well as they are guaranteed to work with FreeBSD.

Checking the overall system health

check system $HOST
if loadavg (1min) per core > 2 for 5 cycles then alert
if loadavg (5min) per core > 1.5 for 10 cycles then alert
if cpu usage > 75% for 2 cycles then alert
if memory usage > 75% for 2 cycles then alert
if swap usage > 75% for 2 cycles then alert

This one is self explanatory. You have to find out yourself through experience what do you consider an abnormal usage of resources. Watching the web interface for a while should give you an idea.

Monitoring a web server

check process nginx with pidfile /var/run/nginx.pid
start program = "/usr/local/etc/rc.d/nginx start"
stop program = "/usr/local/etc/rc.d/nginx stop"
if failed host www.mydomain.xyz port 80 protocol http and request '/index.html' then alert
if failed host patch.mydomain,xyz port 80 protocol http and request '/index.php' then alert
check process php-fpm with pidfile /var/run/php-fpm.pid
start program = "/usr/local/etc/rc.d/php-fpm start"
stop program = "/usr/local/etc/rc.d/php-fpm stop"
if children > 30 then alert

This one is a bit trickier. We test two domains we host on this machine and both the nginx and php-fpm processes - they will be restarted if down.

Check network interface

This could warn us about DDoS attacks that rely on flooding the connection.

check network **ix0** with interface **ix0**
if failed link then alert
if saturation > 50% for 2 cycles then alert

Replace ix0 with your network interface - you can find out its name with ifconfig.

Check filesystem usage

So we don't run out of disk space unexpectedly

check filesystem zroot/ROOT/default with path /
if space usage > 75% then alert

Check some game processes

So we will get alerts on a core crash. Tailor with your own paths:

check process db with pidfile /home/game/db/pid.db
start program = "/home/game/db/run.sh"
stop program = "/home/game/db/shutdown.sh"
mode passive
check process auth with pidfile /home/game/auth/pid.auth
start program = "/home/game/auth/run.sh"
stop program = "/home/game/auth/shutdown.sh"
mode passive
check process channel1_game1 with pidfile /home/game/channel1/game1/pid.game
start program = "/home/game/channel1/game1/run.sh"
stop program = "/home/game/channel1/game1/shutdown.sh"
mode passive
check process channel1_game2 with pidfile /home/game/channel1/game2/pid.game
start program = "/home/game/channel1/game2/run.sh"
stop program = "/home/game/channel1/game2/shutdown.sh"
mode passive
check process channel2_game1 with pidfile /home/game/channel2/game1/pid.game
start program = "/home/game/channel2/game1/run.sh"
stop program = "/home/game/channel2/game1/shutdown.sh"
mode passive
check process channel2_game2 with pidfile /home/game/channel2/game2/pid.game
start program = "/home/game/channel2/game2/run.sh"
stop program = "/home/game/channel2/game2/shutdown.sh"
mode passive

Note that I add "mode passive" because I do not want monit to restart the process if it's down. I only want the alert sent to me. If you want monit to do that, just remove that line for each core.

Also note that channel1_game1 are just names I give them so I can recognize them in the web interface; what really matters here is the path for the pid file. Other than that, there are plenty of examples in that file to get you started.

Once we are happy with our settings, let's save the file and start the service:

service monit start If we set everything properly, we should start getting e-mail alerts on our defined events. It may take some finetuning for you to discover which loads are out of the ordinary for your server and you should be warned about. For example, a simple web server with a 1 Gbps connection having 25% saturation is not normal, but if that web server is being used as a file server, then you will probably average more than that. You may also want to use the web interface:

http://:2812

M/Monit

M/Monit is a logging application which collects data from any number of monit hosts and stores it to keep statistics, simplifying the task of watching over several servers. It also allows for stopping and starting services remotely by setting a "stop program" and "start program" parameter for them in monitrc, and you can even show the output of a command from the process list using the "check program" directive in monitrc and hovering over the process name in the M/Monit list.

Now it's when things get pretty, with pie charts, responsive interfaces and whatnot. Take a look at this beauty:

M/Monit will take all the data produced by monit and present it in an easy to understand way, including analytics of RAM, CPU, Disk usage and so on over time. It is an invaluable tool in finding the source or symptom of problems quickly.

Now as I said at the beginning there's lots of pretty monitoring interfaces you can buy in the internet and M/Monit is not the prettiest but it has two qualities that make it stand out for me:

  • It works perfectly with FreeBSD, as it integrates with monit which has been a standard package in FreeBSD for many years.
  • It's not a SaaS cash cow charging you a fortune every month - you pay once and it's yours forever.

At the time of writing this a license for 5 hosts costs 65€ and you can buy it from https://mmonit.com/shop/

The hosts do not need to be FreeBSD, it can be any mix of systems, but I will explain here how to install it on FreeBSD because the official documentation for our system is, as it happens often, quite sparse. If you want to use it for Linux, refer to the official documentation.

While monit should be installed in every server you want to monitor, M/Monit only needs to be installed once. It can be in any of the servers you monitor on in a different one. Just make sure nginx is installed and your firewall rules allow traffic between the monit and mmonit instances (For monit 2812 and for mmonit: 8080). This M/Monit server should also be running a database - I will assume you have MySQL or MariaDB but SQLite and PostgreSQL are supported too.

Once you purchased the package, download the latest version and extract it to your chosen server:

tar -C /usr/local -zxvf mmonit-x.x.x-freebsd-x64.tar.gz
ln -s /usr/local/mmonit- /usr/local/mmonit

This symbolic link allows you to keep the old versions and switch back in case of problems, so whenever you want to install a new version you just need to delete the symbolic link and then follow these same steps, just copying over your configuration files (found under the /conf subdirectory) to the new directory.

Nginx will act as a forwarding proxy in front of the M/Monit application. Let's set up /usr/local/etc/nginx/nginx.conf. This configuration assumes we have a LetsEncrypt certificate for our domain - more about this in an upcoming article.

server {
        listen 443 ssl http2;
        listen [::]:443 ssl http2; # If using IPv6
        server_name **mydomain.xyz**;

    root /usr/local/www/nginx;

            location / {
                    proxy_set_header Connection "";
                    proxy_set_header Host $http_host;
                    proxy_set_header X-Real-IP $remote_addr;
                    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                    proxy_set_header X-Forwarded-Proto $scheme;
                    proxy_set_header X-Frame-Options SAMEORIGIN;
                    proxy_pass http://127.0.0.1:8080;
            }

    ssl_certificate /usr/local/etc/letsencrypt/live/**mydomain.xyz**/fullchain.pem;
    ssl_certificate_key /usr/local/etc/letsencrypt/live/**mydomain.xyz**/privkey.pem;

    ssl_dhparam /usr/local/etc/letsencrypt/dhparams.pem;

    ssl_session_timeout 1d;
    ssl_session_cache shared:le_nginx_SSL:1m;
    ssl_session_tickets on;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;

    add_header Strict-Transport-Security "max-age=63072000" always;
}</code></pre>


Now let's edit the M/Monit config file at /usr/local/mmonit/conf/server.xml


These are the parts to change - remember to delete comment out with HTML style comments () the Connectors and Realms you don't use:






As usual with my tutorials this configuration is intended for MariaDB. For MySQL you may need to change the socket patch in Realm url to /tmp/mysql.sock.


Edit the file /usr/local/mmonit/conf/server.xml and paste the license key you obtained with your purchase.


Now login to mysql and create the database and user for M/Monit


CREATE DATABASE mmonit;
CREATE USER 'mmonit'@'localhost' IDENTIFIED BY '**somepassword**';
GRANT ALL PRIVILEGES ON mmonit.* TO 'mmonit'@'localhost';


Now let's add a startup script for the service by creating the following file as  /usr/local/etc/rc.d/mmonit


#!/bin/sh

PROVIDE: mmonit


REQUIRE: LOGIN nginx


KEYWORD: shutdown




Add the following line to /etc/rc.conf to enable M/Monit




mmonit_enable="YES"


. /etc/rc.subr


name="mmonit"
desc="M/Monit monitoring hub"
rcvar=mmonit_enable


load_rc_config $name


: ${mmonit_enable:="NO"}


command="/usr/local/mmonit/bin/mmonit"
command_args=""
start_precmd="mmonit_checkconfig"
stop_cmd="mmonit_stop"
start_cmd="mmonit_start"


mmonit_checkconfig () {
echo "Performing sanity check on M/Monit configuration:"
eval ${command} ${mmonit_flags} -t
}


mmonit_start () {
echo "Starting M/Monit:"
eval ${command} ${mmonit_flags} start
}


mmonit_stop () {
echo "Stopping M/Monit:"
eval ${command} stop
}


run_rc_command "$1"


Make it executable:


chmod 777 /usr/local/etc/rc.d/mmonit


Before we run the service, we should let the monit instances that we set up in the first part know how to communicate with mmonit. So edit the /usr/local/etc/monitrc in each server where you installed it and uncomment & edit the following line:


set mmonit https://monit:@**mydomain.xyz**:8080/collector


In order to issue commands from MMonit to Monit, such as stopping a service remotely, we also need to do the following change in monitrc (once again if you don't have your own domain you can replace mmonit.mydomain.com with the M/Monit server IP)


set httpd port 2812 and
    allow 127.0.0.1      
    allow **mydomain.xyz**
    allow monit:**somepassword**


Now let's go back once again to the server where we just installed M/Monit and add this in /etc/rc.conf to auto start mmonit.


mmonit_enable="YES" 


And start the service with:


service mmonit start


Now we should be able to browse to the IP or hostname we configured previously.


http://mmonit.mydomain.com:8080/
If everything is done properly, we will get those pretty statistics right in our browser. You can read more about mmonit in the official manual:


https://mmonit.com/documentation/mmonit_manual.pdf
← Back to Blog