System monitoring with email notifications

I currently have an Ubuntu Server 16.04 LTS and I want to be able to receive email notifications with information about the resource usage on my server. CPU, memory, swap, you name it.

I understand that I can use Glances for local monitoring but I won't be on the console looking at the information all the time. I just need email notifications that inform me of resources that hit above normal/critical values and get notified.

Does anyone know of a method that allows me to do this?

I'll give you a small list of tools - while I would personally recommend Nagios simply as I am used to it. It comes with a big user base, good collection of plug-ins etc - but I've heard good things about Icinga as well.

Be aware that setting up Nagios or Icinga will take some time until you have a perfectly working setup which covers / monitors all your existing services. The start is most likely not that easy, but once you got it working it's a lovely setup which will notify for all your defined services if states do change so something which is reportable/defined as critical

Project names & links

First of all - a quick overview:

Client & Server Setups

• Nagios
• Munin
• Icinga
• Spiceworks
• Monit
• Cacti
• Zabbix

Other interesting software

• Smokeping (limited function-wise)

Single host setups

• Glances
• Monitorix

More Information

About Nagios

Wikipedia writes

Nagios /ˈnɑːɡiːoʊs/, now known as Nagios Core, is a free and open source computer-software application that monitors systems, networks and infrastructure. Nagios offers monitoring and alerting services for servers, switches, applications and services. It alerts users when things go wrong and alerts them a second time when the problem has been resolved.

Example image - showing 1 of several views you have:

Here is a good AskUbuntu post about How to Install Nagios. Nagios Exchange offers a huge collection of plugins, addons, documentation, extensions, and more.

About Munin

Wikipedia writes

Munin is a free and open-source computer system monitoring, network monitoring and infrastructure monitoring software application. It offers monitoring and alerting services for servers, switches, applications, services, etc. It alerts the users when things go wrong and alerts them a second time when the problem has been resolved

Example Image

About Icinga

Wikipedia writes

Icinga is an open source computer system and network monitoring application. It was originally created as a fork of the Nagios system monitoring application in 2009.

Icinga is attempting to get past perceived short-comings in Nagios' development process, as well as adding new features such as a modern Web 2.0 style user interface, additional database connectors (for MySQL, Oracle, and PostgreSQL), and a REST API that lets administrators integrate numerous extensions without complicated modification of the Icinga core.

Example Image showing 1 of several views:

About Spiceworks

Wikipedia writes:

Spiceworks is a professional network for the information technology (IT) industry that is headquartered in Austin, Texas. The company was founded in January 2006 by Scott Abel, Jay Hallberg, Greg Kattawar, and Francis Sullivan to build IT management software.

Spiceworks is an online community where users can collaborate and seek advice from one another and also engage in a marketplace to purchase IT-related services and products. The network is estimated to be used by more than six million IT professionals and 3,000 technology vendors.

About monit

Wikipedia writes

Monit is a free, open source process supervision tool for Unix and Linux. With Monit, system status can be viewed directly from the command line, or via the native HTTP(S) web server. Monit rose to popularity with Ruby on Rails and the Mongrel web server,[citation needed] because a tool was needed that could manage the many identical Mongrel processes that needed to be run to support a scalable Ruby on Rails site, and Monit was fairly uniquely suited for the needs of the Ruby on Rails community. Many popular Rails sites have used Monit, including Twitter and scribd.

About Cacti

Wikipedia writes:

Cacti is an open-source, web-based network monitoring and graphing tool designed as a front-end application for the open-source, industry-standard data logging tool RRDtool. Cacti allows a user to poll services at predetermined intervals and graph the resulting data. It is generally used to graph time-series data of metrics such as CPU load and network bandwidth utilization. A common usage is to monitor network traffic by polling a network switch or router interface via Simple Network Management Protocol (SNMP).

The front end can handle multiple users, each with their own graph sets, so it is sometimes used by web hosting providers (especially dedicated server, virtual private server, and collocation providers) to display bandwidth statistics for their customers. It can be used to configure the data collection itself, allowing certain setups to be monitored without any manual configuration of RRDtool. Cacti can be extended to monitor any source via shell scripts and executables.

About Zabbix

Wikpedia writes:

Zabbix is enterprise open source monitoring software for networks and applications, created by Alexei Vladishev. It is designed to monitor and track the status of various network services, servers, and other network hardware.

Zabbix uses MySQL, PostgreSQL, SQLite, Oracle or IBM DB2 to store data. Its backend is written in C and the web frontend is written in PHP. Zabbix offers several monitoring options

You can use several solution. One of them is monit.

From command line, just run the following command:

sudo apt-get install monit

Monit should now be installed and accessible through one of the following URLs:

http://localhost:2812
http://IPADDRESS:2812 (local network IP)
http://domain.com:2812 (if you have domain name pointing to your server)

Monit Configuration

Before you can start using Monit for automatic server monitoring, you have to do some basic configuration. First backup the existing default Monit configuration using the following command:

sudo mv /etc/monit/monitrc /etc/monit/monitrc.bak

Next, create a new monitrc file using the command

sudo /etc/monit/monitrc 

and copy the following contents to it.

# How often in seconds should monit check your services.
set daemon 120

set logfile /var/log/monit.log
set idfile /var/lib/monit/id
set statefile /var/lib/monit/state

# Configure your SMTP out server. 
set mailserver smtp-server.columbus.rr.com port 25,localhost

set eventqueue
    basedir /var/lib/monit/events # set the base directory where events will be stored
    # optionally limit the queue size
    slots 100 

# Use one of the following 2 lines. The second line alerts on every little change and can be annoying.
set alert [email protected] but not on { instance, pid, ppid } #does not send alert on pid changes
#set alert [email protected] 

set httpd port 2812 and
    #Change username and password
    allow Username:Password
    # To enable SSL for WebUI uncomment the next 2 lines
    #ssl enable
    #pemfile /path/to/unified/certificate.pem
    # To restrict access to localhost only uncomment the following line
    #allow localhost

include /etc/monit/conf.d/*

Lines that begin with a # are comments to help you customize the configuration. Make sure you have at least the admin email, SMTP server, and SMTP port configured correctly. You should also consider changing Monit WebUI username and password. If you have an SSL certificate you can enable that as well for HTTPS access

For now use the following commands to ensure Monit is working well.

To test Monit configuration for syntax errors:

sudo monit -t

To start Monit:

sudo monit

To check Monit Status:

sudo monit status

System Load Monitoring with Monit

It is required that you have a working Monit instance with a proper /etc/monit/monitrc file. Monit configurations for various services are loaded from /etc/monit/conf.d folder. To monitor server load with Monit, create a Monit configuration file using the following command:

sudo /etc/monit/conf.d/systemload

Copy the following contents to it, save, and exit

# domain.com could be IP, hostname, or localhost
check System domain.com
    if loadavg (1min) > 4 then alert
    if loadavg (5min) > 2 then alert
    if memory usage > 75% then alert
    if swap usage > 25% then alert
    if cpu usage (user) > 80% then alert
    if cpu usage (system) > 30% then alert
    if cpu usage (wait) > 20% then alert

This code will make Monit send you an email alert when one of the above conditions (eg. average load is >4 for at least 1 min or when more than 75% RAM is full) are met. You can customize the above rules as you please. Below is an example email alert sent by Monit along with a description of what condition caused the alert.

Test and Reload Monit

Once you make any changes you have to test Monit configuration:

sudo monit -t

You should see the following message: Control File Syntax OK.

Then, check to see if Monit is already running using the following command:

sudo /etc/init.d/monit status

If Monit is running, reload Monit configurations using the following command:

sudo /etc/init.d/monit reload

Now, fire up your web browser and visit one of the following URLs depending on how your Monit is configured (be sure to use the correct port number):

http://localhost:2812
http://IPADDRESS:2812 (local network IP)
http://domain.com:2812 (if you have domain name pointing to your server)

You should see the system status, load, CPU load, Memory Load, and Swap load

Storage Monitoring with Monit

Next, it is required that you have a working Monit instance with a proper /etc/monit/monitrc file. Monit configurations for various services are loaded from /etc/monit/conf.d folder. For drive space monitoring with Monit, create aMonit` configuration file using the following command:

sudo /etc/monit/conf.d/storagespace

Copy the following contents to it, save, and exit

# add each drive you want to monitor below
check filesystem Ubuntu with path /dev/sda1
    if space usage > 90% then alert
check filesystem Home with path /dev/sda3
    if space usage > 90% then alert
check filesystem Media with path /dev/sdb1
    if space usage > 90% then alert

The code above code will monitor hard drive space in 3 partitions in 2 drives. If any of them is more than 90% full you will got alert.

Also you can monitoring a specific services.

Monitor Apache Server with Monit

Setting up Apache server monitoring with Monit is with Monit’s pre-made configuration templates. All you have to do is copy the existing template from /etc/monit/monitrc.d to /etc/monit/conf.d folder.

sudo cp /etc/monit/monitrc.d/apache2 /etc/monit/conf.d/

Instead of copying, you may also create a symbolic link. Apache web server process creates apache2.pid. The above code monitor apache2.pid file and if does not exist, Monit will try to restart Apache. A restart will trigger an email alert. If restart fails multiple times then Monit stops monitoring Apache server.

...

You can also have a look at SeaLion You can set customized alerts for CPU usage, memory etc. In addition, you can also add your own metrics and configure notifications for them as well. Also, you can schedule the commands to run at specific intervals and see their outputs in a timeline display. You can find more about it here.

The above tools are all great in generating e-mails. I prefer to get e-mails for my daily monitoring since I don't want to sit and watch a dashboard all day. The problem is you get flooded by e-mails and you are only interested in when something goes wrong. Also how do you know if an e-mail for some reason was never delivered and you think everything is working fine.

I searched a long time and came across 2 products that can read your report e-mails and warn you when you need to take action.

1. PRTG you can setup mail filters to search for e-mail containing certain words. Was a bit complicated to do.
2. An online tool called wwww.smtpviewer.com actually does the job plus it has a timeout setting which will warns you when e-mail from your monitored devices has not arrived for a set time.