Log management is a practice which includes collecting, aggregating, storing, rotating and analyzing a large set of log files that are generated by various computer programs and systems. Log management is important, because it’s essential in monitoring both internal and external events happening on the deployed systems. What happened, who did what, when and how? All of those questions need to be immediately answered in a lot of deployed systems and infrastructures in the world.
In some cases, even the law requires some sort of log management capabilities in the software before it can be used on official government equipment, such as HIPAA and others.
Now, a complete log management software monitors a set of the available log files on the system, parses it, aggregates it, and then finally displays it for you in a specific way you choose so that you can make sense out of those huge log files.
A log management software isn’t a simple program that reads data from text files and then generates a chart; There are a lot of complex aspects in the process. Log management is generally about dealing with large volumes of log files coming from multiple sources per day, and the desire to have a conclusion of all of these log files in real time, with the ability to reach a specific log/event anywhere anytime in the whole infrastructure (things like 1GB worth of nginx logs coming from 100 servers per day, for example).
As usual, there are tons of proprietary closed-source log management software that would charge you a lot of money via a monthly subscription based on the size of log files you analyze, the storage/rotation time, the number of users on the system you’ll have and many other criteria. Or, you know, you can try other free and open source solutions that you can use by your own.
In this list, we introduce 5 of them.
This is one of the most used solutions ever for log management (and many other things, actually). It’s called a “stack” because it’s not just one piece of software; It’s multiple ones. Elastic Stack consists of:
- Elastic Search: A powerful open source search engine.
- Kibana: Web-based visualization tool for any data you may have.
- Beats: It’s known as “data shippers”, which are simple programs that get installed on a large set of servers you have in order to continuously send log/monitoring data about each machine into a one united repository of data.
- Logstash: This one “united repository of data” above is Logstash, it’s the place where log files are collectively going to be stored, parsed, filtered and analyzed.
So now you may be wondering, how are all of those things connected? Well, what are you going to is that first, you are going to install Beats on all the machines you want to monitor, and then configure them to send data into a central machine where you would set up Logstash to do the actual log analysis job, and then integrate the data into Elastic Search in order to be able to search for anything specific in that huge data, or run other tasks such as machine learning, auto alerting and other Elastic Search features. Finally, you would integrate Kibana into Elastic Search in order to be able to see various useful kinds of visualizations about your machines log files.
Although it may sound like a huge effort for you in the beginning, we can ensure you that the Elastic Stack is one of the best DevOps tools in the entire market. If you are looking for an enterprise-grade open source solution for log management, then this is the one.
For more information, visit the Elastic Stack homepage.
Another powerful open source log management software is Graylog. Unlike the Elastic Stack, this one is made essentially to just log management, so it’s a specialized software. Graylog offers some premium enterprise solutions for those willing to pay, but also offers a fully open source version that you can self-host.
It’s very easy and quick to install, as it provides packages for all the modern operating systems beside a Docker container. Graylog offers a very clean user interface, along with many features such as advanced search (To build powerful queries and run them quickly), alerts feature, fault tolerance (To avoid losing data in case of network issues), integrations with the most famous automation services (Chef, Puppet, Ansible), REST API, powerful documentation and much much more. It’s the second best solution in this list.
Graylog is written in Java, and is licensed under the GPL 3 license. The software is well-supported and continuously updated, and there are a lot of plugins that are provided by the parent company itself as free and open source too.
LOGalyze is another software that comes to mind when talking about open source log management. The company is based in Hungary and provides a web-based, graphical open source log management software that is written in PHP. And while the program hasn’t been updated in a year, it still works as expected.
It’s also compliant with many law regulations related to data processing, such as HIPAA, PCI DSS, Sarbanes-Oxley act and PZSAF-HPT. And it provides log management in real time, along with graphs and visualization generation, alerts and notifications feature, and support for many various input/output formats.
LOGalyze is capable of monitoring Windows systems and Linux distributions, network devices, various firewalls logs, Oracle audit beside a lot of system-specific applications (E.g nginx on Linux, XAMPP on Windows, MySQL..).
For download instructions and more information, visit LOGalyze’s official homepage.
Unlike many other mentioned software in this list, GoAccess was built in the first-place to be a terminal-based log manager, meaning that it runs inside your terminal emulator. Despite so, GoAccess provides a very beautiful web-based user interface that you can run from inside your browser.
GoAccess is written in the C programming language and licensed under the MIT license. Its main features are:
- All information it displays to you are in real-time.
- Support for almost all log formats, such as Apache, Amazon S3, Nginx and Cloudfront.
- No dependencies except
ncurses; This means that you do not need to install any other libraries and tools for GoAccess to work. You just need to install it along with Ncurses.
- Detailed tracking of application response time, visitor’s time on page, visiting countries, hits, bandwidth and much more.
- Customizable color scheme for the user interface.
- Multi virtual hosts support.
GoAccess is a very good option if you want to monitor one server/machine. It can be installed in no time and configured instantly to feast on your log files. Since it was written in C, prebuilt binaries do exist for all the major Linux distributions, and even some BSD derivatives such as OpenBSD and FreeBSD.
Our final software in this list is Nagios. It is a full open source log management program that is combines both modularity and extensibility. The community around it is huge, and does a lot of development for the software.
First you are going to install Nagois Core, which would give you the basic log management functionalities with a very simple user interface. But later, you’ll discover that you can install more than 4000 different plugins for various logging tasks, along with tons of other types of user interfaces that you may find more useful than the official default one.
Nagios is written in in the C language, and licensed under the GPL 2 license. It can monitor anything that comes to your mind: Network services (HTTP, SMTP, POP3..), machine resources (Disk, RAM, CPU), and any application that you have on your machines. The tricky thing about Nagios is that it may take you some time to fully configure it and install all the other plugins and frontends to address your needs.
So you have seen our recommendations for open source log managers so far in this list. There are many other log daemons and other servant tools that weren’t mentioned in this list, as we aimed to mention only the full solutions that can enable to you to immediately start monitoring after finishing the setup of these tools.
If you have any other similar software, share it with us in the comments.