Log management is a practice that includes collecting, aggregating, storing, rotating and analyzing a large set of log files that are generated by various computer programs and systems. Log management is important because it’s essential in monitoring both internal and external events happening on the deployed systems. What happened, who did what, when and how? All of those questions need to be immediately answered in a lot of deployed systems and infrastructures in the world.

In some cases, even the law requires some sort of log management capabilities in the software before it can be used on official government equipment, such as HIPAA and others.

What is a Log Management Software?

Now, a complete log management software monitors a set of the available log files on the system, parses it, aggregates it, and then finally displays it for you in a specific way you choose so that you can make sense of those huge log files.

Log management software is not just a basic program that reads data from text files and creates a chart. It involves many intricate components. The process of log management involves handling large quantities of log files from multiple sources on a daily basis.

The ultimate goal is to have a comprehensive overview of all the log files in real time and the ability to access a specific log or event anywhere in the infrastructure at any time. This may include managing 1GB of nginx logs from 100 servers each day, as an example.

As usual, there are tons of proprietary closed-source log management software that would charge you a lot of money via a monthly subscription based on the size of log files you analyze, the storage/rotation time, the number of users on the system you’ll have and many other criteria. Or, you know, you can try other free and open source solutions that you can use by your own.

In this list, we introduce 5 of them.

List of Good Open Source Log Management Software

1. Elastic Stack

Open Source Log Management 5

This is one of the most used solutions ever for log management (and many other things, actually). It’s called a “stack” because it’s not just one piece of software; It’s multiple ones. Elastic Stack consists of:

  1. Elastic Search: A powerful open source search engine.
  2. Kibana: Web-based visualization tool for any data you may have.
  3. Beats: It’s known as “data shippers”, which are simple programs that get installed on a large set of servers you have in order to continuously send log/monitoring data about each machine into a one united repository of data.
  4. Logstash: This one “united repository of data” above is Logstash, it’s the place where log files are collectively going to be stored, parsed, filtered and analyzed.

So now you may be wondering, how are all of those things connected? Well, what are you going to is that first, you are going to install Beats on all the machines you want to monitor, and then configure them to send data into a central machine where you would set up Logstash to do the actual log analysis job, and then integrate the data into Elastic Search in order to be able to search for anything specific in that huge data, or run other tasks such as machine learning, auto alerting and other Elastic Search features. Finally, you would integrate Kibana into Elastic Search in order to be able to see various useful kinds of visualizations about your machine log files.

Although it may sound like a huge effort for you in the beginning, we can assure you that the Elastic Stack is one of the best DevOps tools in the entire market. If you are looking for an enterprise-grade open source solution for log management, then this is the one.

For more information, visit the Elastic Stack homepage.

2. Graylog

Open Source Log Management 7

Another powerful open source log management software is Graylog. Unlike the Elastic Stack, this one is made essentially to just log management, so it’s a specialized software. Graylog offers some premium enterprise solutions for those willing to pay, but also offers a fully open source version that you can self-host.

It’s very easy and quick to install, as it provides packages for all the modern operating systems besides a Docker container. Graylog offers a very clean user interface, along with many features such as advanced search (To build powerful queries and run them quickly), alerts feature, fault tolerance (To avoid losing data in case of network issues), integrations with the most famous automation services (Chef, Puppet, Ansible), REST API, powerful documentation and much much more. It’s the second best solution in this list.

Graylog is written in Java, and is licensed under the GPL 3 license. The software is well-supported and continuously updated, and there are a lot of plugins that are provided by the parent company itself as free and open source too.

You can view Graylog repositories on GitHub, or learn more about it from their official website, or check the documentation for details.

3. LOGalyze

Open Source Log Management 9
LOGalyze User Interface (image from official website)

LOGalyze is another software that comes to mind when talking about open source log management. The company is based in Hungary and provides web-based, graphical open source log management software that is written in PHP. And while the program hasn’t been updated in a year, it still works as expected.

It’s also compliant with many law regulations related to data processing, such as HIPAA, PCI DSS, Sarbanes-Oxley act and PZSAF-HPT. And it provides log management in real time, along with graphs and visualization generation, alerts and notifications features, and support for many various input/output formats.

LOGalyze is capable of monitoring Windows systems and Linux distributions, network devices, various firewall logs, Oracle audits besides a lot of system-specific applications (E.g nginx on Linux, XAMPP on Windows, MySQL..).

For download instructions and more information, visit LOGalyze’s official homepage.

4. GoAccess

Open Source Log Management 11

Unlike many other mentioned software in this list, GoAccess was built in the first place to be a terminal-based log manager, meaning that it runs inside your terminal emulator. Despite so, GoAccess provides a very beautiful web-based user interface that you can run from inside your browser.

GoAccess is written in the C programming language and licensed under the MIT license. Its main features are:

  • All information it displays to you are in real-time.
  • Support for almost all log formats, such as Apache, Amazon S3, Nginx and Cloudfront.
  • No dependencies except ncurses; This means that you do not need to install any other libraries and tools for GoAccess to work. You just need to install it along with Ncurses.
  • Detailed tracking of application response time, visitor’s time on the page, visiting countries, hits, bandwidth and much more.
  • Customizable color scheme for the user interface.
  • Multi virtual hosts support.
Open Source Log Management 13
GoAccess in a terminal, via goaccess.io

GoAccess is a very good option if you want to monitor one server/machine. It can be installed in no time and configured instantly to feast on your log files. Since it was written in C, prebuilt binaries do exist for all the major Linux distributions, and even some BSD derivatives such as OpenBSD and FreeBSD.

You can download it from its official GitHub repository, or learn more about it from its website.

5. Nagios Open Source

Open Source Log Management 15

Our final software in this list is Nagios. It is a full open source log management program that combines both modularity and extensibility. The community around it is huge, and does a lot of development for the software.

First, you are going to install Nagois Core, which would give you the basic log management functionalities with a very simple user interface. But later, you’ll discover that you can install more than 4,000 different plugins for various logging tasks, along with tons of other types of user interfaces that you may find more useful than the official default one.

Nagios is written in in the C language, and licensed under the GPL 2 license. It can monitor anything that comes to your mind: Network services (HTTP, SMTP, POP3..), machine resources (Disk, RAM, CPU), and any application that you have on your machines. The tricky thing about Nagios is that it may take you some time to fully configure it and install all the other plugins and frontends to address your needs.

To learn more about Nagios, visit its official website or GitHub repository.

Conclusion

So you have seen our recommendations for open source log managers so far in this list. There are many other log daemons and other servant tools that weren’t mentioned in this list, as we aimed to mention only the full solutions that can enable you to immediately start monitoring after finishing the setup of these tools.

If you have any other similar software, share it with us in the comments.

If you’re looking for a managed logging platform then consider Hosted ELK & save time spent on server maintenance.

Subscribe
Notify of
guest

0 Comments
Inline Feedbacks
View all comments

Newsletter

Enter your email address to subscribe to our newsletter. We only send you an email when we have a couple of new posts or some important updates to share.

Recent Comments

Open Source Directory

Join the Force!

For the price of one cup of coffee per month:

  • Support the FOSS Post to produce more content.
  • Get a special account on our website.
  • Remove all the ads you are seeing (including this one!).
  • Get an OPML file containing +70 RSS feeds for various FOSS-related websites and blogs, so that you can import it into your favorite RSS reader and stay updated about the FOSS world!