Manage Learn to apply best practices and optimize your operations.

How to use the ELK stack log monitoring tool

Open source log monitoring tools are a budget-friendly way for organizations to increase network visibility and improve incident response times.

Security is gaining importance in the IT world, in large part because the number of cyberattacks has mushroomed...

in the past couple of years. Security breaches are becoming more common, and most importantly, more targeted. For example, a recent attack on an Italian security company, the Hacking Team, resulted in 400 GB of data stolen and leaked on the Internet.

Despite many countermeasures in place, attackers are still penetrating enterprises' security defenses. Security is like an onion with multiple layers, and the more layers it has the more difficult it is to penetrate. While there are many cases where secret information is not properly secured, usually the more security layers in place, the more sensitive the information. So despite many security layers being actively used by enterprises, security breaches not only still occur, but are becoming more widespread because the attacks are becoming more targeted than ever.

Once an attacker has penetrated the security defenses, there are many actions that could be executed in the enterprises internal networks. After penetrating the security defenses, the primary concern of an attacker is continuous persistent access to the enterprise's internal network, usually by being unnoticed by system administrators or security professionals.

For this reason, it is beneficial for an enterprise to be able to detect the presence of an attacker and possibly trigger different alerts incorporated into the security policy, such as unusual server requests or suspicious traffic. Security teams can gather security-related log information for both on-premises and cloud infrastructure and analyze them for intruder alerts. There are many different enterprise log monitoring tools available to help gather log data, including Splunk, Fluent, AlientVault and Simple Event Correlator. Let's take a look at an open source log monitoring option: Elasticsearch, Logstash and Kibana, more commonly known as the ELK stack. The stack can be integrated and applied to cloud environments such as Amazon Web Services and Google Compute Engine in addition to on-premises networks.

Companion article

See Infosec Institute's accompanying article on Cloud Security Monitoring with Open-Souce Tools

The ELK stack includes Logstash, a data pipeline that is used to process logs in different formats. Logstash uses different rules to format each log message into multiple fields, which are indexed by the Elasticsearch search engine. Kibana is a Web interface that provides an overview of the collected data, so users can easily view and analyze the collected logs.

Setting up the ELK stack log monitoring environment

Typically setting up the ELK stack environment involves mundane work like downloading the executables, tables or archives to install the necessary applications. The environment setup usually takes a long time, but for our purposes, Protean Security has already done all the heavy lifting. It has provided a Vagrantfile -- as well as a Dockerfile -- to set up the ELK stack running the latest versions of Elasticsearch 1.6.0, Logstash 1.5.2 and Kibana 4.1.1 by using Vagrant and Docker.

The images can be pulled from the Web, so it's possible to start working with them immediately without manual installation or configuration.

To get started with Docker, pull the docker-elk repository from Github with this command:

[plain]

# docker pull proteansec/elk

[/plain]

This will automatically start all the needed services since the image has been programmed in such a way that the users don't need to start anything manually.

To get started with Vagrant, pull down the vagrant-elk repository from Github and issue the "vagrant up" command:

[plain]

# git clone https://github.com/proteansec/vagrant-elk

# cd vagrant-elk/

# vagrant up

[/plain]

The ELK stack environment should be built automatically.

Getting the logs

After the ELK stack environment has been successfully set up it's time to start incorporating security logs into ELK.

Logstash starts on TCP port 5514 and accepts sys log messages, which correspond to the sys log message format. This is why it's possible to pipe most of the files from /var/log/directory directly into Logstash for processing.

For testing purposes, pipe a sys log message to the Logstash TCP port 5514 using the netcat program. Here's an example of this command:

[plain]

# echo "Oct 31 11:31:50 user kernel: [525887.354176] grsec: denied resource overstep by requesting 4096 for RLIMIT_CORE against limit 0" | nc localhost 5514 -q 1

[/plain]

Usually, this isn't done manually, but it's helpful to know how to pipe a sample log message manually to logstash for processing -- it provides a deeper understanding of the underlying technology. After sending a log message to Logstash it is automatically visible in the Kibana Web interface, as shown below. The log message has been successfully parsed into multiple fields, which allows for easy indexing and searching using Elasticsearch capabilities.

To automatically feed all generated log information into Kibana, install a rsyslog deamon and set it up to send all messages to Logstash listening on TCP port 5000.

At this point, the majority of the work is done. The last step is to ensure the logs from all the servers are sent to Logstash, where they are parsed and indexed by Elasticsearch. Using Kibana also provides a nice Web interface that presents the logs through a Web application where simple queries can also be executed to present the interesting log messages.

Benefits of the ELK stack log monitoring tool

The ELK stack for open source log monitoring is a great option for implementing central log monitoring software on a budget. Using an open source tool requires more time to set up the environment and learn about the system, but the results are rewarding, both money-wise and also because of the opportunity to understand the underlying system.

Managing the logs centrally and all in one location provides insight into the network and helps to detect and block malicious activity early in the attack. That further prevents attackers from being on the enterprise network unnoticed for many days, weeks or even months. By incorporating the ELK stack into network defenses, enterprises not only gain insight into their system activities, but also make the life of an attacker far more difficult.

Centrally managing the logs can also be used in forensic analysis to track down the time of intrusion, as well as the method used to penetrate into the network. Consequentially it also gives us a way to monitor attacker activities in the network, and most importantly, it might be a big help in determining the reason behind an attack.

The next feature on log monitoring will discuss how to incorporate cloud security logs into these open source log monitoring tools.

About the author:
Dejan Lukan has an extensive knowledge of Linux/BSD system maintenance, as well as security-related concepts including system administration, network administration, security auditing, penetration testing, reverse engineering, malware analysis, fuzzing, debugging and antivirus evasion. He is also fluent in more than a dozen programming languages, and regularly writes security-related articles for 
his own website.

Next Steps

Learn how to strategically use open source security tools to stretch your budget and how to know when you should use open source tools instead of commercial products

Check out a tutorial on how to use NDPMon for better monitoring and network visibility

How to keep track of leaked data with Web traffic monitoring tools

This was last published in October 2015

Dig Deeper on Cloud Computing Security Issues: Incident Response - Data Breach Prevention

Join the conversation

3 comments

Send me notifications when other members comment.

By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy

Please create a username to comment.

Does your enterprise use open source log monitoring tools?
Cancel
We use what works. If our log management software is open source and it works, so be it. If it's proprietary (which usually only happens with larger, more compliance-focused businesses and industries) and it works, so be it too. I'm easily satisfied when it comes to technology. If it does the job and doesn't create more hurdles or costs, then it's a solution we're going to use until the next great solution comes along. How do you feel about that? Who wants to spend money where they don't have to?
Cancel
Yes, both for logging and log analysis. As Jeff Cutler said, no need to reinvent the wheel.
Cancel

-ADS BY GOOGLE

SearchSecurity

SearchCloudComputing

SearchAWS

SearchCloudApplications

SearchServerVirtualization

SearchVMware

ComputerWeekly.com

Close