JohanSwanepoel - Fotolia


How the cloud can help organizations with security log data

Organizations often have to deal with copious amounts of security log data and that can be challenging. Expert Frank Siemons explains how the cloud can help manage log data.

Something all information security controls have in common is the data output they produce in the form of logged events and alerts. With an increase in the size of an organization or an increase in security levels, the size of this security log data and its storage requirements will also rapidly grow.

The recent migration of many services to cloud service providers has created a few challenges for organizations dealing with these large amounts of data -- which can now be located externally within that same cloud platform. Fortunately, many of these CSPs have been quite active in this field, and some exciting new opportunities have opened up as well.

Analyzing security log data in the cloud

An organization with 1000 staff members and an average network size can easily generate 100 GB of logs in a single day. If most of that organization's environment is hosted within a cloud platform, analysis of that amount of data at the organizations local site with, for instance, a SIEM is nearly impossible. How would that data get synchronized fast enough to allow for real-time analysis? There is also potential for an attacker to delay or halt the data stream by generating a large amount of log data that results in a temporary lack of security monitoring. The most viable option here is to monitor and analyze the log data directly within the cloud platform. A possible hybrid option is to have a SIEM application or a simple log analysis application running on a cloud-based server and feeding some of the more interesting, correlated or filtered data back to the organization's on-premises environment.

Microsoft has released a whitepaper for its Azure platform covering items such as Azure Deployment Monitoring and Windows Event Forwarding. Amazon provides similar options, and most CSPs allow customers to deploy their own SIEM or Splunk related services without issues.

Download security log data from the cloud

The introduction of the SIEM as a service shows that the cloud security field is still very dynamic, and many more exciting developments in this space can be expected in the years to come.

Security log data can be downloaded from providers on a regular or an ad hoc basis, even if there is a significant amount. This data can then be fed into an on-premises SIEM, such as AlienVault or ArcSight for local analysis, and if needed, correlation with other event feeds. A regular download can be based on an API connection. This could be scheduled daily or frequently enough that it will appear as if the data is effectively synchronized continuously. This method is often used to obtain data for cloud-based security products as well, such as cloud antivirus and intrusion detection systems.

As mentioned, bandwidth usage and the potential for the data feed to be interrupted, as well as limiting security event visibility should be taken into account when planning for this setup. For compliance reasons or deep incident investigations, sometimes a bulk of months of data is required. Due to the sheer size of that data, a download would not be feasible. CSPs can usually assist with a customized, fitting solution as well. Amazon, for instance, has developed Snowball, which is a petabyte-sized, secure data transport tool designed to get large volumes of data into and out of AWS cloud. Other providers have similar options available because these bulk data requests are not uncommon.

Upload security log data to the cloud

Some organizations are not looking to download security data from the cloud; they need to upload it to the cloud environment instead. This could be the case if there is a SIEM product in their cloud environment. As mentioned, this is possible because some organizations produce much more security log data in their cloud environment than they produce locally. That locally produced log data will need to be uploaded to the cloud for analysis and correlation.

It could also provide a reliable form of off-site storage, for compliance or data redundancy purposes. An attacker can target security log data and having a secure, off-site copy is an information security best practice.

SIEM as a service

Dedicated third-party cloud-based security operations center (SOC) providers are also gaining popularity. Loggly is an example of an organization that allows customers to upload their security log data. The Loggly SOC monitors and analyzes that data and alerts the customer where needed. This setup is sometimes called SOC as a service or SIEM as a service (SaaS). There are more and more of these SaaS providers available every year, such as Alert Logic and Proficio and it is likely this trend will continue to grow. Using a SaaS provider means organizations do not need to set up their own, highly skilled 24/7 SOC at great expense. It is important to take into consideration, however, that the required bandwidth, service availability and possible compliance and local regulations mean that this system is not the best option for every organization.


The challenges with security log data that cloud customers had to deal with over the last years have mostly been addressed, with a wide range of tools and services now available. Most of these options create a hybrid-like cloud configuration where part of the data resides locally, and part of the data resides in the cloud. That data can and should be synchronized in one form or another by using the relatively easy upload and download options out there. The introduction of the SIEM as a service shows that the cloud security field is still very dynamic, and many more exciting developments in this space can be expected in the years to come.

Next Steps

Learn about best practices for security log management

Find out how to choose a service provider for cloud storage

Check out this introduction to SIEM services and products

Dig Deeper on Cloud Data Storage, Encryption and Data Protection Best Practices