How to effectively manage the cloud logs of security events

Cloud logs of security events produce an abundance of data. Expert Dave Shackleford discusses how to filter through it and get to the important security events.

As development and operations teams provision more resources to the cloud and enterprises leverage more cloud-based service offerings, security teams have to contend with an unanticipated issue -- a huge volume of new alerts and events.

A new report from cloud access security broker Skyhigh Networks Inc. offers some insight into the growth of cloud file sharing and collaboration services and how they are presenting new problems for security teams, namely in the form of "shadow cloud" and alert fatigue from all the events coming in.

The report noted that the average enterprise, according to Skyhigh's customer survey, generates around 2.7 billion cloud events per month, and file sharing/collaboration activity has been the biggest reason that number has skyrocketed. But the report also found that 2,500 of those cloud events, on average, constituted an "anomaly," and just 23 of them were actual security incidents. With this kind of volume, security teams are struggling more than ever to filter out the noise and respond to legitimate security events and suspicious incidents.

Sorting through cloud logs

The first, and perhaps most obvious, thing security analysts need to do is collect logs from all of the relevant cloud service environments. At the same time, analysts need to ensure all the cloud logs are going to a common location.

Many cloud service providers allow logs to be downloaded from their environment or stored in a dedicated storage node, like Amazon CloudTrail or Google Stackdriver Logging. There are also a number of security event aggregation and analysis platforms available today for the cloud, including Splunk Cloud, Sumo Logic, Loggly and Papertrail. These services may offer teams a simpler way to aggregate logs from multiple cloud services, and often integrate more readily with these services through provider APIs.

Once the cloud logs are collected and aggregated, analysts need to sift through all the various events and start prioritizing them. There are several keys to doing this.

  • Add context: If cloud logs can be "tagged" as originating from a specific service provider, then that can help to provide context on the use cases of the service. For example, logs from will focus on user activity and authentication, as well as administrative changes within the environment. Within Amazon Web Services or Azure, there will more than likely be much more varied activity, and from many more different types of users and roles.
  • Define priorities: Security analysts focused on the cloud must decide which events and behaviors are most critical to monitor. Common starting points include any login activity to cloud management consoles; any changes or attempted changes to important cloud objects and data; and any creation, deletion or modification of credentials or cryptographic keys.
  • Tune alerts: While it may seem like common sense, in general, tuning is incredibly important for cloud logging and event management. Suppress redundant alerts, those that are entirely operational in nature and those that aren't directly related to security.

    To build appropriate behavioral baselines of events in the environment, analysts will also likely need to allow several weeks or even months of data to accumulate. Make tuning a regular part of your weekly monitoring processes, as well.

  • Focus on accounts: Leftover user accounts and data are a big problem in the cloud. Work closely with human resources teams to disable cloud accounts quickly, and monitor all attempted logins to disabled or deleted accounts for at least several weeks after a user leaves the organization. It's a good idea to monitor user account activity before they leave, too. This will ensure departing employees don't try to take data with them; look for sudden increases in data exports or overall account use.

A final area of focus for cloud events should be the originating point of cloud activity. For many, a login from a new country or location where you don't do business or have users should be considered a very high-priority alert, and many cloud logs include enough detail to note the location of the login. Monitoring cloud logs for this type of valuable information while reducing the noise will greatly benefit security teams and their organizations.

Next Steps

Learn how Amazon CloudTrail could change cloud logging

Discover more about cloud logging and event management tools

Find out how the cloud can help enterprises with security log data

Dig Deeper on Cloud Computing Security Issues: Incident Response - Data Breach Prevention