nobeastsofierce - Fotolia


How NetFlow data can be used for cloud environments

Collecting and analyzing NetFlow data can help organizations detect security incidents and figure out their cause. Expert Frank Siemons explains how NetFlow works.

Within a network, connectivity is everything, but within a secure network, visibility is also everything. NetFlow data can meet both these requirements.

NetFlow is a Cisco proprietary technology that enables the collection of metadata generated by the traffic flow within a network. This metadata is invaluable for network capacity planning. It is also important for availability and security monitoring, as well as for forensic security analysis. It could, for instance, explain to a network engineer where the bottlenecks are within a network. It could also explain how a user was infected with malware or how compromised data was exfiltrated from an organization.

Some NetFlow analysis tools can alert you based on set thresholds and anomalies, as well. This enables security teams to monitor for incidents such as distributed denial-of-service attacks or port scan activity with a relatively low-cost solution.

What is collected?

The bare-minimum values NetFlow collects are source IP, destination IP, source port, destination port and the used protocol -- User Datagram Protocol (UDP), TCP or Internet Control Message Protocol. Together, these five values are called the five-tuple. Other, more advanced versions of NetFlow, such as Cisco's Flexible NetFlow, can also collect data around TCP flags, interfaces and even parts of actual packets.

NetFlow is only one step short of the collection of full packet capture data. Such a full packet capture tool is significantly harder to implement at scale, and it is usually too expensive for normal organizations to deploy and maintain. Of course, privacy is also more of a concern around a full packet capture functionality than around the collection of network traffic metadata.

Challenges in the cloud

NetFlow data is sent in the form of UDP messages from compatible devices -- such as switches, routers and firewalls -- to a NetFlow collector. In a traditional, physical network full of compatible switches and routers, this is an easy process. It is much harder to collect this data within a cloud platform where servers are virtual machines, connected via the cloud provider's implementation of a transparent virtual network.

If there is no NetFlow-compatible switch between two virtual cloud servers, how can NetFlow data be collected? Even though the servers are virtual, there is still value in collecting the metadata of the traffic between them.

An attacker who has successfully breached the cloud network perimeter -- for instance, via a SQL injection attack -- and has full access to a database server, might want to use another server to exfiltrate the obtained data. Such a spike in traffic between two internal servers is an anomaly that should be picked up by monitoring the network traffic, whether the systems are inside a cloud platform or inside a traditional on-premises data center.

How to make this work

If two virtual servers are in direct connection with each other, these servers are the only points where the traffic metadata is visible. This is why Cisco created an agent that needs to be installed directly on the cloud servers. These agents collect the flow data and send it to the cloud-based or on-premises monitoring software via a cloud concentrator. It can then be ingested as normal NetFlow data.

Companion article

See Infosec Institute's accompanying article on NetFlow Data Collection in Cloud Systems

Cloud providers have come up with a range of tools as well, such as Amazon's VPC Flow Logs. This is NetFlow-like metadata collected in an Amazon Web Services Virtual Private Cloud environment, made available to the customer for storage and analysis. There are some limitations to these solutions, however. Some values in the metadata might not be available in this format, and some analysis and monitoring software might not support the data format out of the box.

Probably the best solution, if the cloud platform allows for it, is the configuration of NetFlow in virtual switches and virtual routers. This is not always an option because many larger cloud platforms do not support managed, internal network devices. However, if the platform supports the use of a virtual switch or router, it is simply a matter of configuring a traditional NetFlow infrastructure around these devices.

Other network traffic metadata collection technologies

There are other technologies similar to NetFlow. The main one is IP Flow Information Export, or IPFIX. This is the Internet Engineering Task Force adaptation of NetFlow. Although there are some differences in the format of the data and some limitations in the fields, the principles of metadata collection and storage are very much the same. The same challenges and solutions around cloud platforms exist. IPFIX has some compatibility advantages, however, because it is not a Cisco proprietary technology.

Network metadata can provide much value to an organization, both from an availability and a security point of view. The challenge around its collection within a cloud environment is that this technology is designed around routers and switches, which are not available in that cloud environment.

There is a point to be made that an entirely new approach or product is needed, especially when cloud adaptation is only increasing in the future. The solution of deploying a Cisco server agent for NetFlow collection, or a third-party agent for IPFIX collection, seems to be going in that direction. It does not require any changes to the cloud infrastructure, provides full visibility and -- other than licensing -- is very low-cost to deploy.

Next Steps

Learn why software-defined networking is not yet an answer for hybrid cloud network bottlenecks

Find out how Google XPN works for virtual private cloud networks

Explore the latest trends in enterprise networking and management

Dig Deeper on Cloud Network Security Trends and Tactics