Another month, another story about cloud assets that aren't well-protected.
In a refreshing break from stories about wide-open Simple Storage Service buckets in Amazon Web Services, RedLock Inc.'s Cloud Security Trends report from October 2017 found that many Kubernetes administrative consoles on Amazon Web Services, Microsoft Azure and Google Cloud Platform were not password protected, enabling hackers to access the consoles with ease. In the course of its research, RedLock found that hackers were mining bitcoin within at least one of the compromised containers, essentially turning the cloud instances into bots.
There are several key issues at the core of how this happened. In truth, Kubernetes security had little to do with the compromise itself; Kubernetes was simply hijacked and used to leverage container instances. The biggest problem was a lack of sound configuration management, coupled with minimal monitoring and alerting.
How to improve Kubernetes security
There are many ways to boost Kubernetes security. First, the main console should be better-protected with strong authentication to leverage strong passwords and multifactor authentication coming from a separate jump host server that admins log into. Placing Kubernetes consoles directly on the internet is never a good idea, and to make matters worse, the report suggests that these systems were using simple or default credentials that were easily guessed, or no credentials at all.
Aside from isolating and controlling access to Kubernetes consoles, there are a number of additional steps organizations can take to improve Kubernetes security.
- Implement the strongest authentication to Kubernetes that you can, ideally using a separate directory service and an additional factor, like certificates. Kubernetes offers authentication options ranging from basic username/password flat files to more sophisticated federation schemes using OpenID Connect. More information on Kubernetes authentication can be found on the Kubernetes website.
- Use only trusted images that meet best practices for configuration controls, and continually reassess the state of any running images using scanners and local tools. There are many commercial and open source tools available for assessing containers. In the compromised systems described by RedLock, admins failed to detect changes made to container instances when bitcoin mining tools were uploaded and executed in the environment. This demonstrates a lack of configuration control, as well as a lack of monitoring within containers.
- Implement administrative boundaries in containers with namespaces. This enables admins to hide resources from different groups, and then apply different authorization controls to the namespaces to control and limit access as needed. Similarly, pods and containers should have a security context assigned to them as defined in the deployment configuration that restricts privileges as much as possible for operation.
- Define resource quotas for CPU and memory that are attached to namespaces. By default, all Kubernetes containers have unlimited CPU and memory request capabilities, which can easily lead to abuse in a denial-of-service or bitcoin mining scenario.
- Define network segmentation policies between Kubernetes pods, services and containers where appropriate.
- Log everything in the environment. Kubernetes and other container technologies, like Docker, support robust logging at the node, cluster and administrative levels. Security, operations and development teams can all benefit from comprehensive logs describing activities and events that occur in the environment at all levels, and most of these logs are either natively Syslog-compatible or output in formats like JSON, which can be parsed easily and ingested into many log and event management systems.
In addition to all these Kubernetes security recommendations, cloud users should configure billing thresholds that match normal usage patterns, and alerts should be sent when usage exceeds the threshold that has been set. Even if a cloud asset is compromised, there's no excuse for high intensity activities like bitcoin mining to occur for any length of time without someone being alerted. Most major cloud providers readily support many ways to monitor the environment as a whole.