This content is part of the Conference Coverage: Black Hat 2018 conference coverage
News Stay informed about the latest enterprise technology news and product updates.

Netflix launches tool for monitoring AWS credentials

At Black Hat 2018, a Netflix security engineer introduced a new open source tool designed to more effectively monitor AWS credentials in large cloud environments, like Netflix's.

LAS VEGAS -- A new open source tool looks to make monitoring AWS credentials easier and more effective for large organizations.

The tool, dubbed Trailblazer, was introduced during a session at Black Hat USA 2018 on Wednesday by William Bengtson, senior security engineer at Netflix, based in Los Gatos, Calif. During his session, Bengtson discussed how his security team took a different approach to reviewing AWS data in order to find signs of potentially compromised credentials.

Bengtson said Netflix's methodology for monitoring AWS credentials was fairly simple and relied heavily on AWS' own CloudTrail log monitoring tool. However, Netflix couldn't rely solely on CloudTrail to effectively monitor credential activity; Bengtson said a different approach was required because of the sheer size of Netflix's cloud environment, which is 100% AWS.

"At Netflix, we have hundreds of thousands of servers. They change constantly, and there are 4,000 or so deployments every day," Bengtson told the audience. "I really wanted to know when a credential was being used outside of Netflix, not just AWS."

That was crucial, Bengtson explained, because an unauthorized user could set up infrastructure within AWS, obtain a user's AWS credentials and then log in using those credentials in order to "fly under the radar."

However, monitoring credentials for usage outside of a specific corporate environment is difficult, he explained, because of the sheer volume of data regarding API calls. An organization with a cloud environment the size of Netflix's could run into challenges with pagination for the data, as well as rate limiting for API calls -- which AWS has put in place to prevent denial-of-service attacks.

"It can take up to an hour to describe a production environment due to our size," he said.

To get around those obstacles, Bengtson and his team crafted a new methodology that didn't require machine learning or any complex technology, but rather a "strong but reasonable assumption" about a crucial piece of data.

"The first call wins," he explained, referring to when a temporary AWS credential makes an API call and grabs the first IP address that's used. "As we see the first use of that temporary [session] credential, we're going to grab that IP address and log it."

The methodology, which is built into the Trailblazer tool, collects the first API call IP address and other related AWS data, such as the instance ID and assumed role records. The tool, which doesn't require prior knowledge of an organization's IP allocation in AWS, can quickly determine whether the calls for those AWS credentials are coming from outside the organization's environment.

"[Trailblazer] will enumerate all of your API calls in your environment and associate that log with what is actually logged in CloudTrail," Bengtson said. "Not only are you seeing that it's logged, you're seeing what it's logged as."

Bengtson said the only requirement for using Trailblazer is a high level of familiarity with AWS -- specifically how AssumeRole calls are logged. The tool is currently available on GitHub.

Dig Deeper on Cloud Provisioning and Cloud Identity Management Issues

Join the conversation


Send me notifications when other members comment.

Please create a username to comment.

What challenges have you faced when monitoring credentials in your AWS environment?
How does the tool ascertain if the call came from within the organisation if it has no prior knowledge of the IP Address allocation?
Bengtson explains Trailblazer's methodology in his blog post, which is linked in the above article, but to summarize -- the tool collects the log data from CloudTrail, which includes the Amazon Resource name for each AssumeRole API call, and then it builds a table with other data such as the Instance-ID. Here's an excerpt from the blog post that explains the rest:

"For each CloudTrail event, we will analyze the type of record to make sure it came from an assumed role. You can do this by checking the value of userIdentity.type and making sure it equals AssumedRole. If it is AssumedRole, we will grab the userIdentity.arn field which is equivalent to the AssumeRole-Arn column in the table. Since the userIdentity.arn has the requestParameters.roleSessionName in the value, we can extract the instance-id and do a lookup in the table to see if a row exists. If the row exists, we then check to see if there are any IPs that this AssumeRole-Arn is locked to. If there aren’t any, then we update the table with the sourceIPAddress from the record and this becomes our IP address that all calls should come from. And here’s the key to the whole method: If we see a call with a sourceIPAddress that doesn’t match the previously observed IP, then we have detected a credential being used on an instance other than the one to which it was assigned, and we can assume that credential has been compromised."
Using the AWS Credentials is one thing and getting a client to use AWS Cloud is another thing. It is becoming a challenge to find a client that uses AWS Cloud among the clients I have worked/interviewed in India.There are very few takers for AWS cloud in my opinion even though the AWS offering itself is strong and has good features, Amazon is yet to do a good job in selling these solutions to clients who have outsourced project to India. What Amazon has to do is set up its own consulting shop in India, hire consultants/employees in India, provide sales pitch as well as provide end to end solutioning for projects, only then their market share can improve. Besides some clients have unique requirements that call for a private cloud or hybrid cloud. How will you deal with that ??
Wait wait, hold all the horses guys! Does that mean my cousin can't logon to my account from her humble abode while I'm logged on simultaneously at my lavish palace?