Mike Richardson in Automation 9 minutes

Secure Credentials with AWS KMS and Chef

Configuration files. They’re fantastic, hey? But they’re also pretty scary, as they tend to have a lot of sensitive data all stored in one place.

Populating those configuration files from automatic management systems like OpsWorks or Chef Server can be equally scary, as often it involves exposing those credentials in plain text. There can be many scenarios where you don’t want all of the administrators of these environments to have unrestricted access to all credentials.

In OpsWorks, the best example of this is variables stored in Custom JSON or App settings. Take an SSL keyphrase as an example: How can you pass this keyphrase through Custom JSON without exposing it to a developer with read-only access to the OpsWorks stack?

AWS Key Management Service

KMS makes it ridiculously easy to encrypt and decrypt sensitive data. Combined with IAM Roles, it is a powerful method to inject secure credentials into automated instances.

In this post, I’ll show you how to use the AWS Ruby SDK and Chef to decrypt an SSL keyphrase (or anything else) using KMS. Although this example will use Linux, I originally wrote these scripts for Windows and they work just as well.

Step 1 - Creating a keypair

This part happens entirely outside of your server environment, using the AWS CLI. So you’ll need to have that installed and be using a user with KMS access to create and use keys.

We will create a Custom Master Key (CMK) that we can then call any time we want to encrypt data. Decryption is even easier, as AWS can detect which key it should reference when encountering a request to decrypt data.

To create a key, we simply call aws kms create-key. You can optionally pass --description if you like.

$ aws kms create-key --description "Example Key"
{
    "KeyMetadata": {
        "KeyId": "bc631b98-4382-449a-c7a3-0e7fb1da6dd3",
        "Description": "Example Key",
        "Enabled": true,
        "KeyUsage": "ENCRYPT_DECRYPT",
        "CreationDate": 1438049747.793,
        "Arn": "arn:aws:kms:ap-southeast-2:186964239446:key/bc631b98-4382-449a-c7a3-0e7fb1da6dd3",
        "AWSAccountId": "186964239446"
    }
}

You should retain the KeyId returned here. We’ll be using it in just a moment.

Step 2 - Encrypting a passphrase

The CMK we have just created can be used to encrypt anything up to 4KB. Plenty of room for a SSL passphrase. If you want to use KMS to encrypt, you can. But since we only want to encrypt passwords here, we can do this very easily with our CMK.

To encrypt a passphrase we simply use the aws kms encrypt command:

aws kms encrypt \
--key-id dc631c98-4882-449d-b7a3-0d7fd1da9dd5 \
--plaintext "Password1" \
--output text | base64 --decode > encrypted-keyphrase.txt

Both the encrypt and decrypt functions in KMS return base64 encoded values. As such, the AWS Documentation instructions us to decode the output from these functions. On OS X and Linux, base64 decoding will work out of the box. On Windows, you can use certutil.

Manually Decryption

Before we move on, let’s manually decrypt our text file. We can do this by using the decrypt command.

aws kms decrypt \
--ciphertext-blob fileb://encrypted-keyphrase.txt \
--output text \
--query Plaintext | base64 --decode

Note that we don’t need to specify the key-id in decrypt. If we have access to the KMS key, we’ll be able to use it.

Step 3 - Inject the Encrypted Passphrase into Your Instance

This step can vary based on the automation platform your using. Normally, I upload the key to an S3 bucket and pass the S3 bucket and S3 key values to the instance using Custom JSON in OpsWorks, but you can do this any way you like.

Below is an example of code using the Ruby SDK in Chef to download the S3 file. The file is written to /root/credentials, and the run_state array is loaded with the filename of the cipherfile.

Here I’m using an IAM Role granted to the instance which allows it to connect to S3 and download the file. If you want to use IAM access and secret keys directly, you can pass these as a credentials array in the S3::Client call.

chef_gem "aws-sdk" do
  compile_time false
  action :install
end

ruby_block "download-password-cipherblob" do
  block do
    require 'aws-sdk'

    credentials = Aws::InstanceProfileCredentials.new(retries: 3)
    s3 = Aws::S3::Client.new(region: 'ap-southeast-2', credentials: credentials)

    node.run_state['cipherfile_name'] = node['cipherfile_s3key'].split('/')[1]

    s3.get_object(bucket: node['cipherfile_s3bucket'],
                  key: node['cipherfile_s3key'],
                  response_target: "/root/credentials/#{node.run_state['cipherfile_name']}")
  end
  action :run
end

Step 4 - Using a Chef Recipe to Decrypt

Let’s jump straight ahead to the fun stuff. The following code will use IAM Role permissions to call a KMS Decrypt on the cipherfile.

ruby_block "extract-password" do
  block do
    require 'aws-sdk'

    credentials = Aws::InstanceProfileCredentials.new(retries: 3)
    kms = Aws::KMS::Client.new(region: 'ap-southeast-2', credentials: credentials)

    cipher_file = File.new("/root/credentials/#{node.run_state['cipherfile_name']}", "r")

    resp = kms.decrypt({
      ciphertext_blob: cipher_file
    })

    node.run_state['decrypted_passphrase'] = resp.plaintext

  end
  action :run
end

The end result of this operation is that node.run_state['decrypted_passphrase'] will contain the string ‘Password1’.

Bonus Warning

If you’re relatively new to Chef, you may not have come up against the different phases of Chef execution. This can be really important to watch out for.

The Ruby code above is executed in the Converge phase, and node.run_state variables will be empty if you attempt to reference them in the earlier Compile phase.

For example, if you wanted to refer to the run_state as a variable in another resource, you should call it lazy:

variables(
  lazy {
    {
       :passphrase => node.run_state['decrypted_passphrase'],
    }
 }
)

This will delay evaluation of that resource until the Converge phase.