Using AWS KMS Custom Key Store with CloudHSM to Encrypt Your Data

I tend to follow cloud security news closely these days, particularly anything related to data encryption. That’s why one of the AWS re:Invent announcements that was of most interest to me actually occurred at the beginning of the conference, before the first keynote. Amazon Web Services announced the general availability of KMS custom key store, allowing users of AWS CloudHSM to take advantage of the AWS Key Management Service (KMS).

You can read more about KMS custom key store and the problems it solves for customers in this AWS security blog post. Primarily, custom key stores give users that have stringent security requirements the best of both worlds.

They can use CloudHSM to store and self-manage encryption keys that need to be protected in a hardware security module (HSM) that is single-tenant or need to be audited independently of KMS.
They can still get all the benefits of KMS and its tight integration with other AWS services and with customer applications that can make use of the AWS Encryption SDK.

A custom key store replaces the default KMS key store with a dedicated key store housed in single-tenant CloudHSM clusters. When a user needs to generate a Customer Master Key (CMK) in KMS, the key material is generated by CloudHSM and never leaves CloudHSM. KMS forwards all data key encryption requests to CloudHSM as part of its standard encryption process. This process is transparent to any client application that is leveraging KMS for encryption.

To illustrate how this would work in the real world, I am going to use the rest of this blog post to walk though how to configure a custom key store for use with a client-side encryption application. The application is called Rubrik CloudOut and is a component of my current employer’s cloud data management platform. CloudOut gives user the ability to archive their backup data to cloud object store solutions, such as Amazon Simple Storage Service (S3). Rubrik CloudOut encrypts all data client-side before it is uploaded to S3. Date keys are generated using the AWS Encryption client and the CMK can be generated using KMS. Below, I will walk through how to use custom key stores with CloudHSM to generate and store the CMK.

Note that the setup of CloudHSM and the a custom key store in this post will work the same with any application that integrates with KMS. But if you want a deep dive on how Rubrik CloudOut works with KMS, I suggest my deep dive post on the Rubrik blog site. To read more about data encryption in general and cloud encryption in particular, I invite you to read my earlier blog series on the topic.

Creating and Setting Up CloudHSM

We start by creating a CloudHSM cluster using the AWS CloudHSM console. Click on the Create cluster button on the upper left-hand of the console which bring you to the Create cluster wizard. For cluster configuration, we need to select a VPC and either select existing private subnets or create new private subnets. Each subnet needs to be in a different Availability Zone (AZ). For this post, we will choose 2 subnets in us-west-2 (Oregon). We click on the Review button and then confirm creation of the new cluster.
This will create a new CloudHSM cluster which can leverage 2 Availability Zones for redundancy. The console will assign a Cluster ID and show the cluster creation is in progress.

It will take several minutes for the cluster to be created. At that point, the status will change from Create in progress to Uninitialized.
Th next step is to initialize the new cluster for use. We click on the word Initialize next to the new cluster and we are prompted to create the first HSM in the cluster. We’ll choose one of the AZs we assigned to this cluster and click on the Create button.The creation process will take some time since a dedicated HSM has to be provisioned and configured.
Once the HSM has been created, we click on Next to download the Certificate Signing Request (CSR) for the cluster. We will need to sign the CSR with a self-signed signing certificate to initialize the cluster.
We will be creating our signing certificate using a private key we create through OpenSSL. In a production scenario, you would want to create the private key using something more secure like an offline HSM. To start, we will generate a private key, named customerCA.key, using a pass phrase.
Now we create a self-signed certificate file, named customerCA.crt, using our new private key. This certificate file will need to be copied to every host that will connect to our CloudHSM cluster.
Now we use the private key and self-signed certificate file to sign the CSR we downloaded earlier to create a signed certificate, named cluster-nshw4diyt7h_CustomerHsmCertificate.crt, which will be used to initialize our cluster.
To finish initializing the cluster, we go back to the CloudHSM console and click Next through the Initialize cluster wizard until we reach the screen for uploading our certificates. We need to upload our signed certificate (cluster-nshw4diyt7h_CustomerHsmCertificate.crt) and the certificate file (customerCA.crt).
Once the cluster is initialized, we can click on the Cluster ID to get more details about our cluster.
The next step is to activate the cluster. This has to be done through the command line, using a Linux or Windows instance with the AWS CloudHSM client software installed. The instructions for installing and configuring the Linux client software can be found here and the instructions for installing and configuring the Windows client software can be found here.

For this post, we will be using an Amazon EC2 instance, running Amazon Linux 2, that is in the same VPC as our CloudHSM cluster. Note that our EC2 instance will need to be attached to a Security Group that was automatically created when our CloudHSM cluster was created.
From our Linux instance, we start the cloudhsm_mgmt_util command line utility which will connect to our new HSM node and drop us into the CloudHSM CLI.
Next we need to enable end-to-end encryption.
When you create a CloudHSM cluster, a temporary user will be created on the first HSM. This temporary user is called the precrypto officer (PRECO) and its credentials are what we use to activate the cluster.
1. We first use the loginHSM command to log in to the HSM as the PRECO user using the default user name and password.
2. We then use the changePswd command to change the PRECO user password. This will also change the PRECO user to a crypto officer (CO). The CO has HSM management permissions.
3. We use the quit command to exit out of the CloudHSM CLI.
One of the requirements for a custom key store is that the CloudHSM cluster it uses must have at least 2 HSM nodes across 2 different AZs. We can add our second HSM by going to the Details page of our cluster and clicking on the Create HSM button.

Under the hood, the following is happening:
1. A new HSM is added to the cluster
2. A backup of the first HSM is taken and used to sync data to the new HSM
3. The first HSM notifies the AWS CloudHSM client that a new HSM has been added to the cluster and a connection is established
After adding a new HSM cluster, we have to update the HSM list in the configuration file that is used by the CloudHSM client. This will help keep the HSM nodes in our cluster in sync. To run the second command below, we need to copy the Elastic Network Interface (ENI) IP address of the new HSM node. The information can be found on the Details page for our CloudHSM cluster.

Note that NOT updating the HSM list will likely cause the next set of tasks to fail since all HSMs need to be in sync to connect our custom key store.

Creating and Connecting a Custom Key Store

The first step in creating a custom key store, after creating CloudHSM cluster, is to create to kmsuser crypto user account. A crypto user (CU) is a CloudHSM user that has permissions to perform key management and cryptographic operations. AWS KMS will use the kmsuser account to administer our custom key store. We log in as the CO admin user to create our new kmsuser CU account. Note that we are now logged into both HSM nodes in the cluster.
To create our custom key store, we need to switch over to the AWS Key Management Service console. Since one of the requirements is that the custom key store must be in the same AWS account and Region as the CloudHSM cluster, we will be creating be key store in us-west-2 (Oregon). We navigate to the Custom key stores page and click the Create a custom key store button.
In the Create custom key store dialog box, we need to do the following and then click on the Create button.
1. Create a name for our custom key store. For this post, we will use rubrik_cloudout_keystore.
2. Choose the CloudHSM cluster we created earlier.
3. Upload the trust anchor certificate for the CloudHSM cluster. This is the self-signed certificate file we created that we named customerCA.crt.
4. Enter the password for the kmsuser CU user we created earlier.

Once the new custom key store is create, it will need to be connected to our CloudHSM. Click on the custom key store, rubrik_cloudout_keystore, we just created and click on the Connect custom key store option under the Key store actions menu.

Under the hood, AWS KMS connects to our CloudHSM cluster, logs in as the kmsuser and rotates the kmsuser password.

After a few minutes, the console will report the key store as being CONNECTED.

Creating a Customer Master Key Using a Custom Key Store

Now we are ready to create a new Customer Master Key (CMK). Navigate to the Customer managed keys page in the KMS console and click on the Create key button.
In the Add alias and description dialog box, we provide an alias and a description. Expand the Advanced options section, select Custom key store (CloudHSM), and click on Next.
Select the custom key store, rubrik_cloudout_keystore, we created and connected to earlier and click on Next.
Click on Next again to skip creating Tags. Now we need grant an IAM user administrative permission to the CMK we are creating and click on Next.
We select the IAM user and/or role that will be using this CMK to encrypt data keys and click on NEXT. For this post, we will be assigning the CMK to a user named rubrik-cloudout that will be uploading files to an S3 bucket as part of the Rubrik CloudOut solution.
Review and edit the key policy that will be granted to our IAM users. We will leave the policy as is and click on Finish.
After the CMK is created, we will see that a Key ID has been assigned to the new key.
We can drill down into our new CMK to get more details.

Our new CMK is used and managed the same way as any other CMK. Under the covers, however, the key material is not generated and stored in KMS, which is a multi-tenant solution. Instead all key material for the customer key store-backed CMK is stored in CloudHSM.

Configuring Rubrik CloudOut to use KMS

To show you an example of how the integration would work with an application, we will quickly walk though configuring Rubrik to use our new custom key store-backed CMK for client-side encryption of data.

We log into a Rubrik cluster and navigate to the Archival Locations page.
We then create new Rubrik Archival Location using Amazon S3. The bucket, relevant bucket policy, and IAM user have been previously created and configured.
For Encryption Type, we will be choosing KMS Master Key ID. We then copy and paste in the key ID for the custom key store-backed CMK we created earlier and click on Add. The KMS Master Key ID is a representation of the underlying CMK and allows us rotate the CMK without having to edit the Master Key ID each time.
Rubrik will confirm access to the S3 bucket and correct permissions to the CMK prior to creating the new Archival Location.

From this point, all data that is designated for our rubrik-tme-or bucket will be encrypted client-side by Rubrik using our newly created CMK to encrypt the data keys. Behind the scenes, KMS will leverage CloudHSM via our custom key store connection.

Hopefully this blog post will help you to get started with using custom key stores to integrate AWS KMS with CloudHSM. Feel free to reach out to me with any questions, comments, and corrections.