EBS RDS DR & Recovery with KMS key

Summary

When copying cross account an EBS/RDS Volume encrypted with custom KMS, a KMS key should also be available in the other account.

There are 2 ways that CPM uses for checking KMS key - Alias & Tag

Description

KMS Tag

When using custom tag,  you are telling CPM to use that custom ARN for the backup/DR (which could be in same account or even 3rd account).  A recovered resource will be encrypted with it as well.

The flow is:

      1. DR account snapshot is encrypted with custom key X 

      2. Volume recovered (cross-account) with custom key X.

      3. DR and recovery use the same key X, because it’s custom - original volume/instance has this tag (we use instance tag if it has one).

      4. When using a custom key, the original volume’s encryption is not relevant during DR.

This is useful if you have KMS in 3rd account that you want to use for all resources in both accounts.

For details how to use this option, see this KB: https://n2ws.zendesk.com/hc/en-us/articles/28815492111005-Use-custom-KMS-encryption-keys-for-cross-region-cross-account-DR

Alias:

When there is no custom tag, CPM will search in target account/region for KMS with same alias as the original KMS alias and will use that for the DR process.

So, in this scenario you just need to make sure KMS’s on both accounts have same alias, for example account A’s KMS key with alias ‘my_kms_1’ & account B’s KMS key with alias ‘ my_kms_1’.

Then when you recover to account A it will use account A KMS.

Possible Errors:

If CPM can't find matching Alias then you will see following error in backup log for EBS DR

Volume DR copy snapshot failed (to DR account). No matching KMS alias on target region (source US East (N. Virginia), target US East (N. Virginia), snapshot snap-01b452ad99d4ce30d, KMS alias: alias/test2)

And you will see similar error for RDS

ERROR:  start_copy_region(dr_rds.py:301) RDS DR copy snapshot failed (in Backup account). No matching KMS alias on target region (source EU (Frankfurt), target Asia Pacific (Singapore), RDS snapshot cpm-policy-RDS-daily, KMS alias: alias/ec2-prod-rds-custom-kms-key.

RDS  Alias KB: Cross region copy of RDS snapshots may fail with error: RDS DR copy snapshot failed (in Backup account) No matching KMS alias in target region

Another error that you might encounter is:

ERROR:  check_region_copy_state(dr_volume.py:409)  Volume DR region: EU (Ireland), snapshot snap-123456789 failed (source snap-987654321), has status error (policy mytestpolicy,to DR account). Error: Given key ID is not accessible

As part of copying snapshots cross account, the target account need to have access to the source KMS, otherwise it won't be able to re-encrypt the snapshot.

Go to KMS on the source account and add the DR account under "Other AWS accounts" on the Key Policy and the DR should be successful.

If you get this exception: Exception The source snapshot KMS key [arn:aws:kms:eu-central-1:123456789012:key/fdf5c33f-d9ba-4b1-99e3-3c1d45675b] does not exist, is not enabled or you do not have permissions to access it.

Check this KB: https://n2ws.zendesk.com/hc/en-us/articles/28815563284253-Cross-Account-and-Cross-Region-DR-of-an-encrypted-RDS-database-may-fail

Confirmation Steps

The above options being utilized will prevent recovery or backup failures due to KMS keys.