New CDK Bootstrap and the EKS Cluster

New CDK Bootstrap and the EKS Cluster

Rafael Zamana Kineippe

In the AWS CDK Version v1.25.0, the CDK team added a new bootstrap template that includes new resources like IAM Role and S3 Buckets. From the AWS CDK Documentation:

The AWS CDK supports two bootstrap templates. At this writing, the AWS CDK is transitioning from one of these templates to the other, but the original template (dubbed “legacy”) is still the default. The newer template (“modern”) is required by CDK Pipelines today, and will become the default at some point in the future.

The new bootstrap brings new functionalities and new problems as well! This article will not focus on whether to use the modern or the legacy bootstrap. I will show how the new bootstrap might lead to a problem in the deployment of an EKS cluster and how to solve it.

How to use the new CDK Bootstrap

When bootstrapping an account, it’s essential to set the CDK_NEW_BOOTSTRAP environment variable.

$ export CDK_NEW_BOOTSTRAP=1
$ cdk bootstrap

After that is recommended to add the context flag @aws-cdk/core:newStyleStackSynthesis in the cdk.json, this way:

{
  // ...
  "context": {
    "@aws-cdk/core:newStyleStackSynthesis": "true"
  }
}

You can find these steps and more information in the AWS CDK Documentation.

IAM Roles

After the bootstrap, it’s possible to see some differences, and one of these differences is the four new IAM Roles created.

Role Usage
cdk-${Qualifier}-deploy-role-${AWS::AccountId}-${AWS::Region} role assumed by the CLI and Pipeline to deploy
cdk-${Qualifier}-file-publishing-role-${AWS::AccountId}-${AWS::Region} role used for file asset publishing (assumed from the deploy role)
cdk-${Qualifier}-image-publishing-role-${AWS::AccountId}-${AWS::Region} role used for Docker asset publishing (assumed from the deploy role)
cdk-${Qualifier}-cfn-exec-role-${AWS::AccountId}-${AWS::Region} role passed to CloudFormation to execute the deployments

These roles ensure that the AWS CDK can deploy your application.

The ${Qualifier} is a 10 (max) characters defined in the cdk.json, like:

{
  // ...
  "context": {
    "@aws-cdk/core:newStyleStackSynthesis": "true",
    "@aws-cdk/core:bootstrapQualifier": "hnb659fds"
  }
}

The default value for the qualifier is hnb659fds. The qualifier helps in case you need to bootstrap the same Account multiple times for different applications.

Cross-Account

The typical scenario is to have one AWS Account per environment and one AWS Account as Tooling Account, where it will have CodePipeline/CodeBuild in place.

This Tooling Account will handle the deployment of the application in other accounts.

For the AWS CDK to work with a cross-account capability, it’s necessary to define a trust account during the bootstrap process.

$ export CDK_NEW_BOOTSTRAP=1
$ cdk bootstrap \
    --trust {ACCOUNT_ID}

Adding the trust argument will ensure that the roles (deploy, file-publishing, and image-publishing) in the Account where you are bootstrapping can be assumed by the trusted Account.

NOTE: Please keep in mind that the cfn-exec role does not have this permission, even when setting the trust argument

EKS Cluster

From the AWS EKS Documentation:

When you create an Amazon EKS cluster, the IAM entity user or role, such as a federated user that creates the cluster, is automatically granted system:masters permissions in the cluster’s RBAC configuration in the control plane. This IAM entity does not appear in the ConfigMap, or any other visible configuration, so make sure to keep track of which IAM entity originally created the cluster. To grant additional AWS users or roles the ability to interact with your cluster, you must edit the aws-auth ConfigMap within Kubernetes.

There is already a request to give the possibility to define this Admin Role, and you can see it here.

But why is this a problem?

With the legacy bootstrap from the AWS CDK, there is no problem. If you deploy an EKS Cluster locally, your user would be the Cluster Administrator, and you need to update the aws-auth ConfigMap.

Legacy Bootstrap Deploy Diagram

With the new CDK Bootstrap, where the CloudFormation is executed using the role cfn-exec-role, the Administrator role is not your user anymore, and it will be the cfn-exec-role.

Modern Bootstrap Deploy Diagram

I will be honest, and say that I liked this change, so the Administrator Role would not be a person, but a defined role, used for specific cases. I had run into many problems because someone had deployed an EKS Cluster from a local machine, and then we needed to add the CodeBuild role into the aws-auth ConfigMap to manage the K8S from a Pipeline.

And this brought other problems, and now the Administrator Role is a role that the bootstrap does not share with the Tooling Account.

EKS Module

Looking at the AWS CDK EKS Module, the Cluster class can indeed receive an IAM Role to the Master Role (if you do not provide this parameter, it will create one for you).

This class is still Experimental, and I wanted to use it, but my experience with the AWS CDK told me to wait a little more.

With this in mind, I tell you that I only use the CfnCluster Class to create my EKS Cluster, and this class does not bring the Master Role parameter, so I would need to think of another solution.

The Problem

Now you have the scenario!

An EKS Cluster being deployed by the CloudFormation using the cfn-exec role, from a Tooling Account with the CodePipeline/CodeBuild assuming the deploy role.

The result is that it’s impossible to connect to the K8S Cluster because the Creator Role is the cfn-exec, and the trust Account does not have access to this role.

Security

I do understand why the cfn-exec role is not shared. This role is meant to be used only by CloudFormation and has too many privileges (Administrator).

So, how can we solve this issue?

Changing the CDK Bootstrap Template

My solution involves changing the bootstrap template from the AWS CDK.

The first step is to output the default template that is going to be used to bootstrap the Account.

NOTE: Don’t forget the CDK_NEW_BOOTSTRAP flag.

$ export CDK_NEW_BOOTSTRAP=1
$ cdk bootstrap --show-template > bootstrap-template.yaml

This will save the template into bootstrap-template.yaml file. Now it’s time to edit this file, open it in your prefereable IDE, and find the CloudFormationExecutionRole definition:

  CloudFormationExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: cloudformation.amazonaws.com
        Version: "2012-10-17"
      ManagedPolicyArns:
        Fn::If:
          - HasCloudFormationExecutionPolicies
          - Ref: CloudFormationExecutionPolicies
          - Fn::If:
              - HasTrustedAccounts
              - Ref: AWS::NoValue
              - - Fn::Sub: arn:${AWS::Partition}:iam::aws:policy/AdministratorAccess
      RoleName:
        Fn::Sub: cdk-${Qualifier}-cfn-exec-role-${AWS::AccountId}-${AWS::Region}

We will add the trust account information in the AssumeRolePolicyDocument, like this:

          - Fn::If:
              - HasTrustedAccounts
              - Action: sts:AssumeRole
                Effect: Allow
                Principal:
                  AWS:
                    Ref: TrustedAccounts
              - Ref: AWS::NoValue

This way, it will be possible to assume the role in the trust account!

Improving Security

Thinking about this solution’s security, I had the idea to add the external-id when the trust Account assumes this role.

This way, we increase the security of how the Account can assume this role.

          - Fn::If:
              - HasTrustedAccounts
              - Action: sts:AssumeRole
                Effect: Allow
                Principal:
                  AWS:
                    Ref: TrustedAccounts
                Condition:
                    StringEquals:
                        sts:ExternalId: "12345"
              - Ref: AWS::NoValue

Final Result

The modified template will look like this:

  CloudFormationExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: cloudformation.amazonaws.com
          - Fn::If:
              - HasTrustedAccounts
              - Action: sts:AssumeRole
                Effect: Allow
                Principal:
                  AWS:
                    Ref: TrustedAccounts
                Condition:
                    StringEquals:
                        sts:ExternalId: "12345"
              - Ref: AWS::NoValue
        Version: "2012-10-17"
      ManagedPolicyArns:
        Fn::If:
          - HasCloudFormationExecutionPolicies
          - Ref: CloudFormationExecutionPolicies
          - Fn::If:
              - HasTrustedAccounts
              - Ref: AWS::NoValue
              - - Fn::Sub: arn:${AWS::Partition}:iam::aws:policy/AdministratorAccess
      RoleName:
        Fn::Sub: cdk-${Qualifier}-cfn-exec-role-${AWS::AccountId}-${AWS::Region}

Save it, and to bootstrap an account with this template, just need to:

$ cdk bootstrap \
    --template bootstrap-template.yaml \
    --trust {ACCOUNT_ID}

or using the CloudFormation CLI:

$ aws cloudformation create-stack \
  --stack-name CDKToolkit \
  --parameters ParameterKey=TrustedAccounts,ParameterValue={ACCOUNT_ID} \
  --template-body file://bootstrap-template.yaml

I hope this could be useful for you! Please let me know if you found a different solution!