AWS IAM Role Chaining using IAM Roles for Service Accounts inside Kubernetes
Today, we're taking a look at using IAM roles inside of AWS EKS, leveraging the combined power of IAM Roles for Service Accounts (IRSA) and role chaining inside of AWS config files.
Prerequisites
This post assumes the reader has a working understanding of AWS IAM, including the differences between IAM users and roles, how roles are assumed/used, and their purpose in the IAM ecosystem.
Why does this even matter?
I've encountered several instances where a service/job/pod/what-have-you running in an AWS EKS cluster needed to assume multiple AWS roles to accomplish its task. Typically, when a pod needs access to several resources (say, several s3 buckets, an AWS Redshift cluster, and an Athena/Glue crawler in a different account from the EKS cluster), common sense dictates we should create a role specifically for that pod to assume and call it a day. This is a fine pattern, and its a good practice to separate roles meant for assumption by humans from roles meant for robots anyhow.
However, there are scenarios where it may not be practical or desirable to create a monolithic role for a specific task. That could be for a myriad of reasons, such as migrating legacy workflows into Kubernetes, using roles that connect to partner/client AWS resources not under your control, or not granting cross-account resource access for roles. Some resources, like AWS Glue and AWS Secrets Manager, don't behave the same way during cross-account access.
So a single role isn't always feasible for a pod. Have an example?
Recently, I tried to run an ansible playbook inside a Kubernetes cronjob, while maintaining the playbook's ability to be run by humans (who all use a standardized set of AWS profiles, that's important later). Ideally, both human operators and robots should be able to use the playbook without any code changes. The playbook uses two profiles, named account-a-admin and account-b-admin. All humans who need to run this playbook locally have account-a-admin and account-b-admin profiles defined already in their AWS config file (~/.aws/config), and each profile assumes a role role in either account a or b respectively. These admin roles are powerful, and not something that should be handed out lightly (and never assumed by robots).
Now, that still doesn't explain why we can't create new role used solely by this pod, granting the specific access required to execute the playbook. Well, because this playbook operates on ec2 instances, it uses the aws_ec2 dynamic inventory plugin. For the inventory plugin to work properly for human users, an AWS profile name must be passed as part of the configuration. Normally, we could override profile names with ansible extra vars (using the -e command line flag) or by using a jinja2 template to look up the value from an environment variable. Sounds simple, right?
Wrong. Some dynamic inventory plugins respect extra vars, but are not required to. They also don't support jinja2 templates. You can see where I'm going with this. There isn't a way to dynamically select a profile for the aws_ec2 dynamic inventory. Therefore, we need to create AWS config profiles with credentials that will satisfy the inventory plugin.
Okay, you need to make AWS config profiles. So? That's common.
That's the impetus behind me writing this. Most engineers who have written AWS config files before have only used them with static credentials stored in ~/.aws/credentials, then used those credentials to assume a role. Instead, we can adapt the AWS config with some lesser-known options assume roles granted to pods through EKS, without defining static credentials.
How IRSA Works
Now, before we get too deep, here's a brief overview on how IAM integrates with Kubernetes in AWS.
At some point, a workload running in Kubernetes will need to access some resource outside the cluster to perform its task. For the sake of example, we'll assume that we have a pod running inside EKS, AWS's cloud Kubernetes distribution. This pod needs to access some files in s3, but it was deemed too risky to stash a non-expiring IAM keypair inside a kubernetes secret and call it a day. Instead, we can grant a pod the temporary credentials required to assume an IAM role through AWS IAM roles for Service Accounts (IRSA) feature.
IRSA works by allowing the exchange of Kubernetes service account OIDC JWT tokens with IAM for set of temporary IAM credentials corresponding to a role. This requires both an IAM OIDC Connect Provider to be created using the cluster OIDC issuer's URL and thumbprint, and the eks pod identity mutating admission webhook. When a Kubernetes service account with the eks.amazonaws.com/role-arn: <arn> annotation is attached to a pod, the pod identity webhook injects a token via a projected serviceAccountToken volume and sets several environment variables.
With this token, the pod can then call the AWS SecurityTokenService (STS) api to exchange the the token for a set of IAM credentials, using the AssumeroleWithWebIdentity action. Modern versions of the AWS SDK will see the AWS_WEB_IDENTITY_TOKEN_FILE environment variable and automatically assume the role on startup. For instance, the AWS cli command aws sts get-caller-identity will return a role session for the role specified in the service account annotation instead of a role session for the underlying host node role when the service account is attached.
To illustrate what is added by the pod identity webhook, see the following diff.
apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/svc-eks-podrole
name: my-serviceaccount
namespace: default
apiVersion: apps/v1
kind: Pod
metadata:
name: myapp
spec:
serviceAccountName: my-serviceaccount
containers:
- name: myapp
image: myapp:1.2
+ env:
+ - name: AWS_ROLE_ARN
+ value: arn:aws:iam::123456789012:role/svc-eks-podrole
+ - name: AWS_WEB_IDENTITY_TOKEN_FILE
+ value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
+ volumeMounts:
+ - mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
+ name: aws-iam-token
+ readOnly: true
+ volumes:
+ - name: aws-iam-token
+ projected:
+ defaultMode: 420
+ sources:
+ - serviceAccountToken:
+ audience: sts.amazonaws.com
+ expirationSeconds: 86400
+ path: token
Now, Lets Set Up Some Profiles
Now, we've established that role credentials provided thorugh the EKS pod identity webhook are WebIdentity roles, and credentials are granted through the AssumeroleWithWebIdentity STS api. While the AWS SDK and cli will automatically detect the environment variable and assume the role, that does not extend to profiles. For example, if we call aws sts get-caller-identity --profile default from within our EKS pod, the command will exit with an error that no config profile was found with the name "default". Which makes sense, since we haven't defined one in the config file yet.
Digging through the AWS cli config reference, there's a section for options relating to AssumeroleWithWebIdentity. We can create a profile that takes a web identity token file and role arn, and the SDK will automatically call STS on our behalf to retrieve credentials for the role. Since our token is automatically injected into the pod using a volume and because we know the mount path, this config becomes pretty trivial.
[profile default]
arn:aws:iam::123456789012:role/svc-eks-podrole
/var/run/secrets/eks.amazonaws.com/serviceaccount/token
Now, calling aws sts get-caller-identity --profile default will produce the same result as not providing a profile. We're now assuming the role the same as the SDK does without a config file, but in a more explicit way.
We can take this a couple steps further though, and chain several role assumptions together. This unlocks a tremendous amount of power when working with multiple roles:
[profile default]
arn:aws:iam::123456789012:role/svc-eks-podrole
/var/run/secrets/eks.amazonaws.com/serviceaccount/token
[profile account-a-admin]
default
arn:aws:iam::345678901234:role/Administrator-role
[profile account-b-admin]
default
arn:aws:iam::567890123456:role/Administrator-role
[profile account-b-specialized]
account-b-admin
arn:aws:iam::567890123456:role/special-s3-access
If we call aws sts get-caller-identity --profile account-b-specialized now, the AWS SDK will automatically chain the assumption of all three roles and return a set of credentials matching the arn:aws:iam::567890123456:role/special-s3-access role. Not only that, but the SDK now has a template on how to automatically refresh these role credentials when they're close to expiring, without having to call STS ourselves or write any code that creates or manages AWS sessions.
Bonus Points: Kubernetes Best Practices
Instead of injecting this AWS config into your container at build time, create a Kubernetes configmap with the AWS config and mount it inside the pod using a volume. Commit these manifests to your kubernetes source control repository (you have one, don't you?), that way other engineers very explicitly know what roles are being consumed by this particular workload, and they can be easily adapted down the line.
Should I use this?
Ultimately, that comes down to your use case and the organization you work in. If your code (script in a cronjob, Airflow DAG, web microservice, etc) can operate off of a single AWS role that can be directly assumed by the pod, then no, don't bother. Continue using web identity role assumption directly through the SDK as you have been and move on with life.
If you need multiple roles for different actions, run with an ancient SDK, or need to maintain code execution compatibility between different types of systems as I had, then this might be an easy solution. Sure, it's possible to write some code against boto3 to assume multiple roles and auto-refresh their credentials before every operation if they're close to expiring. You can gate that code to only running in specific execution environments too. Or, you can offload all that work (and brittleness) to the AWS SDK, and let it do its job handling credentials for you.
Put it this way, do you really want to be parsing temp files to check the credential expiry time with jq every time your bash script needs to connect to s3, and renew them if they're close to expiring? I certainly wouldn't.