Access S3 from Kubernetes Pods and Nodes

TheProtos included in k8s aws

2023-06-10 837 words 4 minutes

Contents

If a container needs access to S3 buckets, this can be achieved with kubernetes node scope and Pod scope

Kubernetes node access to S3 bucket

All pods on the node will have access to S3 bucket

Prerequisites:

eks custer
bucket mybucket-myapp
eksctl
kubectl

eksctl with `aim:attachPolicy`

Nodegroup with aim:attachPolicy can be created, see below is a part of cluster config

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


#...<>
managedNodeGroups:
  - name: mynode
    labels: { role: mylabel }
    instanceType: m5.xlarge
    desiredCapacity: 10
    volumeSize: 80
    privateNetworking: true
    iam:
      attachPolicy:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Action:
              - "s3:ListBucket"
            Resource:
              - "arn:aws:s3:::mybucket-*"
          - Effect: Allow
            Action:
              - "s3:*Object"
            Resource:
              - "arn:aws:s3:::mybucket-*/*"
#...<>

Above NodeGroup creates ASGs/EC2 for k8s cluster, these nodes will have label role: mylabel

We can place our workload that needs S3 bucket to such nodes using nodeSelector as below

1
2


nodeSelector:
  role: mylabel

In order to see all labels run below:

1

kubectl get nodes --show-labels

Alternative. Manual IAM role setup.

A role with below policy must be assigned to instances (EC2, EC2 ASG, Fargate, etc):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24



{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::mybucket-*"
      ],
      "Effect": "Allow"
    },
    {
      "Action": [
        "s3:*Object"
      ],
      "Resource": [
        "arn:aws:s3:::mybucket-*/*"
      ],
      "Effect": "Allow"
    }
  ]
}

Running demo workloads on node with label

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


apiVersion: apps/v1
kind: Deployment
metadata:
  name: s3demo
spec:
  selector:
    matchLabels:
      app: s3demo
  template:
    metadata:
      labels:
        app: s3demo
    spec:
      containers:
        - name: demo-aws-cli
          image: amazon/aws-cli
          command: ["/bin/sh", "-c"]
          args:
            - echo starting;
              aws sts get-caller-identity;
              echo $(date) | aws s3 cp - s3://mybucket-myapp/file-$(date +"%d-%m-%Y").txt;
              echo done;
              sleep infinity;
      nodeSelector:
        role: mylabel

Kubernetes pod access to S3 bucket

Only pods with specific SA(service account) can access S3 Refer to original article

Prerequisites:

bucket mybucket-myapp
eks cluster with IAM OIDC provider
eksctl
kubectl

Create SA via `eksctl` cli

Important to highlight that this method works ONLY within the namespace you pass to eksctl create iamserviceaccount --namespace .., if omitted - it will use namespace default

Create IAM policy document

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


cat <<"EOF" > my-s3-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::mybucket-*"
    },
    {
      "Effect": "Allow",
      "Action": "s3:*Object",
      "Resource": "arn:aws:s3:::mybucket-*/*"
    }
  ]
}
EOF

Create IAM policy based on above

1

aws iam create-policy --policy-name my-s3-policy --policy-document file://my-s3-policy.json

eksctl will create IAM role (with the policy) and kubernetes SA (with annotation to IAM role) in the namespace

1
2


eksctl create iamserviceaccount --name my-s3-service-account --namespace default --cluster mycluster --role-name my-s3-role \
--attach-policy-arn arn:aws:iam::${AWS_ACCOUNT_ID}:policy/my-s3-policy --approve

Create SA via `eksctl` config

The SA will be binded to specific namespace as well

This does the same as previous step

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

#<...>

iam:
  withOIDC: true
  serviceAccounts:
    - metadata:
        name: my-s3-service-account
        namespace: default
      attachPolicy:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Action:
              - "s3:ListBucket"
            Resource:
              - "arn:aws:s3:::mybucket-*"
          - Effect: Allow
            Action:
              - "s3:*Object"
            Resource:
              - "arn:aws:s3:::mybucket-*/*"

#<...>

Using wildcards in IAM role

There is a way how to reuse IAM role from above in different namespaces and SA names

It needs to modify trust relationship policy document that attached to the role from previous steps. Main modifications are:

StringEquals replaced with StringLike
system:serviceaccount:default:my-s3-service-account replaced with system:serviceaccount:*:myapp*

Final document could look like below one:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/<SOME_ID_HERE>"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringLike": {
                    "oidc.eks.us-west-2.amazonaws.com/id/<SOME_ID_HERE>:sub": "system:serviceaccount:*:myapp*",
                    "oidc.eks.us-west-2.amazonaws.com/id/<SOME_ID_HERE>:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}

The SA can be removed from default namespace, however the IAM role will still exist. You can create Service Accounts with the name like myapp* in any namespace

option 1. From manifest below

1
2
3
4
5
6


apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::${AWS_ACCOUNT_ID}:role/my-s3-role
  name: myapp-service-account

option 2. Annotate existing SA

1

kubectl annotate serviceaccount -n NAMESPACE my-s3-service-account eks.amazonaws.com/role-arn=arn:aws:iam::${AWS_ACCOUNT_ID}:role/my-s3-role

Running demo workloads with SA

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/my-s3-role
  name: my-s3-service-account
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: s3demo
spec:
  selector:
    matchLabels:
      app: s3demo
  template:
    metadata:
      labels:
        app: s3demo
    spec:
      serviceAccountName: my-s3-service-account
      containers:
        - name: demo-aws-cli
          image: amazon/aws-cli
          command: ["/bin/sh", "-c"]
          args:
            - echo starting;
              aws sts get-caller-identity;
              echo $(date) | aws s3 cp - s3://mybucket-myapp/file-$(date +"%d-%m-%Y").txt;
              echo done;
              sleep infinity;