There are several tools that allow automatic deployment of Kubernetes clusters in AWS, like kube-aws, kops, kismatic and others. Some of the tools can also provision AWS infrastructure according to your use case.

kube-aws allows you to customize a yaml file and generate a Cloudformation stack that automates the creation of VPCs, subnets, Nat Gateways, security groups, etc. Even if it’s easier to deploy everything using kube-aws, I find it safer to manage your network infrastructure separately, especially when using Cloudformation, which sometimes can roll back and mess your entire stack.

The fallowing setup use a base Cloudformation stack to configure Public and Private Subnets, IGW, NatGW, Route Tables and KMS. After the stack is created we’ll deploy Kubernetes on top of it using kube-aws.

Features:

• all the nodes are deployed in private subnets
• 3 distinct availability zones
• 3 masters in HA, one per availability zone
• workers configured using node pools, similar to [GKE node pools](https://cloud.google.com/container-engine/docs/node-pools)
• HA ETCD with encrypted partitions for data, automatic backups to S3 and automatic recovery from failover
• role based authentication using the RBAC plugin
• users authentication using OpenID Connect Identity (OIDC)
• AWS IAM roles directly assigned to pods using [kube2iam](https://github.com/jtblin/kube2iam)
• cluster autoscaling

Note: all the configurations are available in GitHub

Deploy the Kubernetes cluster

1. Clone my demo repository https://github.com/camilb/kube-aws-secure locally

2. Create the base Cloudformation stack by customising and running ./vpc/deploy.vpc.sh script

3. Install kube-aws; in this example I’m using kube-aws v0.9-7-rc.3

4. Get the output values from the Cloudformation base stack created and update the ./kube-aws/cluster.yaml

aws cloudformation describe-stacks --stack-name kube-aws-vpc | jq -r '.Stacks[].Outputs'

5. Render the stack and credentials. Go to ./kube-aws directory and execute the fallowing command:

kube-aws render credentials --generate-ca 
kube-aws render stack

Please read the kube-aws documentation if you plan to use your own CA.

6. To launch the cluster, first you’ll need a S3 bucket. Use an existing bucket or create a new one.

kube-aws up --s3-uri s3://your-bucket-name

7. Access your Kubernetes cluster. Since we are creating all the resources in private networks, in order to access it, you’ll need a VPN placed in one of the public subnets. I’m using pritunl, which is very easy to configure. In approximately 5 minutes you can get it up and running on a t2.nano

After the VPN is set, make a quick check to list the nodes and pods in kube-system. You should get something similar to this:

$ kubectl --kubeconfig=./kubeconfig get nodes -o wide

NAME                         STATUS    AGE  EXTERNAL-IP  KERNEL-VERSION
ip-10-0-1-176.ec2.internal   Ready     1h     <none>       4.9.24-coreos
ip-10-0-1-219.ec2.internal   Ready     1h     <none>       4.9.24-coreos
ip-10-0-2-151.ec2.internal   Ready     1h     <none>       4.9.24-coreos
ip-10-0-2-245.ec2.internal   Ready     1h     <none>       4.9.24-coreos
ip-10-0-3-124.ec2.internal   Ready     1h     <none>       4.9.24-coreos
ip-10-0-3-96.ec2.internal    Ready     1h     <none>       4.9.24-coreos

$ kubectl --kubeconfig=./kubeconfig get pods -n kube-system

        NAME                                                 READY     STATUS    RESTARTS   AGE       
        calico-node-0ws56                                    1/1       Running   0          1h 
        calico-node-12177                                    1/1       Running   0          1h 
        calico-node-8kxsf                                    1/1       Running   0          1h 
        calico-node-9sq7p                                    1/1       Running   0          1h 
        calico-node-hh3l9                                    1/1       Running   0          1h 
        calico-node-vtzf2                                    1/1       Running   0          1h 
        calico-policy-controller-2265106533-gkl2q            1/1       Running   0          1h 
        dex-3003102726-2zjp0                                 1/1       Running   0          1h 
        heapster-v1.3.0-264939892-ggg1c                      2/2       Running   0          1h 
        kube-apiserver-ip-10-0-1-219.ec2.internal            1/1       Running   0          1h 
        kube-apiserver-ip-10-0-2-245.ec2.internal            1/1       Running   0          1h 
        kube-apiserver-ip-10-0-3-124.ec2.internal            1/1       Running   0          1h 
        kube-controller-manager-ip-10-0-1-219.ec2.internal   1/1       Running   0          1h 
        kube-controller-manager-ip-10-0-2-245.ec2.internal   1/1       Running   0          1h 
        kube-controller-manager-ip-10-0-3-124.ec2.internal   1/1       Running   0          1h 
        kube-dns-1759312207-2cd6s                            3/3       Running   0          1h 
        kube-dns-1759312207-khgqs                            3/3       Running   0          1h 
        kube-dns-autoscaler-2188776582-0rzp8                 1/1       Running   0          1h 
        kube-proxy-ip-10-0-1-176.ec2.internal                1/1       Running   0          1h 
        kube-proxy-ip-10-0-1-219.ec2.internal                1/1       Running   0          1h 
        kube-proxy-ip-10-0-2-151.ec2.internal                1/1       Running   0          1h 
        kube-proxy-ip-10-0-2-245.ec2.internal                1/1       Running   0          1h 
        kube-proxy-ip-10-0-3-124.ec2.internal                1/1       Running   0          1h 
        kube-proxy-ip-10-0-3-96.ec2.internal                 1/1       Running   0          1h 
        kube-rescheduler-3155147949-99k34                    1/1       Running   0          1h 
        kube-scheduler-ip-10-0-1-219.ec2.internal            1/1       Running   0          1h 
        kube-scheduler-ip-10-0-2-245.ec2.internal            1/1       Running   0          1h 
        kube-scheduler-ip-10-0-3-124.ec2.internal            1/1       Running   0          1h 
        kubernetes-dashboard-3963616910-fk65g                1/1       Running   0          1h 

Optionally you can configure your ~/.kube/config according to kubeconfig file to avoid passing the --kubeconfig flag on your commands.

Important

In order to expose public services using ELB or Ingress, the public subnets have to be tagged with the cluster name.

Ex. KubeernetesCluser=cluster_name

Add-ons

kube2iam

First we’ll configure the kube2iam to allow some of our applications to assume AWS IAM Roles.

When RBAC is enabled kube2iam needs permissions to list pods and namespaces. We have to grant these permissions.

$ kubectl create -f ./addons/kube2iam/rbac.yaml

deploy the kube2iam DaemonSet

change the account ID to yours in ./addons/kube2iam/k2i.ds.yaml, then create the DaemonSet

$ kubectl create -f ./addons/kube2iam/k2i.ds.yaml

Now your pods can assume all the roles that have a trust relationship configured.

Route53

This add-on is based on ExternalDNS project which allows you to control Route53 DNS records dynamically via Kubernetes resources.

Create a role named k8s-route53 using this policy. You also have to establish a trust relationship in order to allow the role to be assumed. An example is provided here.

Now change the values in external-dns.yaml and deploy it.

$ kubectl create -f ./addons/route53/external-dns.yaml

Nginx Ingress Controller

I’m choosing nginx over traefik because of the Proxy Protocol support. This allows to use the nginx ingress controller in AWS behind an ELB configured with Proxy Protocol. Here are some benefits from it:

• ingress address is associated with your ELB and doesn’t change when you replace the workers
• workers can be placed in private networks without public IPs
• better DDOS protection from ELB
• unlimited number of services that can be exposed using only one ELB. It’s like a cheap version of AWS ALB

Deploy

$ kubectl create -f ./addons/ingress/nginx/rbac.yaml
$ kubectl create -f ./addons/ingress/nginx/nginx.yaml

kube-lego

Kube-Lego automatically requests certificates for Kubernetes Ingress resources from Let’s Encrypt.

$ kubectl create -f ./addons/kube-lego/rbac.yaml
$ kubectl create -f ./addons/kube-lego/kube-lego.yaml

Dex

Dex runs natively on top of any Kubernetes cluster using Third Party Resources and can drive API server authentication through the OpenID Connect plugin.

By default you have administrator rights using the TLS certificates. If you plan to grant restricted permissions to other users, Dex can facilitate users access using OpenID Connect Tokens.

In this example we use the Github provider to identify the users.

Please configure the ./addons/dex/elb/internal-elb.yaml file then expose the service.

The DNS is configured automatically by ExternalDNS add-on and should be available in approximately 1 minute.

You can now use a client like dex’s example-app to obtain a authentication token.

If you prefer, you can use this app as a always running service by configuring and deploying ./addons/kid/kid.yaml

  $ kubectl create secret \
    generic kid \
    --from-literal=CLIENT_ID=your-client-id \
    --from-literal=CLIENT_SECRET=your-client-secret \
    -n kube-system

  $ kubectl create -f ./addons/kid/kid.yaml

Please check the dex [documentation](https://github.com/coreos/dex/tree/master/Documentation) if you need more informations.

Make a quick test by granting a user permissions to list the pods in kube-system namespace.

$ kubectl create -f ./examples/rbac/pod-reader.yaml
$ kubectl create rolebinding pod-reader --role=pod-reader [email protected] --namespace=kube-system

Example of ~/.kube/config for a user

   apiVersion: v1
    clusters:
    - cluster:
        certificate-authority-data: ca.pem_base64_encoded
        server: https://kubeapi.example.com
      name: your_cluster_name
    contexts:
    - context:
        cluster: your_cluster_name
        user: [email protected]
      name: your_cluster_name
    current-context: your_cluster_name
    kind: Config
    preferences: {}
    users:
    - name: [email protected]
      user:
        auth-provider:
          config:
            access-token: id_token
            client-id: client_id 
            client-secret: client_secret
            extra-scopes: groups
            id-token: id_token
            idp-issuer-url: https://dex.example.com
            refresh-token: refresh_token
          name: oidc

If you already have the ~/.kube/config set, you can use this example to configure the user authentication

    $ kubectl config set-credentials [email protected] \
      --auth-provider=oidc \
      --auth-provider-arg=idp-issuer-url=https://dex.example.com \
      --auth-provider-arg=client-id=your_client_id \
      --auth-provider-arg=client-secret=your_client_secret \
      --auth-provider-arg=refresh-token=your_refresh_token \
      --auth-provider-arg=id-token=your_id_token \
      --auth-provider-arg=extra-scopes=groups

Once your id_token expires, kubectl will attempt to refresh your id_token using your refresh_token and client_secret storing the new values for the refresh_token and id_token in your ~/.kube/config

At this point you have a pretty secure, highly available, Kubernetes cluster in AWS.

For even better security please also consider using the Pod Security Policy, Calico Network Policy and Istio]

Monitoring

A easy to setup, in-cluster, monitoring solution using Prometheus is available here https://github.com/camilb/prometheus-kubernetes