Certified Kubernetes Application Developer

10.02.2020 — Tutorial, CKAD, Kubernetes, K8s — 8 min read

This post is just notes to myself for my CKAD exam prep

Linux Foundation and CNCF together prepare CKAD exam

DEVOPS15 gets you 15% discount when registering for the exam

Core Concepts

Nodes, Clusters

Node is a single machine, that can have multiple pods

Cluster is the combination of multiple machines, or nodes

Master node is configured to manage cluster of machines

Components

API Server: the front end of the cluster

etcd: store the data to manage the cluster, key:value pairs

Scheduler: distributing works across the cluster

Controller: brain of the cluster, makes decision to run containers

Container runtime: in this case docker

kubelet: agents running in the nodes

Master vs Worker nodes

Worker node is where containers are hosted

We need container runtime to execute container processes

Pod is the simplest form of K8s object

Pod has 1 to 1 relationship between Pod and Container, thus to scale up, increase pods

However, if there is a helper container, it can be deployed to the pod too

Remember, aim is to build replicas that can be increased/decreased as necessary

To find the Nginx image from dockerhub and run in a pod

$ kubectl run nginx --image nginx

then get the pods running

$ kubectl get pods

It is also possible to deploy a pod via yaml file

pod.definition.yaml

apiVersion: v1
kind: Pod 
metadata:
  name:myapp-pod
  label:
    app: myapp
    type: front-end
  annotation:
spec:
  containers: 
    - name: my-container-name-definition
      image: nginx

Metadata children are name and labels

Pods

To create a redisapp pod with redis image imperative way

$ kubectl run redis --image=redis --generator=run-pod/v1

For more info on generators: https://kubernetes.io/docs/reference/kubectl/conventions/#generators

To edit pods where a pod definition is not given

$ kubectl get pod <pod-name> -o yaml > pod-definition.yaml

Important: you have to delete and re-create the pod

$ kubectl delete pod <pod-name>
$ kubectl apply -f pod-definition.yaml

To edit pod properties

$ kubectl edit pod <pod-name>

Replication controller

We need replication controller to manage the desired state and scaling

Replication controller is the old version of the Replica Sets

Skeleton yaml file below for Replication Controller, not complete

rc-definition.yaml

apiVersion: v1
kind: ReplicationController
metadata:
spec:
  template:
  ... #Pod definition here, without apiVersion and kind 
  replicas: 3

A simple version

rc-definition.yaml

apiVersion
kind: ReplicationController
metadata:
  name: myapp-rc
  labels:
    app: myapp
    type: front-end
spec:
  template:
    metadata:
      name: myapp
      labels:
        app:myapp
    spec:
      containers:
      - name: nginx-controller
        image: nginx
  replicas: 3

Now let's look at a replicaset definition:

replicaset-definition.yaml

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: myapp-rs
  labels:
    app: myapp
    type: front-end
spec:
  template:
    metadata:
      name: myapp
      labels:
        app:myapp
    spec:
      containers:
      - name: nginx-controller
        image: nginx
  replicas: 2
  selector:
    matchLabels:
      app:myapp

Note that replicaset requires selectors, unlike replica controller

declarative way to update replicaset pods

$ kubectl replace -f replicaset-manifest.yaml

where in the replicaset-manifest.yaml file we updated the replicas

another way is to just update the file and use apply command

$ kubectl apply -f replicaset-manifest.yaml

the imperative ways of doing the same:

$ kubectl scale --replicas=3 rs/myapp
#OR
$ kubectl scale --replicas=3 -f replicaset-manifest.yaml 
#OR 
$ kubectl scale --replicas=3 replicaset name=myapp

the code above assumes that repliaset is different than 3, and the replicaset name is myapp

Deployments

Create a deployment and scale it up, imperatively

$ kubectl create deployment webapp --image=kodekloud/webapp-color
$ kubectl scale deployment/webapp --replicas=3

then expose the deployment created, containers listen on 8080 and expose the service port 30082, expose as NodePort

$ kubectl expose deployment webapp --type=NodePort --port=30082 --target-port=8080 --name=webapp-service

Just a few tricks for the exam: use --dry-run=client and -o yaml options to try if the code will work, instead of waiting pods to come online

Namespaces

to create a namespace, you can use a yaml file with namespace in the metadata
you can also do it imperatively by creating a ns, $ kubectl create ns namespace-name
to refer to other pods in another namespace, just append --namespace flag to the end
to create pods in an another namespace, you can also use namespace tag under the metadata, similar to name, label, annotations
to switch, use set-context under the config

$ kubectl config set-context mycurrentnamespace --namespace=myothernamespace

if you don't remember your current context or don't want to look it up

$ kubectl config set-context $(kubectl config current-context) --namespace=SwitchToThisNamespace

to set quota for a namespace, you can use ResourceQuota kind yaml

rc-definition.yaml

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-quota
  namespace: dev
spec:
  hard:
    pods: "10"
    requests.cpu: "4"
    requests.memory: 5Gi
    limits.cpu: "10"
    limits.memory: 10Gi

to connect to a source within the same namespace, you can directly refer to it,
to connect to a source in a different namespace, you use servicename.namespace.svc.cluster.local such as, db.prod.svc.cluster.local

Configuration

Commands and Arguments in YAML

Docker Commands
To run a container use
```
$ docker run <image-name>
```
- docker build from dockerfile
  Dockerfile
```
1FROM busybox
2CMD ["Sleep", "5"]
```
  Then use docker build function to run the Ubuntu container The Ubuntu container will run and exit after 5 seconds

What if you want to run the container but sleep after 10 seconds, but wanted to keep the default option as 5 seconds

Dockerfile

1FROM busybox
2ENTRYPOINT ["Sleep"]
3CMD ["5"]

$ docker build -t mybusybox .
$ docker run --name mybboxcontainer mybusybox 10

How can you pass these arguments via kubernetes yaml?

arguments-manifest.yaml

1apiVersion: v1
2kind: Pod
3metadata:
4  name: busybox 
5spec:
6  containers:
7    - name: busybox-container
8      image: mybusybox
9      args: ["10"]

note that the image name must match our docker build image tag, pushed to the dockerhub so we can conclude that args overrides CMD

what if I want to override the ENTRYPOINT?, then add command above args

arguments-manifest.yaml

1apiVersion: v1
2kind: Pod
3metadata:
4  name: busybox 
5spec:
6  containers:
7    - name: busybox-container
8      image: mybusybox
9      command: ["sleep"]
10      args: ["10"]

Environment Variables

how to pass environment variables to pods?
```
1...
2env:
```

name: APP_COLOR value: pink

1Note that the environment variables are arrays, hence the dash 
2Structure must be name/value pairs

Also possible to pass information via configMaps and Secrets

1...
2env: 
3  - name: APP_COLOR
4    valueFrom: 
5      configMapKeyRef:
6
7  - name: APP_SECRET
8    valueFrom:
9      secretKeyRef:
10
11...

configMaps

Motivation of configMap is to cope with increasing complexity under env tags
As the application grows, managing each env under Pod definition gets more complex, so configMaps are to rescue
configMap variables are key value pairs and injected to pods
Typical of k8s, we can do it imperatively and declaratively, before injecting to pods

Imperatively:

$ kubectl create configmap 
  configName --from-literal=key=value
$ kubectl create configmap
  configName --from-file=config.filepath

Declaratively:

$ kubectl create -f configmap.yaml

configmap.yaml

apiVersion: v1
kind: configMap
metadata: 
  name: app-config 
data:
  key:value
  anotherkey:anothervalue

How to refer to configmaps in pod template?

pod-template.yaml

apiVersion: v1
kind: Pod 
metadata:
  name: simple-app
  labels:
    env: dev
spec: 
  containers:
    - name: containername
      image: mycontainerimage
      ports:
        - containerPort: 8080
      envFrom:
        - configMapRef:
            name: app-config #Must match configMap name

there are 3 ways to inject variables

1#option 1
2...
3envFrom:
4  - configMapRef: 
5      name: app-config 
6...
7#option 2 (as single env)
8env: 
9  - name: CONFIG_KEY
10    valueFrom:
11      configMapKeyRef:
12          name: app-config 
13          key: CONFIG_KEY
14
15#option 3
16volumes:
17  - name: app-config-volume 
18    configMap:
19      name: app-config

Secrets

Motivation is similar to configMap but for secret items such as api keys or passwords
Similar to ConfigMaps it can be done imperatively and declaratively
Declaratively, it can be passed as an env:

pod-definition-extract.yaml

...
spec:
  containers:
    - name: myapp
      envFrom: 
        -secretRef:
            name: app-secret

apiVersion: v1
kind: Secret
metadata:
  name: app-secret
data:
  DB_Host: ab12d3

Secrets can be attached as volume too:

volumes:
- name: app-secret-volume 
  secret: 
    secretName: app-secret

the information under secret.yaml are encrypted using base64, but can be decrypted too
Secrets can be injected as env, single env and volumes
Secret volumes mounted to the pod creates individual files
Since secrets are encoded via base64, it can be decoded as well, so it's not that secure
However, if the secrets are not checked into the code repository, can make it safer
Also, enabling encryption at rest for secrets so they are stored encrpyted in /etcd
Secrets are sent to pods if only needed, kubelet stores secrets in tmpfs in memory, not on disk and once that pod is deleted, kubelet will delete its local copy of the secret data as well
imperatively you can create secrets as:

$ kubectl create secret generic 
  secretName --from-literal=key=value

Security Context in Kubernetes

To allow pod to run as non-root, such as userID 1000, just add the securityContext
SecurityContext can also be added to the container level, so that other containers in the pod and the host system are not impacted

pod-manifest.yaml

apiVersion: v1
kind: Pod
metadata: 
  name: myPod
spec:
# Pod level security
  securityContext:
    runAsUser: 1000
  containers: 
    - name: Ubuntu
      image: Ubuntu
      command: ["sleep", "3600"]
      # Container level security, run as user 1000 instead of root 
      securityContext:
        runAsUser: 1000
        capabilities:
          add: ["MAC_ADMIN"] #only on the container level

Service Accounts

In kubernetes, there are user accounts and service accounts
Humans use user accounts and bots/applications can use service accounts
Service account examples are monitoring, logging, pipeline management
Examples are promatheus (unsure if typed correctly) and jenkins
The applications that interact with my cluster has to be authenticated, for that we use service accounts
to create a service account imperatively

$ kubectl create serviceaccount ServiceAccountName

to get service accounts

$ kubectl get serviceaccount

to describe the service account, called ServiceAccountName

$ kubectl describe ServiceAccountName

When service accounts interact with Kubernetes API, they have to be authenticated
When a service account is created, it also creates a Token, the token is saved in the secret object
Outside the scope of this course, however on a high level the structure for external application to interact with API, you create a service account, assign permissions using role-based control, export the service token to the application to authenticate
What if the third party application itself is hosted in the cluster? Then simply mount the secret as a volume to the pod
To add service account to the pod, just add the following to pod yaml

mypod-manifest.yaml

1apiVersion: v1
2kind: Pod 
3metadata:
4  name: my-k8s-dashboard
5spec:
6  containers:
7  - name: mypodname
8    image: mypodimage
9  serviceAccount: dashboard-sa # the name of the serviceAccount object

Resource Requirements

Set the pod's CPU, Memory and Disk Requirements
K8s schedules the application to the necessary node
If there is no sufficient space available in the node, then it fails
To set the resource requirements, add the following to the container definition

mypod-manifest.yaml

1apiVersion: v1
2kind: Pod 
3...
4spec:
5  containers:
6  - name: myapp
7    image: containerimage
8    resources:
9      request:
10        memory: "1Gi"
11        cpu: 0.5

also can set the limits

mypod-manifest.yaml

1apiVersion: v1
2kind: Pod 
3...
4spec:
5  containers:
6  - name: myapp
7    image: containerimage
8    resources:
9      requests:
10        memory: "1Gi"
11        cpu: 0.5
12      limits:
13        memory: "2Gi"
14        cpu: 2

Taints & Tolerations

The motivation is to set Pod and Node relationship, so selected Pods can / cannot be run on selected Nodes
Tainting is applied to the Node and Toleration is applied to the Pod
Pods without taint toleration cannot be scheduled on the tainted Node
How to taint a node

$ kubectl taint nodes node-name key=value:taint-effect

where taint-effect is what happens to Pods that do not tolerate this taint

there are three taint effects NoSchedule: do not execute, what we have been discussing PreferNoSchedule: try not to schedule here ** NoExecute: do not schedule new and if there is any non-tolerant pods kill the pod (careful pods are gone!)
How to tolerate a pod

$ kubectl taint nodes node1 app=express: NoSchedule
--- mypod.yaml 
apiVersion: v1
kind: Pod
metadata: 
  name: myserver
spec:
  containers:
  - name: express-container
    image: node
  tolerations:
  - key: "app" #see app on the taint function above 
    operator: "Equal" # see the equal sign on the taint function above 
    value: "express"
    effect: "NoSchedule"

Note that toleration to taint does not guarantee that the tolerant Pod will run on the tainted Node, see Affinity Rules for that requirement

For untaint, check $ kubectl taint -h
To see which nodes the pods are running on $ kubectl get pods -o wide

Node Selectors

The objective is to select certain Node, i.e. database pod should run on a higher CPU machine
Say you want to add your application to a node that has significant CPU power, thus in your pod manifest you can add nodeSelector command

my-pod-manifest.yaml

1apiVersion: v1
2kind: Pod 
3metadata:
4  name: my-app
5spec:
6  containers:
7  - name: my-sifnificant-resource-requiring-app 
8    image: dockerhub-image-name
9  nodeSelector:
10    sizeCPU: significantlyhigh

note that nodeSelector key/value pair is something we define, thus we should name the node to match this pod manifest

$ kubectl label nodes NodeName label-key=value-key
$ kubectl label nodes myNode sizeCPU=significantlyhigh

Note that the node selectors work for single label and cannot use logical operators, such as select any node where the label is either X or Y, or select any node where the label is NOT equal to X

To select nodes such as above, use Node Affinity

Node Affinity

To place a pod to a certain node, you can label the node and use nodeSelector in the Pod spec, such as addition of size=Large
What if you want to create more complicated selection, such as size NOT small or size Medium or Large?
Such complex selection is achieved via Node Affinity on pod definition

mypod-manifest.yaml

1apiVersion: v1 
2kind: Pod 
3metadata:
4  name: nginx 
5spec:
6  affinity: 
7    nodeAffinity:
8      requiredDuringSchedulingIgnoredDuringExecution:
9        nodeSelectorTerms:
10        - matchExpressions:
11          - key: disktype
12            operator: In 
13            values:
14            - ssd 
15  containers:
16  - name: nginx 
17    image: nginx 
18    imagePullPolicy: IfNotPresent

What if a sneaky engineer updates the node label along the way, will the Pods get deleted?
The answer is under the nodeAffinity line requiredDuringSchedulingIgnoredDuringExecution preferredDuringSchedulingIgnoredDuringExecution ** Planned to be added to K8s: requiredDuringSchedulingRequiredDuringExecution

Multi-Container Pods

Multi Containers

Most of the time, one pod contains single container
And sometimes (and almost always in distributed systems a.k.a microservices) you want to trace your app logs, track the traffic etc.
However, you don't want to bloat the app with that, so you add another container to the pod that takes care of such app support functions/services, which scales up/down along with app container
There are three patterns: Ambassador, Adapter, Sidecar
To add extra container into pod, just expand the pod spec

my-pod-manifest.yaml

1apiVersion: v1
2kind: Pod 
3metadata:
4  name: my-app
5spec:
6  containers:
7  - name: my-web
8    image: ibmcr-web-app 
9  - name: logger 
10    image: ibmcr-logger

Observability

Readiness

Readiness probe helps Scheduler to wait before sending the traffic to the newly created pod
That is pretty helpful if you are running a script that creates pods / deployments back to back

readiness-pod-manifest.yaml

1apiVersion: v1
2kind: Pod
3metadata:
4  labels:
5    test: liveness
6  name: liveness-exec
7spec:
8  containers:
9  - name: liveness
10    image: k8s.gcr.io/busybox
11    args:
12    - /bin/sh
13    - -c
14    - touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600
15    #OPTION 1:
16    readinessProbe:
17      exec:
18        command:
19        - cat
20        - /tmp/healthy
21      initialDelaySeconds: 5
22      periodSeconds: 5
23    
24    OPTION 2:
25    readinessProbe:
26      httpGet:
27        path: /healthz
28        port: 8080
29        httpHeaders:
30        - name: Custom-Header
31          value: Awesome
32      initialDelaySeconds: 3
33      periodSeconds: 3

Liveliness

Liveliness probe is the same as the readiness probe, however serves for a different reason
Liveliness checks if your container is responding to client requests. Sometimes, your pod can be running and ready but your container is frozen or your code is broken
For YAMLs use livelinessProbe instead of readinessProbe and the rest is the same

Logging

use kubectl logs -f [pod-name][container-name-if-multiple-containers-in-the-pod]

$ kubectl logs -f my-pod my-first-container

If the container has stopped, add -c flag, if you have multiple containers add --all-containers=true flag
Kubernetes does not have native metrics monitoring solution, so we use 3rd party such as promatheus or ELK stack. Kubernetes metrics server is also available at github https://github.com/kubernetes-incubator/metrics-server.git
Clone the link and apply the files under /deploy/[metrics-version] folder
Similar to linux performance command 'top' use kubectl top [element] where element is node or pod

Pod Design

Labels and Selectors

Labels help multiple objects to be accessed at once
For replicaSets or deployments, the label for the replicaset can be different from pod spec, the most important line is Selector in the yaml to select the pods

Rolling Updates and Rollbacks

Jobs and CronJobs

Services and Networking

Services and NodePort

Pod to pod communication can be done through services, because Pods are ephemeral, when pods are deleted the assigned IPs are gone
Services keep track of the pods and can expose the service to outside of cluster
There are 3 type of services, ClusterIP, NodePort and LoadBalancer
ClusterIP is the default choice, it creates a cluster wide IP address that Pods can communicate with
NodePort is ClusterIP + a static port open within the Node, thus all pods are accessible from the port; to access you can ping NodeIP:NodePort
LoadBalancer allows a service to distribute the traffic among the pods in the cluster. Can be run on cloud providers which support LBs.

NodePort

From the Service object perspective, Service object has a ClusterIP and a Port. Service object connects to a Pod at the TargetPort Node Port opens up a static port with a range between 30000 and 32767 on the Node (i.e machine)

1apiVersion: v1 
2kind: Service 
3metadata:
4  name: myapp-svc 
5spec: 
6  type: NodePort 
7  ports:
8  - targetPort: 80
9    port: 80
10    nodePort: 30008 # optional, if not given, auto selected 
11  #selector is needed to match the pod label 
12  selectors:
13    name: myapp

1apiVersion: v1 
2kind: Service 
3metadata:
4  name: myapp-svc 
5spec: 
6  type: ClusterIP #default option, can be left out  
7  ports:
8  - targetPort: 80 #
9    port: 80
10    nodePort: 30008 # optional, if not given, auto selected 
11  #selector is needed to match the pod label 
12  selectors:
13    name: myapp

Certified Kubernetes Application Developer

Core Concepts

Nodes, Clusters

Components

Master vs Worker nodes

Pods

Replication controller

Deployments

Namespaces

Configuration

Commands and Arguments in YAML

Environment Variables

configMaps

Secrets

Security Context in Kubernetes

Service Accounts

Resource Requirements

Taints & Tolerations

Node Selectors

Node Affinity

Multi-Container Pods

Multi Containers

Observability

Readiness

Liveliness

Logging

Pod Design

Labels and Selectors

Rolling Updates and Rollbacks

Jobs and CronJobs

Services and Networking

Services and NodePort

NodePort

State Persistence