Introduction

Kubernetes is a service orchestration framework that provides many of the plumbing pieces required for running services. These services include ...

• DNS for naming and discovery of services,
• Load balancer to distribute requests across many instances of a service,
• Automatic recovery when a service crashes,
• Automatic scaling when a service comes under load,
• Password / certificate / secrets management for services,
• etc..

Containers

Kubernetes is structured around containers.

In the context of containers, an ...

• image is an application (or set of applications) packaged with all of its dependencies as an immutable and isolated filesystem. The filesystem typically contains all dependencies required for the application(s) to run sealed at their correct version:

  • libraries (e.g. correct version of libssh),
  • applications (e.g. correct version bash and Python),
  • files (e.g. embedded SQLite databases)
  • etc..

  Images also typically include metadata describing its needs and operational standards. For example, the metadata may stipulate that the image ...

  • launches by running /opt/my_app/run.sh
  • requires 4gb of memory, 1.5 CPU cores
  • etc..

• container is an instance of an image. A container creates an isolated copy of the image's filesystem, isolates the resources required for that image, and launches the entry point application for that image. That container can't see or access anything outside of the container unless explicitly allowed to by the user. For example, opening a port 8080 on a container won't open port 8080 on the host running it, but the user can explicitly ask that port 8080 in the container map to some port on the host.

As shown in the entity diagram above, each container is created from a single image, but that same image can be used for to create multiple containers. Another way to think about it is that an image is the blueprint of a factory and a container is the actual factory built from that blueprint. You can build multiple factories from the same blueprint.

Kubernetes requires two core components to run:

• open container initiative (OCI) runtime - A runtime responsible for only creating and launching containers.
• container runtime interface (CRI) - A runtime responsible for the high-level management of containers and images: image management, image distribution, container mounts / storage, container networking, etc..

Different vendors provide different implementations of each. For example, certain vendors provide an OCI runtime that use virtualization technology for isolation instead of standard Linux isolation (e.g. cgroups).

OCIs and CRIs are also the basis for container engines. Container engines tools responsible for creating and running containers, creating images, and other high-level functionality such as local testing of containers. Docker Engine is an example of a container engine.

Objects

Kubernetes breaks down its orchestration as a set of objects. Each object is of a specific type, referred to as kind. The main kinds are ...

• Node - A physical worker machine that runs containers.
• Pod - A set of containers tightly coupled to run in unison on a single node.
• Persistent Volume - A storage mechanism that lets pods persist and / or share data.
• Service - A load balancer that routes traffic to pods.
• Configuration Map - A configuration mechanism for applications running within pods (non-security related configurations).
• Secret - A security configuration mechanism for applications running within pods (e.g. passwords as certificates).
• Ingress - An access point in which traffic comes in from the outside world to the internal Kubernetes network.

Of these kinds, the two main ones are nodes and pods.

Nodes, pods, and other important kinds are discussed further on in this document.

Labels

An object can have two types of key-value pairs associated with it:

• Labels - These key-value pairs organize objects into logical groups, such that those groups can be targeted as a whole (e.g. give me all pods running Wordpress).
• Annotations - These key-value pairs allow tools to gather and share information about an object (e.g. when was this object created).

Finding objects based on labels is done via label selectors, described in the following table.

Kubernetes uses labels to orchestrate. Labels allow objects to have loosely coupled linkages to each other as opposed to tightly coupled parent-child / hierarchy relationships. For example, a load balancer decides which pods it routes requests to by searching for pods using label selector.

If there are a large number of labels / annotations, either because the organization set them directly or because they're being set by external tools, the chance of a collision increases. To combat this, keys for labels and annotations can optionally include a prefix (separated by a slash) that maps to a DNS subdomain to help disambiguate it. For example, company.com/my_key rather than just having my_key.

Configuration

Objects can be created, accessed, and modified through either a REST web interface or a command-line interface called kubectl. Changes can be supplied in two ways:

• Imperative configuration - specify which actions to take on which objects (via kubectl).

  kubectl label pods my_pod unhealthy=true                   # Set label on pod "my_pod"
  kubectl set pods my_pod --requests='cpu=500m,memory=128Mi' # Set pod "my_pod"'s resource reqs
  

• Declarative configuration - supply the entire object as YAML or JSON, called a manifest (via kubectl or REST).

  #
  # obj.yaml
  #
  apiVersion: v1
  kind: Pod
  metadata:
    name: my-pod
    labels:
      unhealthy: true
  spec:
    containers:
      - image: my-image:1.0
        name: my-container
        resources:
          requests:
            cpu: "500m"
            memory: "128Mi"
  

  kubectl apply -f obj.yaml   # Apply object in file "obj.yaml"
  

Generally, declarative configurations are preferred over imperative configurations. When a declarative configuration is submitted, Kubernetes runs a reconciliation loop in the background that changes the object to match the submitted manifest, creating that object if it doesn't already exist. Contrast this to the imperative configuration method, where changes have to be manually submitted by the user one by one.

Kinds

The following subsections give an overview of the most-used kinds and example manifests for those kinds. All manifests, regardless of the kind, require the following fields ...

• apiVersion: API version.
• kind: Class of object (kind).
• metadata.name: Name of object.

apiVersion: v1
kind: Pod
metadata:
  name: my-name
  annotations:
    author: "Jimbo D."
    created_on: "Aug 20 2021"
  labels:
    app_server: jetty

In addition, the metadata.labels and metadata.annotations contain the object's labels and annotations (respective).

Pod

Containers are deployed in Kubernetes via pods. A pod is a set of containers grouped together, often containers that are tightly coupled and / or are required to work in close proximity of each other (e.g. on the same host).

By default, containers within a pod are isolated from each other (e.g. isolated process IDs) except for sharing the same ...

• network (containers within a pod have the same IP, same host, and share the port space).
• IPC bus (containers within a pod can communicate with each other over POSIX message queues / System V IPC channels).
• volumes (containers within a pod may have shared storage assigned to them in addition to their isolated storage).

While the point of Kubernetes is to orchestrate containers over a set of nodes, the containers for a pod are all guaranteed to run on the same node. As such, pods are usually structured in a way where their containers are tightly coupled and uniformly scale together. For example, imagine a pod with comprised of a container running a WordPress server and a container running the MySQL database for that WordPress server. This would be a poor example of a pod because the two containers within it ...

1. don't scale uniformly (e.g. you may need to scale the database up before the WordPress server, or vice versa).
2. don't communicate over anything other than the network (e.g. they don't need a shared volume).
3. are intended to be distributed (e.g. it's okay for them to be running on separate machines).

Contrast that to a pod with a container running a WordPress server and a container that pushes that WordPress server's logs to a monitoring service. This would be a good example of a pod because the two containers within ...

1. communicate over the filesystem (e.g. application server is writing logs to a shared volume and the log watcher is tailing them).
2. aren't intended to be distributed (e.g. log watcher is intended for locally produced logs).
3. are written by different teams (e.g. SRE team wrote the log watcher image while another team wrote the application server image).

Example manifest:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
    - image: my-image:1.0
      name: my_container
      resources:
        requests:
          cpu: "500m"
          memory: "128Mi"
        limits:
          cpu: "1000m"
          memory: "256Mi"
      volumeMounts:
        - mountPath: "/data"
          name: "my_data"
      ports:
        - containerPort: 8080
          name: http
          protocol: TCP
      livenessProbe:
        httpGet:
          path: /healthy
          port: 8080
        initialDelaySeconds: 5
        timeoutSeconds: 1
        periodSeconds: 10
        failureThreshold: 3
      readinessProbe:
        httpGet:
          path: /ready
          port: 8080
        initialDelaySeconds: 5
        timeoutSeconds: 1
        periodSeconds: 10
        failureThreshold: 3
  volumes:
    - name: "my_data"
      hostPath:
        path: "/var/lib/my_data"  # literally mounts a path from the worker node? not persistent if node modes
    - name: "my_data_nfs"
      nfs:
        server: nfs.server.location
        path: "/path/on/nfs"

Images

Each container within a pod must have an image associated with. Images are specified in the Docker image specification format, where a name and a tag are separated by a colon (e.g. my-image:1.0).

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
    - name: my-container1
      image: my-image:1.0
    - name: my-container2
      image: my-image:2.4

Pull Policy

Each container in a pod has to reference an image to use. How Kubernetes loads a container's image is dependent on that container's image pull policy.

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
    - name: my-container
      image: my-image:1.0
      imagePullPolicy: IfNotPresent  # Only download if the image isn't present

A value of ...

• IfNotPresent only downloads the image if it's not already locally present on the node.
• Always always downloads the image.
• Never never downloads the image (will fail if image does not exist locally on the node).

If unset, the image pull policy differs based on the image tag. Not specifying a tag or specifying latest as the tag will always pull the image. Otherwise, the image will be pulled only if it isn't present.

Private Container Registries

Images that sit in private container registries require credentials to pull. Private container registry credentials are stored in secret objects of type kubernetes.io/dockerconfigjson in the format of Docker's config.json file.

apiVersion: v1
kind: Secret
metadata:
  name: my-docker-creds
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: ... # base64 encoded ~/.docker/config.json goes here

Those secret objects are then referenced in the pod's image pull secrets list.

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  # Place secret here. This is a list, so you can have many container registry credentials
  # here.
  imagePullSecrets:
    - name: my-docker-creds
  containers:
    - name: my-container
      image: my-registry.example/tiger/my-container:1.0  # Image references registry.

Resources

Each container within a pod can optionally declare a ...

• minimum set of resources that it needs to operate,
• maximum set of resources that it will ever need to operate.

Setting these options allows Kubernetes to choose a node with enough available resources to run that pod.

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
    - name: my-container
      image: my-image:1.0
      resources:
        requests:  # Minimum CPU and memory for this container
          cpu: "500m"
          memory: "128Mi"
        limits:    # Maximum CPU and memory for this container
          cpu: "1000m"
          memory: "256Mi"

requests are the minimum resources the container needs to operate while limits are the maximum resources the container can have. Some resources are dynamically adjustable while others require the pod to restart. For example, a pod ...

• can have its CPU usage dynamically adjusted because Kubernetes can just ask the operating system's CPU scheduler to give it less/more time.
• can't have its memory usage dynamically adjusted because if it loses access to a block of memory, the applications running within the pod won't know and will likely crash.

The example above lists out CPU and memory as viable resource types. The unit of measurement for ...

• cpu is either in ...
  • whole cores: no suffix
  • millicpus: suffix of m (1 core is equivalent to 1000m -- e.g. 0.5 = 5000m).
• memory is either in ...
  • bytes: no suffix
  • 1000 scale: suffix of k = 1000, M = 1,000,000, G = 1,000,000,000
  • power of two scale: suffix of k = 1024, M = 1,048,576, G = 1,073,741,824

Ports

Each container within a pod can expose ports to the cluster.

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
    - name: my-container
      image: my-image:1.0
      ports:
        - containerPort: 8080
          name: http
          protocol: TCP

The example above exposes port 8080 to the rest of the cluster (not to the outside world). Even with the port exposed, other entities on the cluster don't have a built-in way to discover the pod's IP / host or the fact that it has this specific port open. For that, services are required.

A pod can have many containers within it, and since all containers within a pod share the same IP, the ports exposed by those containers must be unique. For example, only one container within the pod expose port 8080.

Command-line Arguments

An image typically provides a default entry point (process that gets started) and default set of arguments to run with. Each container within a pod can override these defaults.

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
    - image: my-image:1.0
      command: [/opt/app/my-app]
      args: [--no-logging, --dry-run]

Environment Variables

Each container within a pod can be assigned a set of environment variables.

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
    - name: my-container
      image: my-image:1.0
      env:
        - name: LOG_LEVEL
          value: "OFF"
        - name: DRY_RUN
          value: "true"

Once defined, an environment variables value can be used in other parts of the manifest using the syntax $(VAR_NAME). For example, an environment variable's value may be placed directly within an argument.

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
    - name: my-container
      image: my-image:1.0
      env:
        - name: LOG_LEVEL
          value: "OFF"
      args: [--logging_telemetry=$(LOG_LEVEL)]

Configuration

A container's configuration can come from both config maps and secrets. For config maps, a config map's key-value pairs can be accessed by a container via environment variables, command-line arguments, or volume mounts. To set a ...

• environment variable:

  apiVersion: v1
  kind: Pod
  metadata:
    name: my-pod
  spec:
    containers:
      - name: my-container
        image: my-image:1.0
        # Under each "env" entry to come from a config map, add a "valueFrom" that contains
        # the config map to pull an entry from and the key for the config map entry to pull.
        env:
          - name: ENV_VAR_NAME1  # Env var to assign value to.
            valueFrom:
              configMapKeyRef:
                name: my-config       # Config map to pull from
                key: CONFIG_MAP_KEY1  # Config map entry to get value from
          - name: ENV_VAR_NAME2
            valueFrom:
              configMapKeyRef:
                name: my-config
                key: CONFIG_MAP_KEY2
  

  In certain cases, you may want to map all entries within a config map directly as a set of environment variables. This is useful when many entries of a config map are required for configuration, so many that it becomes tedious and error-prone to map them all to environment variables manually.

  apiVersion: v1
  kind: Pod
  metadata:
    name: my-pod
  spec:
    containers:
      - name: my-container
        image: my-image:1.0
        # "envFrom" maps all entries of a config map as env vars.
        envFrom:
          - prefix: CONFIG_    # Prefix to tack on to each config map entry (optional).
            configMapRef:
              name: my-config
  

  ⚠️NOTE️️️⚠️

  If a config map name can't map to an environment variable, it's silently omitted. For example, env names can't contain dashes.

• command-line argument:

  apiVersion: v1
  kind: Pod
  metadata:
    name: my-pod
  spec:
    containers:
      - name: my-container
        image: my-image:1.0
        # You can't pass in config map entries directly as command-line arguments, but what
        # you can do is load them up first as environment variables and then reference the
        # environment variables in the "command" (or "args") field.
        env:
          - name: ENV_VAR_NAME1
            valueFrom:
              configMapKeyRef:
                name: my-config
                key: CONFIG_MAP_KEY1
          - name: ENV_VAR_NAME2
            valueFrom:
              configMapKeyRef:
                name: my-config
                key: CONFIG_MAP_KEY2
        command:
          - "/my-app.sh"
          - "$(ENV_VAR_NAME1)"
          - "$(ENV_VAR_NAME2)"
  

• volume mount:

  apiVersion: v1
  kind: Pod
  metadata:
    name: my-pod
  spec:
    # Place a "configMap" type volume into the pod.
    volumes:
      - name: config-volume
        configMap:
          name: my-config
          # "items" lists out specific config map entries to include and mounts each as
          # a specific filename. If you don't include this, all config map entries will
          # be included (filenames will map to config map entry names).
          items:
            - key: CONFIG_MAP_KEY1
              path: file1.cfg
            - key: CONFIG_MAP_KEY2
              path: file2.cfg
    # In the container, mount that volume to whichever containers you want.
    containers:
      - name: my-container
        image: my-image:1.0
        volumeMounts:
          - name: config-volume
            mountPath: /config
            readOnly: true       # Make the mount read-only (optional).
            defaultMode: "6600"  # File access permissions of mounted files (optional).
  

  If the directory you're mounting to already exists on the container, that existing directory is entirely replaced. In the example above, if the container already has a "/config" directory, it'll get replaced entirely with the config map mount (this is bad because the container's "/config" might have other necessary files required for the container to work). A workaround to this is to use the volume mount's "subPath" property, which allows you to mount a single file / directory from a volume.

  apiVersion: v1
  kind: Pod
  metadata:
    name: my-pod
  spec:
    volumes:
      - name: config-volume
        configMap:
          name: my-config
    containers:
      - name: my-container
        image: my-image:1.0
        # The use of "subPath" here ensures that that original "/config" directory on the
        # container doesn't go away. It remains in place, and files / directories are just
        # added to it.
        volumeMounts:
          - name: config-volume
            mountPath: /config/file1.cfg  # Destination file to mount to.
            subPath: CONFIG_MAP_KEY1      # Source config map entry name.
          - name: config-volume
            mountPath: /config/file2.cfg  # Destination file to mount to.
            subPath: CONFIG_MAP_KEY2      # Source config map entry name.
  

For secrets, a secret object's key-value pairs can be accessed in almost exactly the same way as config maps with almost exactly the same set of options and restrictions. To set a ...

• environment variable:

  apiVersion: v1
  kind: Pod
  metadata:
    name: my-pod
  spec:
    containers:
      - name: my-container
        image: my-image:1.0
        env:
          - name: ENV_VAR_NAME1
            valueFrom:
              secretKeyRef:  # This has been changed from "configMapKeyRef" to "secretKeyRef".
                name: my-secret
                key: SECRET_KEY1
          - name: ENV_VAR_NAME2
            valueFrom:
              secretKeyRef:  # This has been changed from "configMapKeyRef" to "secretKeyRef".
                name: my-secret
                key: SECRET_KEY2
  

• command-line argument:

  apiVersion: v1
  kind: Pod
  metadata:
    name: my-pod
  spec:
    containers:
      - name: my-container
        image: my-image:1.0
        env:
          - name: ENV_VAR_NAME1
            valueFrom:
              secretKeyRef:  # This has been changed from "configMapKeyRef" to "secretKeyRef".
                name: my-secret
                key: SECRET_KEY1
          - name: ENV_VAR_NAME2
            valueFrom:
              secretKeyRef:  # This has been changed from "configMapKeyRef" to "secretKeyRef".
                name: my-secret
                key: SECRET_KEY2
        command:
          - "/my-app.sh"
          - "$(ENV_VAR_NAME1)"
          - "$(ENV_VAR_NAME2)"
  

• volume mount:

  apiVersion: v1
  kind: Pod
  metadata:
    name: my-pod
  spec:
    volumes:
      - name: secret-volume
        secret:  # This has been changed from "configMap" to "secret"
          name: my-secret
    containers:
      - name: my-container
        image: my-image:1.0
        volumeMounts:
          - name: secret-volume
            mountPath: /secrets
            readOnly: true
  

Both config maps and secrets can be dynamically updated. If a pod is running when an update gets issued, it may or may not receive those updates depending on the configurations are exposed to the container:

• environment variables don't receive updates.
• command-line arguments don't receive updates.
• individual volume mounts don't receive updates.
• whole volume mounts do receive updates.

Command-line arguments and environment variables don't update because an application's command-line arguments and environment variables can't be changed from the outside once a process launches. Individual files/directories mounted from a volume don't update because of technical limitations related to how Linux filesystems work (see here). Whole volume mounts do update files under the mount, but it's up to the application to detect and reload those changed files.

The typical workaround to config map dynamic updates is to use deployments. In deployments, secrets / config maps and pods are bound together as a single unit, meaning that all pods restart automatically on any change.

Volume Mounts

A pod is able to supply multiple volumes, where those volumes may be mounted to different containers within that pod.

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  # Volumes supplied are listed here.
  volumes:
    - name: my-data1
      hostPath:
        path: "/var/lib/my_data1"
    - name: my-data2
      hostPath:
        path: "/var/lib/my_data2"
  # Each container in the pod can mount any of the above volumes by referencing its name.
  containers:
    - name: my-container
      image: my-image:1.0
      volumeMounts:
        # Mount "my-data1" volume to /data1 in the container's filesystem.
        - mountPath: /data1
          name: my-data1
        # Mount "my-data2" volume to /data2 in the container's filesystem.
        - mountPath: /data2
          name: my-data2

In the example above, the two volumes supplied by the pod are both of type hostPath. hostPath volume types reference a directory on the node that the pod is running on, meaning that if two containers within the same pod are assigned the same hostPath volume, they see each other's changes on that volume. The type of volume supplied defines the characteristics of that volume. Depending on the volume type, data on that volume ...

• may be shared across pods, shared across containers within a single pod, or not shared at all
• may be encrypted or unencrypted
• if shared, may have delayed and / or unreliable read-write consistency
• if not shared, may be transient or persistent across pod or container restarts
• if not shared, may be transient or persistent when a pod is moved to a new node
• etc..

Each supplied volume within a pod can either reference a direct piece of storage or it can reference a persistent volume claim. In most cases, directly referencing a piece of storage (as done in the above example) is discouraged because it tightly couples the pod to that storage and its parameters. The better way is to use persistent volume claims, where volumes are assigned from a pool (or dynamically created and assigned) to pods as required. Assuming that you have a persistent volume claim already created, it can be referenced by using persistentVolumeClaim as the volume type.

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  volumes:
    - name: my-data
      persistentVolumeClaim:  # Volume type of "persistentVolumeClaim"
        claimName: my-data-pv-claim
  containers:
    - name: my-container
      image: my-image:1.0
      volumeMounts:
        - mountPath: /data
          name: my-data

Lifecycle

A pod's lifecycle goes through several phases:

• Pending - Pod is waiting to run. This could be either because it hasn't been scheduled on a node yet or because it has been scheduled on a node but some of its containers aren't ready to run yet (e.g. images for those containers are still being downloaded).
• Running - Pod is running. It's been scheduled on a node, all of its containers are ready to run, and at least one of those containers is either running, starting, or restarting.
• Success - Pod's containers all finished successfully.
• Failed - Pod's container all finished, some unsuccessfully. This could be either because some containers terminated with an error or because Kubernetes itself terminated those containers for some reason.
• Unknown - Special marker indicating that the state of a pod couldn't be obtained (e.g. from a network outage).

Each container in a pod can be in one of several states:

• Waiting - Container is performing operations it needs to run (e.g. downloading image for the container).
• Running - Container is running.
• Terminated - Container has terminated.

The following subsections detail various lifecycle-related configurations of a pod and its containers.

Probes

Probes are a way for Kubernetes to check the state of a pod. Containers within the pod expose interfaces which Kubernetes periodically pings to determine what actions to take (e.g. restarting a non-responsive pod).

Different types of probes exists. A ...

• liveness probe is something that Kubernetes pings to check if a container is alive and responsive (typical action on failure: restart pod).
• readiness probe is something that Kubernetes pings to check if a container is able to accept traffic (typical action on failure: stop traffic until readiness probes on all containers pass).
• startup probe is something that Kubernetes pings to check if a container has started (typical action on failure: retry until startup probes on all containers pass before allowing liveness and readiness probes).

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
    - name: my-container
      image: my-image:1.0
      # Probe to check if a container is alive or dead. Performs an HTTP GET with path
      # /healthy at port 8080.
      livenessProbe: 
        httpGet:
          path: /healthy
          port: 8080
        initialDelaySeconds: 5
        timeoutSeconds: 1
        periodSeconds: 10
        failureThreshold: 3
      # Probe to check if a container is able to service requests. Performs an HTTP GET
      # with path /ready at port 8080.
      readinessProbe:
        httpGet:
          path: /ready
          port: 8080
        initialDelaySeconds: 5
        timeoutSeconds: 1
        periodSeconds: 10
        failureThreshold: 3

In the example above, each of the probes check an HTTP server within the container at port 8080 but at different paths. The field ...

• initialDelaySeconds is the number of seconds to wait before performing the first probe.
• timeoutSeconds is the number of seconds to wait before timing out.
• periodSeconds is the number of seconds to wait before performing a probe.
• failureThreshold is the maximum number of successive failure before Kubernetes considers the probe failed.
• successThreshold is the maximum number of successive successes before Kubernetes considers the probe passed.

There are types of probes other than httpGet. A probe of type ...

• httpGet will perform an HTTP GET operation to a server on the container and fail it if it's non-responsive.
• tcpSocket will attempt to connect a TCP socket to the container and fail if the container doesn't accept.
• exec will run a command on the container and fail if it gets a non-zero exit code.

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
    - name: my-container
      image: my-image:1.0
      readinessProbe:
        exec:
          command:
            - cat
            - /tmp/some_file_here
        initialDelaySeconds: 5
        timeoutSeconds: 1
        periodSeconds: 10
        failureThreshold: 3

Graceful Termination

Kubernetes terminates pods by sending a SIGTERM to each container's main process, waiting a predefined amount of time, then forcefully sending a SIGKILL to that same process if the process hasn't shut itself down. The predefined waiting time is called the termination grace period, and it's provided so the application can perform cleanup tasks after it's received SIGTERM (e.g. emptying queues).

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  restartPolicy: Always
  terminationGracePeriodSeconds: 60  # Default to 30
  containers:
    - name: my-container
      image: my-image:1.0

On termination (either via SIGTERM or voluntarily), a pod's container can write a message to a special file regarding the reason for its termination. The contents of this file be visible in the pod container's "last state" property.

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  restartPolicy: Always
  containers:
    - name: my-container
      image: my-image:1.0
      terminationMessagePath: /var/exit-message  # Defaults to /dev/termination-log

Maximum Runtime

The runtime of a pod can be limited such that, if continues to run for more than some duration of time, Kubernetes will forcefully terminate it and mark it as failed.

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  activeDeadlineSeconds: 3600  # When set, pod can't run for more than this many seconds
  containers:
    - name: my-container
      image: my-image:1.0

Lifecycle Hooks

Lifecycle hooks are a way for Kubernetes to notify a container of when ...

• it's been brought up, called a post-start hook.
• it's about to be brought down, called a pre-stop hook.

Similar to probes, containers within the pod expose interfaces which Kubernetes invokes. A lifecycle hook interface is similar to a probe interface in that it can be one of multiple types: httpGet, tcpSocket, and exec.

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
    - name: my-container
      image: my-image:1.0
      lifecycle:
        # Hook to invoke once the container's main process starts running. This hook runs
        # along-side the main process (not before it launches), but Kubernetes will treat
        # the container as if it's waiting for it to still be created until this hook
        # completes.
        #
        # A pod will be a "Pending" state until all each of its container post-start hooks
        # complete.
        postStart:
          exec:
            command: [sh, sleep 15]  # Artificial sleep
        # Hook to invoke just before the container is voluntarily terminated (e.g. its
        # moving to a new node). This hook runs first and once it's finished, a SIGTERM is
        # sent to the main container process, followed by a SIGKILL if the container's
        # main process hasn't terminated itself.
        #
        # It's important to note that the termination grace period begins as soon as the
        # pre-stop hook gets invoked, not after the pre-stop hook finishes.
        preStop:
          httpGet:
            path: /shutdown
            port: 8080

A post-start hook is useful when some form of initialization needs to occur but it's impossible to do that initialization within the container (e.g. initialization doesn't happen on container start and you don't have access to re-create / re-deploy the container image to add support for it). Likewise, a pre-stop hook is useful when some form of graceful shutdown needs to occur but it's impossible to do that shutdown within the container (e.g. shutdown procedures don't happen on SIGTERM and you don't have access to re-create / re-deploy the container image to add support for it).

Init Containers

Init containers are pod containers that run prior to a pod's actual containers. Their purpose is to initialize the pod in some way (e.g. writing startup data to some shared volume) or delay the start of the pod until some other service is detected as being online (e.g. database).

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  # Init containers are defined similarly to main containers, but they're run before main containers, one after the other in the order they're defined. After the last init container successfully completes, the main containers for the pod start.
  initContainers:
    - name: my-initA
      image: my-init-imageA:1.0
    - name: my-initB
      image: my-initB-image:1.0
      command: ['launchB', '--arg1']
  containers:
    - name: my-container
      image: my-image:1.0

Restart Policy

Restart policy is the policy Kubernetes uses for determining when a pod should be restarted.

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  restartPolicy: Always
  containers:
    - name: my-container
      image: my-image:1.0

A value of ...

• Always always restarts the pod regardless of how it exists (default).
• OnFailure only restarts the pod if it failed execution.
• Never never restarts the pod.

Always is typically used when running servers that should always be up (e.g. http server) while the others are typically used for one-off jobs.

When a container within a pod fails, that entire pod is marked as failed and may restart depending on this property. Kubernetes exponentially delays restarts so that, if the restart is happening due to an error, there's some time in between restarts for the error to get resolved (e.g. wait for some pending network resource required by the pod comes online). The delay increases exponentially (10 seconds, 20 seconds, 40 seconds, 80 seconds, etc..) until it caps out at 5 minutes. The delay resets once a restarted pod is executing for more than 10 minutes without issue.

Service Discovery

For a pod to communicate with services, it needs to be able to discover the IP(s) of those services. The mechanisms for discovering services within a pod are environment variables and DNS.

These service discovery mechanisms are details in the subsections below.

Environment Variables

When a pod launches, all services within the same namespace have their IP and port combinations added as environment variables within the pod's containers. The environment variable names are in the format {SVCNAME}_SERVICE_HOST / {SVCNAME}_SERVICE_PORT, where {SVCNAME} is the service converted to uppercase and dashes swapped with underscores. For example, service-a would get converted to SERVICE_A.

SERVICE_A_SERVICE_HOST=10.111.240.1
SERVICE_A_SERVICE_PORT=443
SERVICE_B_SERVICE_HOST=10.111.249.153
SERVICE_B_SERVICE_PORT=80

If a service exposes multiple ports, only the first port goes in {SVCNAME}_SERVICE_PORT. When multiple ports are present, additional environment variables get created in the format {SVCNAME}_SERVICE_PORT_{PORTNAME}, where {PORTNAME} is the name of service's port modified the same way that {SVCNAME} is. For example, service-c with two exposed ports named web-1 and metrics-1 would get converted to SERVICE_C_SERVICE_PORT_WEB_1 and SERVICE_C_SERVICE_PORT_METRICS_1 respectively.

SERVICE_C_SERVICE_HOST=10.111.240.1
SERVICE_C_SERVICE_PORT=443
SERVICE_C_SERVICE_PORT_WEB_1=443
SERVICE_C_SERVICE_PORT_METRICS_1=8080

Using environment variables for service discovery has the following pitfalls:

• Services outside the pod's namespace aren't provided.
• Services added to the namespace after the pod launches won't be picked up (env vars can only be changed before the launch of a container process).
• Services removed from the name after the pod launches won't be picked up (env vars can only be changed before the launch of a container process).
• Service naming conflicts can happen during normalization (e.g. uppercase-ing and converting dashes to underscores can cause conflict: My-name and my_namE both end up as MY_NAME).

On the plus side, inspecting environment variables within a container essentially enumerates all services within the pod's namespace. Enumerating services isn't possible when using DNS for service discovery, discussed in the next section.

DNS

Kubernetes provides a global DNS server which is used for service discovery. Each pod is automatically configured to use this DNS server and simply has to query it for a service's name. If the queried service is present, the DNS server will return the stable IP of that service.

The general domain query format is {SVCNAME}.{NAMESPACE}.svc.{CLUSTERDOMAIN}, where ...

• {SVCNAME} is the name of the service.
• {NAMESPACE} is the name of the namespace that {SVCNAME} is in.
• {CLUSTERDOMAIN} is the cluster domain suffix of the cluster that {SVCNAME} and {NAMESPACE} are in.

For example, to query for the IP of service serviceA in namespace ns1 within a cluster that has the domain name suffix cluster.local, the domain name to query is serviceA.ns1.svc.cluster.local. Alternatively, if the pod doing the querying is ...

• within the same cluster, the cluster domain suffix can be omitted: serviceA.ns1.
• within the same cluster and namespace, both the namespace and the cluster domain suffix can be omitted: serviceA.

Using DNS for service discovery has the following pitfalls:

• Service port information isn't included in queries.
• Enumerating services isn't possible with DNS.

On the plus side, DNS queries can extend outside the pod's namespace and services started after a container launches are queryable. These aren't possible when using environment variables for service discovery, discussed in the previous section.

Metadata

Information about a pod and its containers such as ...

• pod name,
• pod labels and annotations,
• pod IP,
• pod namespace,
• node running the pod,
• container resource limits,
• etc..

... can all be accessed within the container via either a file system mount or environment variables.

Environment Variables

All pod information except for labels and annotations can be assigned to environment variables. This is because a running pod can have its labels and annotations updated but the environment variables within a running container can't be updated once that container starts (updated labels / annotations won't show up to the container).

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
    - image: my-image:1.0
      name: my-container
      resources:
        requests:
          cpu: 15m
          memory: 100Ki
        limits:
          cpu: 100m
          memory: 4Mi
      env:
        # These entries reference values that would normal be "fields" under a running pod
        # in Kubernetes. That is, these entries reference paths that you would normally see
        # when you inspect a pod in Kubernetes by dumping out its YAML/JSON. For example,
        # by running "kubectl get pod my_pod -o yaml" -- it produces a manifest but with
        # many more fields (dynamically assigned fields and fields with default values
        # filled out).
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name  # Pulls in "my_pod".
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace  # Pulls in the default namespace supplied by Kubernetes.
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP  # Pulls in the pod's IP.
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName  # Pulls in the name of the node that the pod's running on.
        # When referencing resource requests / limits for a container, if you're requesting those
        # for a different container than the one the container you're assigning the env var to,
        # you'll need to supply a "containerName" field. Otherwise, you can omit the
        # "containerName" field.
        #
        # Resource requests / limits by optionally be provided a "divisor" field, which will
        # divide the value before assigning it.
        - name: CPU_REQUEST
          valueFrom:
            resourceFieldRef:
              resource: requests.cpu
              containerName: my-container  # If you omit this, it'll default to "my-container" anyways.
              divisor: 5m  # Divide by 5 millicores before assigning (15millicores/5millicores=3)
        - name: CPU_LIMIT
          valueFrom:
            resourceFieldRef:
              resource: limits.cpu
              containerName: my-container  # If you omit this, it'll default to "my-container" anyways.
              divisor: 5m  # Divide by 5 millicores before assigning (100millicores/5millicores=20)
        - name: MEM_REQUEST
          valueFrom:
            resourceFieldRef:
              resource: requests.memory
              containerName: my-container  # If you omit this, it'll default to "my-container" anyways.
              divisor: 1Ki  # Divide by 1 kibibyte before assigning (100Kebibytes/1Kibibyte=100)
        - name: MEM_LIMIT
          valueFrom:
            resourceFieldRef:
              resource: limits.memory
              containerName: my-container  # If you omit this, it'll default to "my-container" anyways.
              divisor: 1Ki  # Divide by 1 kibibyte before assigning (4Mebibytes/1Kibibyte=4096)

Volume Mount

All pod information can be exposed as a volume mount, where files in that mount map to pieces of information. Unlike with environment variables, a volume mount can contain labels and annotations. If those labels and annotations are updated, the relevant files within the mount update to reflect the changes. It's up to the application running within the container to detect and reload those updated files.

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
  labels:
    drink: Pepsi
    car: Volvo
  annotations:
    key1: value1
    key2: |
      Good morning,
      Today is Sunday.
spec:
  volumes:
    # A volume of type downwardAPI will populate with files, where each file contains the
    # value for a specific field. Fields are specified in a similar manner to the
    # environment variable version above.
    #
    # Each "path" under "items" is file within the volume.
    - name: downward_vol
      downwardAPI:
        items:
          - path: podName
            fieldRef:
              fieldPath: metadata.name
          - path: podNamespace
            fieldRef:
              fieldPath: metadata.namespace
          - path: podIp
            fieldRef:
              fieldPath: status.podIP
          - path: nodeName
            fieldRef:
              fieldPath: spec.nodeName
          - path: cpuRequest
            resourceFieldRef:
              resource: requests.cpu
              containerName: my-container  # MUST BE INCLUDED, otherwise it's impossible to know which container.
              divisor: 5m
          - path: cpuLimit
            resourceFieldRef:
              resource: limits.cpu
              containerName: my-container  # MUST BE INCLUDED, otherwise it's impossible to know which container.
              divisor: 5m
          - path: memRequest
            resourceFieldRef:
              resource: requests.memory
              containerName: my-container  # MUST BE INCLUDED, otherwise it's impossible to know which container.
              divisor: 1Ki
          - path: memLimit
            resourceFieldRef:
              resource: limits.memory
              containerName: my-container  # MUST BE INCLUDED, otherwise it's impossible to know which container.
              divisor: 1Ki
          # The following two entries supplies labels and annotations. Note that, if labels
          # or annotations change for the pod, the files in this volume will be updated to
          # reflect those changes.
          #
          # Each file below will contain multiple key-value entries. One key-value entry per line, where
          # the key and value are delimited by an equal sign (=). Values are escaped, so the new lines in
          # the multiline example annotation in this pod (see key2, where the value is a good morning
          # message) will be appropriately escaped.
          - path: "labels"
            fieldRef:
              fieldPath: metadata.labels
          - path: "annotations"
            fieldRef:
              fieldPath: metadata.annotations
  containers:
    - image: my-image:1.0
      name: my-container
      resources:
        requests:
          cpu: 15m
          memory: 100Ki
        limits:
          cpu: 100m
          memory: 4Mi
      # Mount the volume declared above into the container. The application in the container
      # will be able to access the metadata as files within the volume mount.
      volumeMounts:
        - name: downward_vol
          mountPath: /metadata

Node Placement

A pod can have soft / hard requirements as to which nodes it can run on. It's common to segregate which nodes a pod can / can't run when dealing with...

• multiple environments in the same cluster (e.g. testing nodes vs staging nodes vs production nodes)
• multiple zones in the same cluster (e.g. eastern US vs western US).
• specific hardware requirements (e.g. node that has a specific type of SSD, CPU, or GPU).
• etc..

Three different mechanisms are used to define these requirements:

• node selectors: hard requirements for which nodes a pod can run on.
• node taints: hard and soft requirements for which nodes a pod can't run on.
• node affinity: hard and soft requirements for which nodes a pod can and can't run on.

These mechanisms are documented in further detail in the subsections below.

Node Selectors

A node selector forces a pod to run on nodes that have a specific set of node labels.

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  # This pod can only run on nodes that have the labels disk=ssd and cpu=IntelXeon.
  nodeSelector:
    disk: ssd
    cpu: IntelXeon
  containers:
    - name: my-container
      image: my-image:1.0

Taints and Tolerations

A taint is a node property, structured as a key-value pair and effect, that repels pods. For a pod to be scheduled / executed on a node with taints, it needs tolerations for those taints. Specifically, that pod ...

• needs tolerations for any taints with effect NoSchedule or NoExecute.
• can have tolerations for any taints with effect PreferNoSchedule (having tolerations increases odds that pod gets scheduled on that node).

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  # This pod can only run on nodes that have the taints "environment=production:NoExecute" and
  # "environment=excess-capacity:NoExecute". The example below uses the "Equal" operator, but
  # there's also an "Exists" operator which will match any value ("value" field shouldn't be
  # set if "Exists" is used).
  tolerations:
    - key: environment
      operator: Equal
      value: production
      effect: NoSchedule
    - key: environment
      operator: Equal
      value: excess-capacity
      effect: NoSchedule
  containers:
    - name: my-container
      image: my-image:1.0

Node Affinity

Node affinity is a set of rules defined on a pod that repels / attracts it to nodes tagged with certain labels, either as soft or hard requirements.

• For hard requirements, a set of label selectors need to be specified.
• For soft requirements, a set of label selectors need to be specified along with weights that define the desirability of those selectors.

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  affinity:
    nodeAffinity:
      # This field lists the hard requirements for node labels. The possible operators
      # allowed are ...
      #
      #  * "In" / "NotIn" - Tests key has (or doesn't have) one of many possible values.
      #  * "Exists" / "DoesNotExist" - Tests key exists (or doesn't exist), value ignored.
      #  * "Gt" / "Lt" - Tests key's value is grater than / less than.
      #
      # Use "In" / "Exists" for attraction and "NotIn" / "DoesNotExist" for repulsion.
      #
      # In this example, it's requiring that the CPU be one of two specific Intel models and
      # the disk not be a hard-drive (e.g. it could be a solid-state drive instead).
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
            - key: cpu
              operator: In
              values: [intel-raptor-lake, intel-alder-lake]
            - key: disk-type
              operator: NotIn
              values: [hdd]
      # This field lists out the soft requirements for node labels and weights those
      # requirements. Each requirement uses the same types of expressions / operators as the
      # hard requirements shown above, but it also has a weight that defines the
      # desirability of that requirement. Each weight must be between 1 to 100.
      #
      # In this example, the first preference outweighs the second by a ratio of 10:2 (5x
      # more preferred).
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          preference:
            - matchExpressions:
              - key: ram-speed
                operator: gt
                values: [3600]
              - key: ram-type
                operator: In
                values: [ddr4]
        - weight: 20
          preference:
            - matchExpressions:
              - key: ram-speed
                operator: lt
                values: [3601]  # This is <, 3601 means speed needs to be <= 3600.
              - key: ram-type
                operator: In
                values: [ddr4]
  containers:
    - name: my-container
      image: my-image:1.0

Unlike ...

• node selectors, ...
  • the requirements for node affinity can be soft requirements.
  • the requirements for node affinity can repel as well as attract.
  • the requirements for node affinity has more ways to specify label selection criteria (more expressive).
• node taints,
  • the requirements for node affinity are attracting / repulsing based on node labels, not repulsing based on lack of pod tolerations.
  • node affinity can't evict running pods from a node whereas taints with a NoExecute effect which will cause evictions.

Pod Affinity

Pod affinity is a set of rules defined on a pod that repels / attracts it to the vicinity of other pods tagged with certain labels. Vicinity is determined via a topology key, which is a label placed on nodes to define where they live. For example, nodes within the same ...

• rack can have a "rack" label (e.g. nodes on rack 15 have label rack=15, nodes on rack 16 have label rack=16, ...).
• data center can have a "dc" label (e.g. nodes in data center 1 have label dc=1, nodes in data center 2 have label dc=2, ...).
• geographic region can have a "geo" label (e.g. nodes in Texas have label geo=Texas, nodes in Ohio have label geo=Ohio, ...).

Pod affinity defines pod attraction / repulsion by looking for pod labels and a topology key. For example, it's possible to use pod affinity to ensure that a pod gets scheduled on the same rack (via topology key) as another pod with the label app=api-server. There could be multiple pods with the label app=api-server, in which case the scheduler will pick one, figure out which rack it lives on, and place the new pod on that same rack.

Similar to node affinity, pod affinity can have hard and soft requirements, where soft requirements have weights that define the desirability of attraction / repulsion. Selector expressions and weights for pod affinity are defined similarly to those for node affinity.

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
    - image: my-image:1.0
      name: my-container
  affinity:
    # This field defines the rules specifically for **attraction**. Like with node affinity,
    # pod affinity uses ...
    #
    #  * "requiredDuringSchedulingIgnoredDuringExecution" for hard requirements.
    #  * "preferredDuringSchedulingIgnoredDuringExecution" for soft requirement.
    #  * the same selector operators as node affinity ("In", "NotIn", "Exists",
    #    "DoesNotExist", "Gt", and "Lt").
    #  * the same weight requirements as node affinity (range between 1 to 100 per soft
    #    requirement).
    #
    # Unlike with node affinity, the negation operators ("NotIn" / "DoesNotExist") don't
    # define repulsion, they just attract to pods that don't have something. For example,
    # here we're looking to have affinity to pods that don't have the labels
    # "stability-level=alpha" and "stability-level=alpha". In addition, it strongly
    # prefers to live on the same rack as pods with label "app=api-server" and less strongly
    # prefers to live on the same rack as pods with label "app=db-server".
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - topologyKey: rack
          labelSelector:
            matchExpressions:
              - key: stability-level
                operator: NotIn
                values: [alpha, beta]
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            - topologyKey: rack
              labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values: [api-server]
        - weight: 20
          podAffinityTerm:
            - topologyKey: rack
              labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values: [db-server]
    # This field defines the rules specifically for **repulsion**. It set up exactly the
    # same way as the field for attraction shown above, but the criteria here repulses away.
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - topologyKey: data-center
          labelSelector:
            matchExpressions:
              - key: security-context
                operator: NotIn
                values: [privileged-pod]

Kubernetes comes with pre-define topology keys:

• Same node: kubernetes.io/hostname
• Same availability zone: topology.kubernetes.io/zone
• Same geographical region: topology.kubernetes.io/region

Container Isolation

The isolation guarantees of the containers within a pod can be modified. Specifically, a pod can ask that its containers get ...

• access to parts of the node that it's running on (e.g. share node's network interfaces).
• updated container isolation parameters, potentially giving it access to more / less / different features (e.g. change user ID running the main container process).

The following subsections document these mechanisms in further detail.

Security Context

A pod and its containers can have security-related features configured via a security context.

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
    - image: my-image:1.0
      name: my-container
      securityContext:
        # When set, the default user of the container is updated to use be this user ID
        # (note that this is a user ID, not a user name). This is useful if multiple
        # applications are reading / writing to the sane shared volume (no permission
        # problems if they're are read/writing as the same UID?).
        runAsUser: 99 
        # When set, the group of the default user of the container is updated to use be this
        # group ID (there are multiple group IDs here, 123 is the main one used when
        # creating files and directories). This is useful if multiple applications are
        # reading / writing to the sane shared volume (no permission problems if they're are
        # read/writing as the same GID? -- if file permissions allow).
        fsGroup: 123
        supplementalGroups: [456, 789]
        # When set to true, the container will run as a non-root user. This is useful if
        # the pod breaks isolation by exposing the internals of the node to some of its
        # containers.
        runAsNonRoot: true
        # When set to true, the container runs in "privileged mode" (full access to the
        # Linux kernel). This is useful in cases where the pod manages the node somehow
        # (e.g. modified iptables).
        privileged: true
        # An alternative to giving a container "privileged" access (shown above) is to
        # instead provide the container with fine-grained permissions to the kernel.
        # This can also be used to revoke fine-grained permissions that are provided
        # by default (e.g. remove the ability to change ownership of a dir).
        capabilities:
          add:
            - SYS_TIME
          drop:
            - CHOWN
        # When set to true, the container is unable to write to its own filesystem. It
        # can only read from it. Any writing it needs to do has to be done on a mounted
        # volume.
        readOnlyRootFilesystem: true
        #
        # Not all features are listed here. There are many others.

These security-related features can also be used at the pod-level rather than the container-level. When used at the pod level, the features are applied as defaults for all containers (containers can override them if needed).

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  # Pod-level security context. This gets applied as a default for all container in the pod.
  securityContext:
    runAsUser: 99 
    runAsNonRoot: true
    privileged: true
    capabilities:
      add:
        - SYS_TIME
      drop:
        - CHOWN
    readOnlyRootFilesystem: true
  containers:
    - image: my-image1:1.0
      name: my-container1
    - image: my-image2:1.0
      name: my-container2
      securityContext:
        privileged: false  # Override the default "privileged" security context option.

Node Access

A pod's isolation guarantees can be relaxed so that it has access to the internals of the node it's running on. This is important for in certain system-level scenarios, such as pods that collect node performance metrics.

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
    - image: my-image:1.0
      name: my-container
  # By default, all containers within a pod share the same IP and port space (unique to that
  # pod. However, when this property is set to true, the node's default "network namespace"
  # is shared with the containers of the pod (network interfaces are exposed to the pod).
  hostNetwork: true
  # By default, each container within a pod has its own isolated process ID space. However,
  # when this property is set to true, the node's default "process ID namespace" is used for
  # each pod container's processes.
  hostPID: true
  # By default, each container within a pod has its own isolated IPC space. However, when
  # this property is set to true, the node's default "IPC namespace" is used for each pod
  # container's processes.
  hostIPC: true

If the only requirement is that requests from a node's port get forwarded to a container's port, that node's network interfaces don't need to be exposed to the pod. Instead, a container can also simply ask that the node running it directly map a node port to it.

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
    - image: my-image:1.0
      name: my-container
      ports:
        # Map port 9999 on the node that this pod runs to port 8080 on this pod's container.
        #
        # Note that the Kubernetes scheduler ensures that a node has port 9999 available
        # before scheduling this pod to run on it.
        - containerPort: 8080
          hostPort: 9999
          protocol: TCP

API Access

Containers within a pod can access the Kubernetes API server via a service called kubernetes, typically found on the default namespace. Communicating with this service requires a certificate check (to verify the server isn't a man-in-the-middle box) as well as an access token (to authentication with the service). By default, containers have a secret object mounted as a volume at /var/run/secrets/kubernetes.io/serviceaccount that contains both these pieces of data as files:

• ca.crt - certificate used for verifying the server's identity.
• token - bearer token used for authenticating with the server.
• namespace - namespace of the pod itself (not the namespace of the kubernetes service).

In most cases, the credentials provided likely won't provide unfettered access to the Kubernetes API.

Configuration Map

A configuration map is a set of key-value pairs intended to configure the main application of a container (or many containers). By decoupling configurations from the containers themselves, the same configuration map (or parts of it) could be used to configure multiple containers within Kubernetes.

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-config
data:
  param1: another-value
  param2: extra-value
  my-config.ini: |
    # This is a sample config file that I might use to configure an application
    key1 = value1
    ket1 = value2

The key-value pairs of a configuration map typically get exposed to a container either as environment variables, files, or command-line arguments. Keys are limited to certain characters: alphabet, numbers, dashes, underscores, and dots.

Secret

A secret object is a set of key-value pairs, similar to a config map, but oriented towards security rather than just configuration (e.g. for storing things like access tokens, passwords, certificates). As opposed to a config map, Kubernetes takes extra precautions to ensure that a secret object is stored and used securely.

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
type: Opaque  # "Opaque" is the default type (can be omitted)
# Both text and binary data are supported. To insert a text entry, place it under
# "stringData". To insert a binary entry, base64 the value and place it under "data". 
stringData:
  username: admin
  password: pepsi_one
data:
  key_file: eWFiYmFkYWJiYWRvbw==  

Many types of secrets exist. Each type either does some level of verification on the entries and / or acts as a tag to convey what data is contained within (e.g. SSH data, TLS data, etc..). In general Opaque is the secret type used by most applications.

Node

Nodes are the machines that pods run on. A Kubernetes environment often contains multiple nodes, each with a certain amount of resources. Pods get assigned to nodes based on their resource requirements. For example, if a pod A requires 2gb of memory and node C has 24 gigs available, that node may get assigned to run that pod.

Kubernetes typically attempts to schedule multiple instances of the same pod on different nodes, such that a downed node won't take out all instances of the service that pod runs. In the example above, pod instances of the same type are spread out across the 3 nodes.

Kubernetes has a leader-follower architecture, meaning that of the nodes, a small subset is chosen to lead / manage the others. The leaders are referred to as master nodes while the followers are referred to as worker nodes.

A master node can still run pods just like the worker nodes, but some of its resources will be tied up for the purpose of managing worker nodes.

Taints

A taint is a node property that repels pods, either as a preference or as a hard requirement. Each taint defined as a key-value pair along with an effect that defines how it works (value can be null, leaving just a key and effect). An effect can be either ...

• NoSchedule, which means pods won't be scheduled on the node but already running pods will keep running.
• NoExecute, which means pod won't be scheduled on the node and already running pods will be evicted.
• PreferNoSchedule, which means pod can be scheduled on the node but that node will be avoided if possible.

Multiple taints on a node repel pods based on each taint.

A taint is formatted as key=value:effect Node taints can be added and removed via command-line.

kubectl taint node my-staging-node-1 environment-type=production:NoExecute  # Add taint
kubectl taint node my-staging-node-1 environment-type=production:NoExecute- # Remove taint (note the - at the end)

Volume

Volumes are disks where data can be persisted across container restarts. Normally, Kubernetes resets a container's filesystem each time that container restarts (e.g. after a crash or a pod getting moved to a different node). While that works for some types of applications, other application types such as database servers need to retain state across restarts.

Volumes in Kubernetes are broken down into "persistent volumes" and "persistent volume claims". A ...

• persistent volume is the volume itself.
• persistent volume claim is the assignment of a volume.

The idea is that a persistent volume itself is just a floating block of disk space. Only when it's claimed does it have an assignment. Pods can then latch on to those assignments.

In the example above, there are 4 volumes in total but only 3 of those volumes are claimed. podA latches on to claim1 and claim2 while podB latches on to claim3 and claim2 (both pods can access the volume claimed in claim2).

Example persistent volume manifest:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: test-vol
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
    - ReadOnlyMany
  persistentVolumeReclaimPolicy: Recycle  # once a claim on this volume is given up, delete the files on disk
  awsElasticBlockStore:
    volumeID: volume-id
    fsType: ext4

Example persistent volume claim manifest:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-vol-claim
spec:
  resources:
    requests:
      storage: 1Gi  # volume must have at least this much space
  accessModes:
    - ReadWriteOnce  # volume must have this access mode
  storageClassName: ""  # MUST BE EMPTY STRING to claim test-vol described above (if set, uses dynamic provisioning)

There are two types of volume provisioning available:

• static provisioning - a claim is assigned a pre-created persistent volume.
• dynamic provisioning - a claim triggers a new persistent volume to get created and is assigned to it.

Dynamic provisioning only requires that you make a persistent volume claim with a specific storage class name. The administrator is responsible for ensuring a provisioner exists for that storage class and that provisioner automatically creates a volume of that type when a claim comes in. Each storage class can have different characteristics such as volume type (e.g. HDD vs SSD), volume read/write speeds, backup policies, etc.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2

Capacity

Each persistent volume has a storage capacity.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: test-vol
spec:
  capacity:
    storage: 10Gi  # Capacity of the persistent volume.

A persistent volume claim can then be set to a capacity within some range.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-vol-claim
spec:
  resources:
    requests:
      storage: 1Gi  # Minimum capacity that the persistent volume should have.
    limits:
      storage: 5Gi  # Maximum capacity that the persistent volume should have.

Access Modes

A persistent volume can support multiple access modes:

• ReadWriteOnce - volume is mountable by a single node in read-write mode.
• ReadWriteOncePod - volume is mountable by a single pod in read-write mode.
• ReadWriteMany - volume is mountable by many nodes in read-write mode.
• ReadOnlyMany - volume is mountable by many nodes in read-only mode.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: test-vol
spec:
  # Access modes for the persistent volume listed here.
  accessModes:
    - ReadWriteOnce
    - ReadOnlyMany
  capacity:
    storage: 10Gi

A persistent volume claim can then be set to target one or more access modes.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-vol-claim
spec:
  # Persistent volume selected must have these access modes
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

Reclaim Policy

A persistent volume claim, once released, may or may not make the persistent volume claimable again depending on the volume reclaim policy. The options available are ...

• Retain - keep all existing data on the persistent volume and prevent a new persistent volume claim from claiming it again.
• Recycle - delete all existing data on the persistent volume and allow a new persistent volume claim to claim it again.
• Delete - delete the persistent volume object itself.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-vol-claim
spec:
  persistentVolumeReclaimPolicy: Recycle  # Recycle the persistent volume once released
  resources:
    requests:
      storage: 1Gi

If the data on disk is critical to operations, the option to choose will likely be Retain.

Types

A persistent volume needs to come from somewhere, either via a cloud provider or using some internally networked (or even local) disks. There are many volume types: AWS elastic block storage, Azure file, Azure Disk, GCE persistent disk, etc.. Each type has its own set of restrictions such as what access modes it supports or the types of nodes it can be mounted.

The configuration for each type is unique. The following are sample configurations for popular types...

# Amazon Elastic Block Storage
apiVersion: v1
kind: PersistentVolume
metadata:
  name: test-vol
spec:
  awsElasticBlockStore:
    volumeID: volume-id  # a volume with this ID must already exist in AWS
    fsType: ext4

# Google Compute Engine Persistent Disk
apiVersion: v1
kind: PersistentVolume
metadata:
  name: test-vol
spec:
  gcePersistentDisk:
    pdName: test-vol  # a disk with this name must already exist in GCE
    fsType: ext4

# Azure Disk
apiVersion: v1
kind: PersistentVolume
metadata:
  name: test-vol
spec:
  azureDisk:
    # a volume with this name and URI must already exist in Azure
    diskName: test.vhd
    diskURI: https://someaccount.blob.microsoft.net/vhds/test.vhd

# Host path
#   -- this is a path on the node that the pod gets scheduled on, useful
#      for debugging purposes.
apiVersion: v1
kind: PersistentVolume
metadata:
  name: test-vol
spec:
  hostPath:
    path: /data

Storage Classes

Defining a storage class allows for dynamic provisioning of persistent volumes per persistent volume claim.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard
# The two fields below ("provisioner" and "parameters" define how persistent volumes are to
# be created and are unique to each volume type. In this example, the storage class
# provisions new persistent volumes on AWS. Any persistent volume claim with storage class
# name set to `standard` will call out to this AWS elastic store provisioner to create a
# persistent volume of type "awsElasticBlockStore" which gets assigned to it.
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
# This field ("reclaimPolicy") maps to a persistent volume's
# "persistentVolumeReclaimPolicy", except that "Recycle" isn't one of the allowed options:
# Only "Delete" and "Retain" are allowed. This example uses "Retain". If unset, the reclaim
# policy of a dynamically provisioned persistent volume is "Delete". 
reclaimPolicy: Retain
# When this field ("allowVolumeExpansion") is set to true, the persistent volume can be
# resized by editing the persistent volume claim object. Only some volume types support
# volume expansion. This example will work because AWS elastic block store volume types do
# support volume expansion.
allowVolumeExpansion: true

To use a storage class in a persistent volume claim, supply the name of that storage class.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-vol-claim
spec:
  storageClassName: standard  # Use the storage class defined above for this claim
  resources:
    requests:
      storage: 1Gi
  accessModes:
    - ReadWriteOnce

If a persistent volume claim provides no storage class name, that persistent volume claim will use whatever storage class Kubernetes has set as its default. Recall that leaving the storage class name unset is not the same as leaving it as an empty string. To leave unset means to keep it out of the declaration entirely. If the storage class name is ...

• set to an empty string, it tells Kubernetes to find any existing persistent volume for the persistent volume claim.
• set to a non-empty string, it tells Kubernetes to use that storage class to dynamically provision a persistent volume for the persistent volume claim.
• unset, it tells Kubernetes to use the default storage class to dynamically provision a persistent volume for the persistent volume claim.

Most Kubernetes installations have a default storage class available, identified by the storage class having the annotation storageclass.kubernetes.io/is-default-class=true.

# kubectl get sc
# Note how the name identifies it as the default.
NAME                          PROVISIONER            RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
microk8s-hostpath (default)   microk8s.io/hostpath   Delete          WaitForFirstConsumer   false                  6s

# kubectl get sc microk8s-hostpath -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"},"name":"microk8s-hostpath"},"provisioner":"microk8s.io/hostpath","volumeBindingMode":"WaitForFirstConsumer"}
    storageclass.kubernetes.io/is-default-class: "true"
  creationTimestamp: "2022-07-22T19:41:28Z"
  name: microk8s-hostpath
  resourceVersion: "2775"
  uid: 1df92cbc-6e2f-4726-a487-a81b1fcd8d2b
provisioner: microk8s.io/hostpath
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

Endpoints

Endpoints (plural) is a kind that simply holds a list of IP addresses and ports. It's used by higher-level kinds to simplify routing. For example, an endpoints object may direct to all the nodes that make up a sharded database server.

Example manifest:

apiVersion: v1
kind: Endpoints
metadata:
  name: database
subsets: 
  - addresses:
      - ip: 10.10.1.1
      - ip: 10.10.1.2
      - ip: 10.10.1.3
    ports:
      - port: 5432 
        protocol: TCP  # TCP or UDP, default: TCP
        name: pg
  - addresses:
      - ip: 10.13.4.101
      - ip: 10.13.4.102
      - ip: 10.13.4.103
    ports:
      - port: 12345
        protocol: TCP  # TCP or UDP, default: TCP
        name: pg2

The endpoints example above points to ...

• 10.10.1.1:5432
• 10.10.1.2:5432
• 10.10.1.3:5432
• 10.13.4.101:12345
• 10.13.4.102:12345
• 10.13.4.103:12345

Service

Services are a discovery and load balancing mechanism. A service exposes a set of pods under a single fixed unified hostname and IP, routing traffic to that set by load balancing incoming requests across the set. Any external application would need to use a service's hostname because the IP / host of the single pod instances aren't fixed, exposed, or known. That is, pods are transient and aren't guaranteed to always reside on the same node. As they shutdown, come up, restart, move between nodes, etc.., there's no implicit mechanism that requestors can use to route their requests accordingly.

A service fixes this my internally tracking such changes and providing a single unified point of access.

Example manifest:

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  selector:
    app: MyApp
  ports:
    - name: webapp-port
      protocol: TCP
      port: 80
      targetPort: 9376

Routing

A service determines which pods it should route traffic to via label selectors.

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  # Label selectors that pick out the pods this service routes to.
  selector:
    key1: value1
    key2: value2
    key3: value3
  ports:
    - name: webapp-port
      protocol: TCP
      port: 80
      targetPort: 9376

Internally, the service creates and manages an endpoints object containing the IP and port for each pod captured by the selector. If no selectors are present, the service expects an endpoints object with the same name to exist, where that endpoints object contains the list of IP and port pairs that the service should route to.

apiVersion: v1
kind: Endpoints
metadata:
  name: database  # Must be same name as the service
subsets: 
  - addresses:
      - ip: 10.10.1.1
      - ip: 10.10.1.2
      - ip: 10.10.1.3
    ports:
      - port: 5432

If no label selectors are present but the service's type is set to ExternalName, the service will route to some user-defined host. This is useful for situations where you want to hide the destination, such as an external API that you also want to mock for development / testing.

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  type: ExternalName
  externalName: api.externalcompany.com  # Route to this host
  ports:
    - name: api-port
      protocol: TCP
      port: 8080
      targetPort: 5000

Ports

A service can listen on multiple ports.

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  # In most cases, each port entry has ...
  #
  #  * "name", which is a friendly name to identify the port (optional)
  #  * "protocol", which is either `TCP` or `UDP` (defaults to `TCP`).
  #  * "port", which is the port that the service listens on.
  #  * "targetPort", which  is the port that requests are forwarded to on the pod (defaults
  #     to value set for "port").
  ports:
    - name: webapp-port
      protocol: TCP
      port: 80
      targetPort: 9376
    - name: api-port
      protocol: TCP
      port: 8080
      targetPort: 1111

The example above forwards requests on two ports. Requests on port ...

• 80 of the service get forwarded to port 9376 of a pod assigned to that service.
• 8080 of the service get forwarded to port 1111 of a pod assigned to that service.

Ports may also reference the names of ports in a pod. For example, the following pod provides names for its ports.

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
    - name: my-container
      image: my-image:1.0
      ports:
        - name: my-http-port  # Name for the port
          containerPort: 8080
          protocol: TCP

In the service targeting that pod, you can use my-http-port as a target port.

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  selector:
    app: MyApp
  ports:
    - name: webapp-port
      protocol: TCP
      port: 80
      targetPort: my-http-port  # The name of the port in the pod

Health

The service periodically probes the status of each pod to determine if it can handle requests or not. Two types of probes are performed:

• liveness probes - When an existing instance of a pod fails a user-defined test that checks if it's still running, the service stops routing traffic to it.
• readiness probes - When a new instance of a pod comes up, the service won't route traffic to it until it passes a user-defined test that says it's ready.

These probes are defined directly in the pod manifest.

Headless

A headless service is one in which there is no load balancer forwarding requests to pods / endpoints. Instead, the domain for the service will resolve a list of ready IPs for the pods (or endpoints) that the service is for.

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  # When this field ("clusterIP") is set to "None", the service is a "headless service".
  clusterIP: None
  selector:
    app: MyApp
  ports:
    - name: webapp-port
      protocol: TCP
      port: 80

Generally, headless services shouldn't be used because DNS queries are typically cached by the operating system. If the IPs that a service forwards to change, apps that have recently queried the service's DNS will continue to use the old (cached) set of IPs until the operating system purges its DNS cache.

Session Affinity

How a service decides to forward incoming requests to the pod instances assigned to it is controlled via a session affinity field. Assigning a value of ...

• None forwards each request to a randomly selected pod instance (default behavior).
• ClientIP forwards each request originating from the same IP to the same pod instance.

When using ClientIP, a maximum session "sticky time" may also be provided.

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  # The two fields below ("sessionAffinity" and "sessionAffinityConfig") define how session
  # affinity works.
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10000  # Defaults value is 108300, which is around 3 hours
  selector:
    app: MyApp
  ports:
    - name: webapp-port
      protocol: TCP
      port: 80

Exposure

The service type defines where and how a service gets exposed. For example, a service may only be accessible within the cluster, to specific parts of the cluster, to an external network, or to the public Internet.

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  type: ClusterIP  # Service type
  selector:
    app: MyApp
  ports:
    - name: webapp-port
      protocol: TCP
      port: 80

If not specified, the type is ClusterIP, meaning that it's exposed only locally within the cluster.

Local

Services of type ClusterIP / ExternalName are only accessible from within the cluster. The hostname of such services are broken down as follows: NAME.NAMESPACE.svc.CLUSTER

• NAME is the name of the service.
• NAMESPACE is the namespace the service is in (defaults to default).
• svc is a constant that identifies the host is for a service.
• CLUSTER is the name of the cluster (defaults to cluster.local.).

Depending on what level you're working in, a hostname may be shortened. For example, if the requestor and the service are within ...

• the same namespace and cluster, hostname NAME is sufficient.
• the same cluster but not the same namespace, hostname NAME.NAMESPACE is sufficient.
• different clusters, the full hostname NAME.NAMESPACE.svc.CLUSTER is required.

The IP for a ClusterIP / ExternalName service is stable as well, just like the hostname.

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  type: ClusterIP  # ClusterIP service type
  selector:
    app: MyApp
  ports:
    - name: webapp-port
      protocol: TCP
      port: 80

Node Port

Services of type NodePort are accessible from outside the cluster. Every worker node opens a port (either user-defined or assigned by the system) that routes requests to the service. Since nodes are transient, there is no single point of access to the service.

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  type: NodePort  # Nodeport service type
  selector:
    app: MyApp
  ports:
    # When "NodePor"` is used as the type, each port should have "nodePort" field that
    # defines the port on the worker nodes to open.
    - protocol: TCP
      port: 80
      targetPort: 9376
      nodePort: 8080

Load Balancer

Services of type LoadBalancer are accessible from outside the cluster. When the LoadBalancer type is used, the cloud provider running the cluster assigns their version of a load balancer to route external HTTP requests to the Kubernetes ingress component. Ingress then determines what service that request should be routed to based on details within the HTTP parameters (e.g. Host).

There is no built-in Kubernetes implementation of ingress. Kubernetes provides the interface but someone must provide the implementation, called an ingress controller, for the functionality to be there. This is because load balancers come in multiple forms: software load balancers, cloud provider load balancers, and hardware load balancers. When used directly, each has a unique way it needs to be configured, but the ingress implementation abstracts that out.

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  type: LoadBalancer
  ports:
    - protocol: TCP
      port: 80
      targetPort: 9376
      nodePort: 8080
    ...

Once provisioned, the object will have the field status.loadBalancer.ingress.ip added to it, which states the IP of the load balancer forwarding requests to this service.

# <REMOVED PREAMBLE>
spec:
  type: LoadBalancer
  ports:
    - protocol: TCP
      port: 80
      targetPort: 9376
      nodePort: 8080
status:
  loadBalancer:
    ingress:
      ip: 192.0.5.6

Ingress

Similar to a service of type LoadBalancer, An ingress object is a load balancer with a publicly exposed IP. However, rather than load balancing at the TCP/UDP level, an ingress object acts as a load balancing HTTP proxy server. An HTTP request coming into an ingress object gets routed to one of many existing services based on host and path HTTP headers. This is useful because the cluster can expose several services under a single public IP address.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
spec:
  rules:
  - host: stats.myhost.com
    http:
      paths:
      - path: /graphana
        pathType: Prefix
        backend:
          service:
            name: graphana-service
            port:
              number: 80
  - host: api.myhost.com
    http:
      paths:
      - path: /v2
        pathType: Prefix
        backend:
          service:
            name: api-service-v2
            port:
              number: 80
      - path: /v1
        pathType: Prefix
        backend:
          service:
            name: api-service-v1
            port:
              number: 80
      - path: /
        pathType: Prefix
        backend:
          service:
            name: api-service-v2
            port:
              number: 80

Hosts

The host in each rule can be either an exact host or it could contain wildcards (e.g. *.api.myhost.com). Each name in the host (split by dot) intended for a wildcard should explicitly have an asterisk in its place. The portion the asterisk is in must exist and it only covers that name. For example, the rule below will match ONE.api.myhost.com, but not TWO.THREE.api.myhost.com or api.myhost.com.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
spec:
  rules:
  - host: "*.api.myhost.com"
    - path: /v2
      pathType: Prefix
      backend:
        service:
          name: api-service-v2
          port:
            number: 80

Path Type

Each rule entry should have a path type associated with it. It can be set to any of the following values:

• Exact - Matches the URL path exactly (case-sensitive).
• Prefix - Matches the URL path prefix (case-sensitive).
• ImplementationSpecific - Based on the class of the ingress object.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
spec:
  rules:
  - host: api.myhost.com
    http:
      paths:
      - path: /my/prefix/path
        pathType: Prefix
        backend:
          service:
            name: api-service-v2
            port:
              number: 80

The most common path type is Prefix. A type of Prefix splits the path using / and matches the rule if the incoming request's path starts with the same path elements as the rule's path. Trailing slashes are ignored (e.g. /p1/p2/p3/ and /p1/p2/p3 are equivalent).

TLS Traffic

Assuming you have a TLS certificate and key files for the host configured on the ingress object, you can add those into Kubernetes as a secret and configure the ingress object to make use of it.

# openssl genrsa -out tls.key 2048
# openssl req -new -x509 -key tls.key -out tls.cert -days 360 -subj /CN=api.myhost.com
# kubectl create secret tls my-api-tls --cert=tls.crt --key=tls.key
apiVersion: v1
kind: Secret
metadata:
  name: my-api-tls
type: kubernetes.io/tls
data:
  tls.crt: base64 encoded cert
  tls.key: base64 encoded key

Each certificate secret used by an ingress object has its own entry that contains the hosts it supports.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
spec:
  # TLS certificates are listed here. Each entry has a certificate secret name under
  # "secretName" and the domain(s) supported by that certificate under "hosts". Hosts must
  # match hosts explicitly listed in the ingress object's rules.
  tls:
    - secretName: my-api-tls
      hosts:                
        - api.myhost.com
    - secretName: my-stats-tls
      hosts:
        - stats.myhost.com
      

Once an encrypted request comes in to the ingress controller, it's decrypted. That decrypted request then gets forwarded to the service it was intended for.

Namespace

A namespace is a kind used to avoid naming conflicts. For example, it's typical for a Kubernetes cluster to be split up into development, testing, and production namespaces. Each namespace can have objects with the same names as those in the other two namespaces.

apiVersion: v1
kind: Namespace
metadata:
  name: production

Namespaces are cluster-level objects. This is contrary to most other kinds in Kubernetes, which are namespace-level objects, meaning that a namespace can be used to disambiguate objects of that type with the same name...

# These are namespace-level objects
apiVersion: v1
kind: Pod
metadata:
  name: mypod
  namespace: testing  # put into the testing namespace
spec:
  containers:
  - name: mypod
    image: my_image:v2_alpha5
---
apiVersion: v1
kind: Pod
metadata:
  name: mypod
  namespace: production  # put into the production namespace
spec:
  containers:
  - name: mypod
    image: my_image:v1

If a namespace-level object doesn't set a namespace, the namespace defaults to default.

Replica Set

A replica set is an abstraction that's used to ensure a certain number of copies of some pod are always up and running. Typical scenarios where replica sets are used include ...

• sharding (e.g. workers that pull job out of a queue for processing).
• scale (e.g. microservices that scale horizontally).
• redundancy (e.g. leader-follower architectures such as Redis-style replica servers).

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: my-replicaset
spec:
  replicas: 2  # Number of pod copies to run.
  # Selectors are label selectors used to identify pods, which match the key-value pairs
  # used for pod template labels further down.
  selector:
    matchLabels:
      app: my-app
  # A template of the pod to launch when there aren't enough copies currently running.
  # Everything under "template" is essentially a pod manifest, except the "apiVersion" and
  # "kind" aren't included.
  template:
    metadata:
      name: my-pod
      # These labels are how this replicate set will determine how many copies are running. It
      # will look around for pods with this set of labels.
      labels:
        app: my-app
    spec:
      containers:
      - name: my-container
        image: nginx

Recall that, to link objects together, Kubernetes uses loosely coupled linkages via labels rather than hierarchical parent-child relationships. As such, the pod template should have a unique set of labels assigned that the replica set can look for to determine how many instances are running. Regardless of how those instances were launched (via the replica set or something else), the replica set will account for them. In the example above, the replica set determines pod instances it's responsible for by looking for the label named app and ensuring its set to my-app.

A replica set's job is to ensure that a certain number of copies of a pod template are running. It won't retain state between its copies or do any advanced orchestration. Specifically, a replica set ...

• won't retain pod IPs / hostnames across time. Each launched pod will have its own unique IP / hostname, even if it's replacing a downed pod (it won't inherit that downed pod's IP / hostname).
• will force all pods to use a single persistent volume claim (if one was specified in the pod template), meaning all pods will use a single volume. For horizontally scalable applications (e.g. microservices, databases, etc..), each running instance of an application typically needs its own persistent storage.

Deployment

Deployment are similar to replica sets but make it easy to do rolling updates, where the update happens while the application remains online and still services requests. Old pods are transitioned to new pods as a stream instead of all at once, ensuring that the application is responsive throughout the upgrade process. Likewise, they allow for rolling back should an update encounter any problems.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
# The manifest for a deployment builds off the manifest for a replica set. Most replica set
# fields are present: Number of copies, label selectors, pod template, etc... In addition,
# it supports several other fields specific to doing updates.
spec: 
  replicas: 2
  selector:
    matchLabels:
      app: my-app
  template:
      labels:
        app: my-app
    spec:
      containers:
        - name: my_container
          image: my-image:1.0
  strategy: RollingUpdate  # How the deployment should perform updates (default value)

A deployment provides mechanisms to control how an update happens (e.g. all at once vs gradual), if an update is deemed successful (e.g. maximum amount of time a rollout can take), fail-fast for bad updates, and rollbacks to revert to previous versions. These features are discussed in the subsections below.

Updates

A deployment can support one of the two update strategies:

• RollingUpdate - updates pods piecemeal (default).
• Recreate - updates by first bringing down all old pods then bringing up all the new pods.

Recreate is simple but results in down (a period of time where no pods are running). Most enterprise applications have up-time guarantees and as such require RollingUpdate. RollingUpdate has several options that control the flow and timing of how pods go down and come up, documented in the example below.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
spec:
  replicas: 10
  selector:
    matchLabels:
      app: my-app
  template:
      labels:
        app: my-app
    spec:
      containers:
        - name: my_container
          image: my-image:1.0
  strategy:
    type: RollingUpdate
    rollingUpdate:
      # "maxUnavailable" - During an update, this is the number (or percentage) of pods
      # that can be unavailable relative to the number of replicas. Since this deployment
      # has 10 replicas, the parameter below is instructing that the number of replicas
      # can't go below 8 during an update (at most 2 pods may be unavailable).
      #
      # If between 0 and 1, this is treated as a percentage of pods (e.g. 0.25 means 25
      # percent of pods may be unavailable during an update). 
      maxUnavailable: 2
      # "maxSurge" - During an update, this is the number (or percentage) of excess pods
      # that can be available relative to the number of replicas. Since this deployment
      # has 10 replicas, the parameter below is instructing that the number of replicas
      # can't go above 12 during an update (at most 2 pods extra pods may be running).
      #
      # If between 0 and 1, this is treated as a percentage of pods (e.g. 0.25 means 25
      # percent of pods may be unavailable during an update). 
      maxSurge: 2
      # "minReadySeconds" - Once all of the readiness probes of a new pod succeed, this
      # is the number of seconds to wait before the deployment deems that the pod has
      # been successfully brought up. If any readiness probes within the pod fail during
      # this wait, the update is blocked.
      #
      # This is useful to prevent scenarios where pods initially report as ready but
      # revert to un-ready soon after receiving traffic.
      minReadySeconds: 10
      # "progressDeadlineSeconds" - This is the maximum number of seconds that is allowed
      # before progress is made. If this is exceeded, the deployment is considered stalled.
      #
      # Default value is 600 (10 minutes).
      progressDeadlineSeconds: 300

It's common for deployments to fail or get stuck for several reasons:

• Insufficient node resources.
• Problems pulling images.
• Probe failures (startup probes / readiness probes).
• etc..

Rollbacks

A deployment retains update history in case it needs to rollback. The mechanism used for this is replica sets: For each update, a deployment launches a new replica set. The replica set for the old version ramps down the number of pods while the replica set for the new version ramps up the number of pods. Once all pods have been transitioned to the new version, the old replica set (now empty of pods) is kept online.

It's good practice to limit the number of revisions kept in a deployments update history because it limits the number of replica sets kept alive.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
spec:
  revisionHistoryLimit: 5  # Keep the 5 latest revisions
  replicas: 2
  selector:
    matchLabels:
      app: my-app
  template:
      labels:
        app: my-app
    spec:
      containers:
        - name: my_container
          image: my-image:1.0

Stateful Set

A stateful set is similar to a deployment but the pods it creates are guaranteed to have a stable identity and each pod can have its own dedicated storage volumes. In the context of stateful sets, ...

• stable identity means that if a pod goes down, the stateful set responsible for it will replace it with a new pod has the exact same identification information (same name, IP, etc..). Contrast that to deployments, where replacement pods have entirely new identities.
• dedicated storage volume means that each stable identity can have its own unique persistent volume claims. Contrast that to deployments, where persistent volume claims are shared across all pods.

A stateful set requires three separate pieces of information:

1. a headless service, which acts as a gateway to a stateful set's pods (referred to as a governing service).
2. a volume claim template, which templates persistent volume claims for a stateful set's pods.
3. a pod template, which templates pods similar to pod template for a deployment.

These three pieces are represented as two separate objects: the governing service and the stateful set itself.

# Manifest #1: Headless service for the stateful set's pods (governing service).
apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  clusterIP: None
  # Routes traffic to pods based on the following label selectors, which are the same
  # key-value pairs used for pod template labels of the stateful set further down.
  selector:
    app: my-app
  ports:
    - name: http
      port: 80
----
# Manifest #2: The stateful set itself, which contains both the pod template and the
# volume claim template.
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: my-stateful-set
spec:
  # Selectors are label selectors used to identify pods, which match the key-value pairs
  # used for pod template labels further down.
  selector:
    matchLabels:
      app: my-app
  serviceName: my-service  # Name of headless service for stateful set (governing service).
  replicas: 3              # Number of replicas for the stateful set.
  # Persistent volume claims will be created based on the following template.
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        resources:
          requests:
             storage: 1Mi
        accessModes:
          - ReadWriteOnce
  # Pod's will be created based on the following template. Note that the volume mount
  # references the persistent volume claim template described above.
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: my-container
          image: my-image:1.0
          ports:
            - name: http
              containerPort: 8080
          volumeMounts:
            - name: data
              mountPath: /var/data
  # Similar to deployments, stateful sets also support updating / rollback mechanisms exist,
  # but not exactly the same ones.
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
   # Once all of the readiness probes of a new pod succeed, this is the number of seconds to
   # wait before the stateful set deems the pod to be available. No readiness probes within
   # the pod can fail during this wait.
  minReadySeconds: 10

The example above creates a stateful set that manages three pod replicas and a governing service for those pods. The pods created by the stateful set are numbered starting from 0: my-stateful-set-0, my-stateful-set-1, and my-stateful-set-2. In addition, each pod gets its own persistent volume claim mounted at /data containing a modest amount of storage space. That persistent volume claim will have the format data-my-stateful-set-N (where N is the ordinal suffix)

The ordinal suffixes of a stateful set's pods are part of their stable identity. If a pod were to die, the volume for that stable identity will be re-bound to its replacement. Stateful sets take great care to ensure that no more than one pod will ever be running with the same stable identity so as to prevent race conditions (e.g. conflicts regarding IP / host, multiple pods using the same volume, etc..). In many cases, that means a pod won't be replaced until the stateful set is absolutely sure that it has died.

Scaling

Because of stable identity guarantees and the fact that each stable identity can have its own distinct volumes, stateful sets have different scaling behavior than deployments. A stateful set scales pods based on the ordinal suffix of its pod names. When the number of replicas is ...

• decreased, the stateful set brings down the pods with the highest ordinal suffixes first (decrementing).
• increased, the stateful set brings up new pods with the next unused ordinal suffixes (incrementing).

For example, given the stateful set my-stateful-set with 3 replicas (those replicas being pods my-stateful-set-0, my-stateful-set-1, and my-stateful-set-2), ...

• decrementing the replica count to 1 is guaranteed to shut down pods my-stateful-set-2 and my-stateful-set-1 (in that order).
• incrementing the replica count to 5 is guaranteed to spin up pods my-stateful-set-4 and my-stateful-set-5 (in that order).

The scaling behavior makes the stable identities of the pods being removed / added known beforehand. In contrast, a deployment's scaling behavior makes no guarantees as to which replicas get removed / added and in what order.

Stateful sets scale one pod at a time to avoid race conditions that are sometimes present in distributed applications (e.g. which database server is the primary vs which database server is the replica). When scaling down, the persistent volumes for a pod won't be removed along with the pod. This is to avoid permanently deleting data in the event of an accidental scale down. Likewise, when scaling up, if a volume for that stable identity is already present, that volume gets attached instead of creating a new volume.

For example, given the same 3 replica my-stateful-set example above, changing the number of replicas to 1 will leave the volumes for my-stateful-set-1 and my-stateful-set-2 lingering undeleted.

Changing the number of replicas back to 3 will then recreate my-stateful-set-1 and my-stateful-set-2, but those new pods will be assigned the lingering undeleted volumes from before rather than being assigned new volumes (all previous data will be present).

A stateful set will not proceed with scaling until all preceding pods (ordinal suffix) are in a healthy running state. This is because, if a pod is unhealthy and the stateful set gets scaled down, it's effectively lost two members at once. This goes against the "only one pod can go down at a time" stateful set scaling behavior.

For example, given the same 3 replica my-stateful-set example above, scaling down to 1 replica will first shut down pod my-stateful-set-2 and then pod my-stateful-set-1. If my-stateful-set-2 shuts down but then my-stateful-set-0 enters into an unhealthy state, my-stateful-set-1 won't shut down until my-stateful-set-0 recovers. Likewise, if my-stateful-set-0 enters into an unknown state (e.g. the node running it temporarily lost communication with the control plane), my-stateful-set-1 won't shut down until my-stateful-set-0 is known and healthy.

Updates

There are two templates in a stable set: a pod template and a volume claim template.

A pod template has two different update strategies:

• RollingUpdate - updates pods piecemeal (default).
• OnDelete - user must manually bring down each pod and the stateful set will replace it with updated version.

OnDelete is simple but requires user intervention to shutdown pods. RollingUpdate is similar to the RollingUpdate strategy for deployments, but it supports fewer parameters and its behavior is slightly different. Specifically, rolling updates for stateful sets support two parameters.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: my-ss
spec:
  selector:
    matchLabels:
      app: my-app
  serviceName: my-service
  replicas: 10
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: my-container
          image: my-image:1.0
  # Rolling update strategy.
  strategy:
    type: RollingUpdate
    rollingUpdate:
      # "maxUnavailable" - During an update, this is the number (or percentage) of pods
      # that can be unavailable relative to the number of replicas. Since this deployment
      # has 10 replicas, the parameter below is instructing that the number of replicas
      # can't go below 8 during an update (at most 2 pods may be unavailable).
      #
      # If between 0 and 1, this is treated as a percentage of pods (e.g. 0.25 means 25
      # percent of pods may be unavailable during an update). 
      maxUnavailable: 1
      # "partition" - Only pods with suffix ordinals that are >= to this number will
      # receive updates. All other pods will remain un-updated. For this stateful set,
      # that means only "my-ss-5", "my-ss-6", "my-ss-7", "my-ss-8", "my-ss-9", and
      # "my-ss-10" get updated. 
      #
      # This is a useful feature for gradual / phased roll outs.
      partition: 5

Rolling updates performed with a pod management policy of OrderedReady (the default) may get into a broken state which requires manual intervention to roll back. If an update results in a pod entering into an unhealthy state, the rolling update will pause. Reverting the pod template won't work because it goes against the "only one pod can go down at a time" behavior of stateful sets.

A volume claim template cannot be updated. The system will reject an updated stateful set if its volume claim template differs from the original. As such, users have devised various manual strategies for modifying volumes in a stateful set:

• Expanding the volumes for a stateful set's pods is documented here. The idea is to manually edit each persistent volume claim then re-create the stateful set without deleting any of its pods.

  kubectl edit pvc <name> # for each PVC in the StatefulSet, to increase its capacity.
  kubectl delete sts --cascade=orphan <name> # to delete the StatefulSet and leave its pods.
  kubectl apply -f <name> # to recreate the StatefulSet.
  kubectl rollout restart sts <name> # to restart the pods, one at a time. During restart, the pod's PVC will be resized.
  

  ⚠️NOTE️️️⚠️

  For this to work, I think the volume type / storage class needs to support expanding volumes (allowVolumeExpansion is true)?

Peer Discovery

A stateful set's governing service allows for its pods to discover each other (peer discovery). For each pod in a stateful set, that pod will have a sub-domain within that stateful set's governing service. For example, a stateful set with the name my-ss in the namespace apple within the cluster will expose its pods as ...

• my-ss-0.my-ss.apple.svc.cluster.local
• my-ss-1.my-ss.apple.svc.cluster.local
• my-ss-2.my-ss.apple.svc.cluster.local
• ...

... , where each sub-domain points to one of the stable identities.

A governing service allows for enumerating all pods within its stateful set via DNS service records (SRV). In the example above, performing an SRV lookup on my-ss.apple.svc.cluster.local will list out the sub-domains and IPs for all my-ss pods.

Job

A job launches one or more pods to perform one-off tasks. Once those one-off tasks complete, the job is effectively over. Typical job use-cases include ...

• database migration
• database compaction
• log file removal

apiVersion: batch/v1
kind: Job
metadata:
  name: my-job
spec:
  parallelism: 5  # Num of pods that can run at the same time (default is 1).
  completions: 10 # Num of pods that must successfully finish for job to end (default is 1).
  backoffLimit: 4 # Max num of retries of a failed pod before failing job (default is 6).
  activeDeadlineSeconds: 99 # Max secs before job forcibly fails, killing all pods.
  # Completion mode, when set to "Indexed", provides an ordinal suffix / stable identity to
  # each launched pod, similar to how a stateful set provides its pods with a stable identity.
  # This is useful in cases where the pods of a job need to communicate with each other (e.g.
  # distributed work-queue processing), but a service will likely also need to be provided.
  #
  # In most cases, this should be set to "NonIndexed" (default value).
  completionMode: NonIndexed
  # Pod template describing job's pods.
  template:
    spec:
      containers:
        - name: my-container
          image: my-image:1.0
      # Restart policy of the launched pods. For jobs, this must be set to either
      # "OnFailure" or "Never".
      restartPolicy: OnFailure

The example job 10 pods to successful completion, keeping up to 5 concurrently running at any one time. If a pod fails, the job will retry it up to 4 times before failing the job entirely. Similarly, the job itself runs no more than 99 seconds before failing entirely.

Common gotchas with jobs:

• activeDeadlineSeconds confusion: There's an activeDeadlineSeconds that can go in the pod template as well which is different from the job's activeDeadlineSeconds. Don't confuse the two.

  🔍SEE ALSO🔍

  • Kinds/Pod/Lifecycle/Maximum Runtime

• Lingering finished jobs: By default, neither a job nor its pods are cleaned up after the job ends (regardless of success or failure). This can end up cluttering the Kubernetes servers.

  🔍SEE ALSO🔍

  • Kinds/Job/Cleanup (Job cleanup strategies)

• No communication between pods: By default, a job will automatically pick pod labels and set its label selectors so as not to conflict with other pods in the system. Without consistent labels, the pods of a job can't communicate with each other.

  🔍SEE ALSO🔍

  • Kinds/Job/User-defined Labels (Job cleanup strategies)

• Unexpected concurrency: Even if concurrency and completions fields are both set to 1, there are cases where a job may launch more than once. As such, a job's pods should be tolerant of concurrency.

Cleanup

One common problem with jobs is resource cleanup. Except for failed pods that have been retried (backoffLimit field), a completed job won't delete its pods by default. Those pods are kept around in a non-running state so that their logs can be examined if needed. Likewise, the job itself isn't deleted on completion either.

The problem with letting jobs and pods linger around in the system is that it causes clutter, putting pressure on the Kubernetes servers. Typically, it's up to the user to delete a job (deleting a job will also deletes any lingering pods). However, there are other mechanisms that can automate the deletion of jobs:

• There's a higher-level kind called cron job that has cleanup policies for ended jobs.

  🔍SEE ALSO🔍

  • Kinds/Cron Job ("Job history limit" fields)

• There's a time-to-live mechanism that can be used to automatically deletes jobs that have ended (regardless of success or failure) after some duration.

  apiVersion: batch/v1
  kind: Job
  metadata:
    name: my-job
  spec:
    # The number of seconds have the job has ended before the job is eligible to be
    # deleted. If this is 0, the job will be deleted immediately after ending.
    ttlSecondsAfterFinished: 100
    template:
      spec:
        containers:
          - name: my-container
            image: my-image:1.0
    restartPolicy: Never
  

User-defined Labels

By default, a job automatically picks out a unique label to identify its pods (such that it definitively knows which pods in the system belong to it). However, it's possible to give custom labels to a job's pods / give custom label selectors to a job. This is useful is cases where a job's pods need to communicate with each other: A headless service can target a job's pods based on its labels, which is similar to how a stateful set's pods can communicate with and discover each other (governing service).

apiVersion: batch/v1
kind: Job
metadata:
  name: my-job
spec:
  parallelism: 5  # Num of pods that can run at the same time (default is 1).
  completions: 10 # Num of pods that must successfully finish for job to end (default is 1).
  backoffLimit: 4 # Max num of retries of a failed pod before failing job (default is 6).
  activeDeadlineSeconds: 99 # Max secs before job forcibly fails, killing all pods.
  # Completion mode, when set to "Indexed", provides an ordinal suffix / stable identity to
  # each launched pod, similar to how a stateful set provides its pods with a stable identity.
  # This is useful in cases where the pods of a job need to communicate with each other (e.g.
  # distributed work-queue processing), but a service will likely also need to be provided.
  #
  # In most cases, this should be set to "NonIndexed" (default value).
  completionMode: NonIndexed
  # Selectors are label selectors used to identify pods, which match the key-value pairs
  # used for pod template labels further down.
  #
  # In most cases, you shouldn't need to specify this (or the labels in the pod template
  # below). When not present, the system will automatically pick labels / label selectors
  # that won't conflict with other jobs / pods.
  selector:
    matchLabels:
      app: my-app
  # Pod template describing job's pods.
  template:
    spec:
      containers:
        - name: my-container
          image: my-image:1.0
      # Restart policy of the launched pods. For jobs, this must be set to either
      # "OnFailure" or "Never".
      restartPolicy: OnFailure

Cron Job

A cron job launches a job periodically on a schedule, defined in cron format.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: my-cronjob
spec:
  schedule: "0 * * * *"  # Schedule of the job in cron format (launches every hour)
  # How much tolerance to have (in seconds) for a scheduled run of a job that's been missed.
  # If a scheduled run gets missed for any reason but is identified within this window,
  # it'll run anyways. If it's past the window, it'll count as a failed job.
  #
  # This is an optional field. If not set, there is no deadline (infinite tolerance).
  startingDeadlineSeconds: 200
  # How should a job launch be treated if the previously launched job is still running.
  # If this is set to ...
  #  * "Allow", it allows the jobs to run concurrently (default value).
  #  * "Forbid", it skips the new job launch, meaning concurrently running jobs not allowed.
  #  * "Replace", it replaces the previously running job with the new job.
  concurrencyPolicy: Forbid
  # How many ended successful/failed jobs should remain in Kubernetes. If set to 0, a job and
  # its corresponding pods are removed immediately after ending. If  > 0, the last N jobs and
  # their pods will remain in Kubernetes (useful for inspection of logs).
  successfulJobsHistoryLimit: 0
  failedJobsHistoryLimit: 0
  # Job template that describes the job that a cron job launches. This is effectively a job
  # definition without "apiVersion", "kind", and "metadata" fields.
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: my-container
              image: my-image:1.0
          restartPolicy: OnFailure

Common gotchas with cron jobs:

• Unexpected concurrency: There are cases where a cron job may launch more than once. As such, a job's pods should be tolerant of concurrency.

• Unexpected misses: There are cases where a cron job may not launch even though it's supposed to.

Daemon Set

A daemon set ensures that a set of nodes each have a copy of some pod always up and running. Typical scenarios where a daemon set is used include ...

• node log collection (e.g. logstash agent).
• node monitoring (e.g. zabbix agent).

The above scenarios are ones which break container / pod isolation. That is, a daemon set is intended to run pods that are coupled to nodes and sometimes those pods will do things such as mount the node's root filesystem and run commands to either install software or gather information.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: my-ds
spec:
  # Selectors are label selectors used to identify pods, which match the key-value pairs
  # used for pod template labels further down.
  selector:
    matchLabels:
      app: my-app
  # Pod template describing daemon set's pods.
  template:
    metadata:
      name: my-pod
      # These labels are how this daemon set will determine if the pod is running on a node.
      # It will look around for pods with this set of labels.
      labels:
        app: my-app
    spec:
      # Put copies of this pod only on nodes that have these labels. There is also a
      # "nodeAffinity" field and "tolerations" field, which allow for more elaborate logic
      # / soft logic for node selection (too vast to cover here).
      #
      # If neither "nodeSelector" nor "nodeAffinity" is set, copies of this pod will run on
      # all nodes.
      nodeSelector:
        type: my-node-type
      containers:
        - name: my-container
          image: my-image:1.0
          resources:
            limits:
              cpu: 100m
              memory: 200Mi
          volumeMounts:
            - name: varlog
              mountPath: /host_log
              readOnly: true
      volumes:
        - name: varlog
          hostPath:
            path: /var/log

The example above runs a copy of the pod template on each node that has the label type=my-node-type and mounts the host's /var/log directory to /host_log in the container. Most daemon sets are used for some form of monitoring or manipulation of nodes, so it's common to have volumes of the type hostPath, which mounts a directory that's directly on the node itself.

Unlike deployments and stateful sets, daemon sets don't have support for rolling updates. On any change to a daemon set's pod template on node selection criteria, old pods are deleted and updated pods are brought up in their place.

Service Account

A service account is a set of credentials that applications within a pod use to communicate with the Kubernetes API server. Service accounts also provide an aggregation point for image pull secrets and other security-related features.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-service-account
# A list of image pull secrets to use with private container registries. When this service
# account is applied to a pod, all image pull secrets in the service account get added to
# the pod.
imagePullSecrets:
  - name: my-dockerhub-secret
  - name: my-aws-ecr-secret
# API access credentials will never be mounted to any pod that uses this service account.
automountServiceAccountToken: false

By default, each namespace comes with its own service account which pods in that namespace automatically use. The service account used by a pod can be overridden to another service account.

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  # Custom service account to use. This can't be changed once the pod's been creatd.
  serviceAccountName: my-service-account
  containers:
    - name: my-container
      image: my-registry.example/tiger/my-image:1.0

Horizontal Pod Autoscaler

A horizontal pod autoscaler (HPA) periodically measures how much work a set of pod replicas are doing so that it can appropriately adjust the number of replicas on that replica set, deployment, or stateful set. If the pod replicas ...

• aren't doing enough work, the numbers of replicas is decreased.
• are doing too much work, the number of replicas is increased.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-hpa
spec:
  # What kind and name are being targeted for autoscaling? Is it a replica set, deployment,
  # stateful set, or something else?
  scaleTargetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: my-ss
  # What are the minimum and maximum replicas that this HPA will scale to? At the time of
  # writing, the minimum number of replicas must be 1 or more (it can't be 0).
  minReplicas: 2
  maxReplicas: 10
  # What type of metrics are being collected? In this example, on average across all active
  # pod replicas, we want the CPU load to be 50%. If the average is more than 50%, scale
  # down the number of replicas. If it's more, scale up the number of replicas.
  #
  # The average utilization is referring to the amount of resource requested by the pod. In
  # this example, this is 50% of the CPU resource AS REQUESTED BY THE POD (via the pod's
  # spec.containers[].resources.requests for CPU).
  #
  # Instead of doing average percentage, you can also do absolute values by changing
  # target.type to AverageValue and replacing target.averageUtilization with
  # target.averageValue.
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50

In the example above, if the average CPU usage of the stateful set my-ss replicas is

• less than 50% that of the CPU resource requested, the replicas will scale down until there are 2 replicas or until the CPU usage reaches 50%.
• more than 50% that of the CPU resource requested, the replicas will scale up until there are 10 replicas or until the CPU usage reaches 50%.

The HPA will at-most double the number of replicas on each iteration. Each scaling iteration has an intentional waiting period. Specifically, scaling ...

• up will only occur if no previous scaling has occurred in the past 3 mins.
• down will only occur if no previous scaling has occurred in the past 5 mins.

Common gotchas with HPAs:

• Scale to zero: Setting the number of minimum replicas to zero isn't supported. See here.

• Scaling memory: It's easy to autoscale based on CPU, but be careful with autoscaling based on memory. If the number of replicas scale up, the existing replicas need to somehow "release" memory, which can't really be done without killing the pods and starting them back up again.

• Scaling on non-linear scaling metrics: Be careful with scaling based on metrics that don't linearly scale. For example, if you double the number of pods, the value of that metric should cut in half. This may end up becoming an issue when scaling based off poorly designed custom user-defined metrics.

Scaling Behavior

How an HPA scales up / down can be controlled through policies. It's common to control scaling behavior to reduce problems such as trashing of replicas (constantly introducing and evicting replicas).

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: my-ss
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50
  # Scaling behavior is defined through policies. Both scaling up and scaling down have
  # their own list of policies to select from. When a list contains more than one policy,
  # the one selected is defined by "spec.selectPolicy" (next field after this one).
  #
  # In addition to policies, scaling up and scaling down both have their own "stabilization
  # window" which is used to prevent thrasing of replicas (constantly introducing / evicting
  # replicas). The window tracks the highest replica count in the past n seconds and won't
  # let policies go through with setting it to a smaller value (e.g. use the highest
  # computed replica count over the past 5 mins).
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Pods # Allow at most 4 pod replicas to be added in a span of 1 min
          value: 4
          periodSeconds: 60 
        - type: Percent # Allow at most a 10% increase of pod replicas in a span of 1 min
          value: 10
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Pods  # Allow at most 3 pod replicas to be removed in a span of 2 mins
          value: 3
          periodSeconds: 120
        - type: Percent  # Allow at most a 10% decrease of pod replicas in a span of 5 mins
          value: 10
          periodSeconds: 300
  # When many policies are present for a scale up / down, the policy chosen can be either
  # the one causing the ...
  #
  #  * most change (e.g. most pods added), set by using "Max" as the value.
  #  * least change (e.g. least pods removed), set by using "Min" as the value.
  selectPolicy: Max

Metric Types

Metrics can be one of the following types types:

• Resource metrics cover CPU and memory metrics of the replicas.
• Pods metrics cover other metrics of the replicas.
• Object are metrics related to some other object in the same namespace.
• External are metrics related to something other than Kubernetes.

Resource is set directly by Kubernetes while Pods, Object, and External are for custom / user-defined metrics.

If multiple metrics being tracked by an HPA, as in the example below, that HPA calculates the replica counts for each metric and then chooses the one with the highest.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: my-ss
  minReplicas: 2
  maxReplicas: 10
  metrics:
    # Target an average of 50% CPU utilization across all replicas
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50
    # Target an average of 1000 queries-per-second across all replicas.
    - type: Pods
      pods:
        metric:
          name: queries-per-second
        target:
          type: AverageValue
          averageValue: 1000
    # Target an average of 1000 queries-per-second across all replicas.
    - type: Object
      object:
        metric:
          name: requests-per-second
        describedObject:
          apiVersion: extensions/v1
          kind: Ingress
          name: my-ingress
        target:
          type: Value
          averageValue: 2000

Cluster Autoscaler

A cluster autoscaler is a component that scales a Kubernetes cluster on a public cloud by adding and removing nodes as needed. Each public could has its own implementation of a cluster autoscaler. In some clouds, the implementation is exposed as a kind / set of kinds. In other clouds, the implementation is exposed as a web interface or a command-line interface.

In most cases, nodes and added and removed to predefined groups called node pools. Each node pool has nodes of the same type (same resources and features). For example, a specific node pool of machines with the same CPU, networking gear, and same amount and type of RAM.

If a pod is scheduled to run but none of the nodes have enough resources to run it, the cluster autoscaler will increase the number of nodes in one of the node pools that has the capability to run the pod. Likewise, the cluster autoscaler will decrease the number of nodes if nodes aren't being utilized enough by actively running pods.

Pod Disruption Budget

A pod disruption budget (PDB) specifies the number of downed pod replicas that a replica set, deployment, or stateful set can tolerate relative to its expected replica count. In this case, a disrupted pod is one that's brought down via ...

• voluntary disruptions (e.g. manually removing a node from the cluster).
• involuntary disruptions (e.g. kernel panic on a node).
• rolling updates (deployments and stateful set).

Once a PDB has reached its downed pod replica limit, it prevents further downing of pod replicas via voluntary disruptions. It cannot prevent further downing of pod replicas via involuntary disruptions or rolling updates (these downed pods will still be accounted for within the PDB).

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-pdb
spec:
  # Selector for pod replicas within the stateful set.
  selector:
    matchLabels:
      app: my-ss-pods
  # Max number of pods that can be unavailable at one time.
  #
  # Instead of "maxUnavailable", this can also be "minAvailable", which is the min number of
  # pods must be available at all times.
  maxUnavailable: 3

Security

Kubernetes comes with several features to enhance cluster security. These include gating off inter-cluster networking, gating off how containers can break isolation, and providing access control mechanisms to the API.

The subsections below detail various security related topics.

Network Policy

Network policies restrict pods from communicating with other network entities (e.g. services, endpoints, other pods, etc..). Restrictions can be for either inbound connections, outbound connections, or both.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: my-np
  namespace: my-ns
spec:
  # Which pods this network policy applies to is defined via pod labels. The pods have to be
  # in the same namespace as the network policy.
  podSelector:
    matchLabels:
      app: backend
  # Which directions is this network policy? Possible options include "Ingress" is for
  # inbound connections, "Egress" is for outbound connections, or both. If empty, it'll
  # default to just "Ingress" and "Egress" will also be set if there are any egress rules
  # below.
  policyTypes: [Ingress, Egress]
  # Inbound connection rules go here. Rules can be for IP blocks (CIDR), namespaces
  # (identified via labels), or pods (identified via labels). Each rule is for a specific
  # port.
  ingress:
    - from:
        # Allow all pods from another namespace.
        - namespaceSelector:
            matchLabels:
              project: my-company
        # Allow all pods in this namespace that have a particular set of labels.
        - podSelector:
            matchLabels:
              app: frontend
        # Allow all pods from another namespace that have a particular set of labels. Note
        # that this is a SINGLE ENTRY, not two separate entries (there is no dash before
        # "podSelector" like the "podSelector" above has).
        - namespaceSelector:
            matchLabels:
              project: my-company
          podSelector:
            matchLabels:
              app: frontend
        # Allow all pods from an IP block (with exceptions).
        - ipBlock:
            cidr: 172.17.0.0/16
            except:
              - 172.17.99.0/24
      ports:
        - protocol: TCP
          port: 8080
  # Outbound connection rules go here. This is specified in exactly the same way as the
  # rules for inbound connections, but the rules apply to outbound connections.
  egress:
    - to:
        - ipBlock:
            cidr: 10.0.0.0/24
      ports:
        - protocol: TCP
          port: 8888

The following are commonly used patterns for network policies.

# DENY ALL INGRESS TRAFFIC
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
spec:
  podSelector: {}
  policyTypes: [Ingress]
----
# ALLOW ALL INGRESS TRAFFIC
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-all-ingress
spec:
  podSelector: {}
  ingress:
  - {}
  policyTypes: [Ingress]
----
# DENY ALL EGRESS TRAFFIC
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-egress
spec:
  podSelector: {}
  policyTypes: [Egress]
----
# ALLOW ALL EGRESS TRAFFIC
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-all-egress
spec:
  podSelector: {}
  ingress:
  - {}
  policyTypes: [Egress]
----
# DENY ALL INGRESS AND EGRESS TRAFFIC
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]

Pod Security Admission

Pod security admissions are used to restrict the security context of pods using policy groups that define up-to-date best practices. Restricting requires two pieces of information: policy and mode. Specifically, ...

• policy defines the level of privileges to grant to pods, which can be either ...

  • privileged: Unrestricted usage (default).
  • baseline: Minimally restricted usage, based on known privilege escalations.
  • restricted: Maximally restricted usage, based on best practices of hardening pods.

  Note that the description above doesn't explicitly define what it is that's being restricted (it just says minimally restricted / maximally restricted). The definition of minimal / maximal are based on up-to-date best practices and update with different releases of Kubernetes. As such, to help keep things consistent, a policy can be pinned to use the definitions from a specific version of Kubernetes (e.g. apply definitions from Kubernetes v1.22 even though the version of Kubernetes being run is v1.25).

• mode defines what to do when a policy violation is detected, which can be either ...

  • enforce: Violating pods are rejected.
  • audit: Violating pods are not rejected, but a message will be recorded in the audit logs.
  • warn: Violating pods are not rejected, but a warning message will be shown to the user.

For example, setting ...

• warn to baseline means that pods will run but also warn if they're breaking the minimal set of restrictions.
• enforce to privileged means that pods won't run if they're unrestricted.

To apply to all pods cluster-wide, use the following object.

apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
  - name: PodSecurity
    configuration:
      apiVersion: pod-security.admission.config.k8s.io/v1
      kind: PodSecurityConfiguration
      defaults:
        enforce: privileged
        enforce-version: latest
        audit: restricted
        audit-version: "1.25"
        warn: baseline
        warn-version: "1.22"
      exemptions:
        usernames: []             # Usernames to exempt
        runtimeClasses: []        # Runtime class names to exempt
        namespaces: [kube-system] # Namespaces to exempt

To apply to all pods in a specific namespace, use the following namespace label templates.

• policy, use pod-security.kubernetes.io/<MODE>:<POLICY>.
• version, use pod-security.kubernetes.io/<MODE>-version:<VERSION> (defaults to latest if omitted).

pod-security.kubernetes.io/warn=baseline
pod-security.kubernetes.io/warn-version=1.22
pod-security.kubernetes.io/audit=restricted
pod-security.kubernetes.io/audit-version=1.25
pod-security.kubernetes.io/enforce=privileged
pod-security.kubernetes.io/enforce-version=latest

API Access Control

Access control to the Kubernetes API is modeled using accounts and groups, where an account can be associated with many groups and each group grants a certain set of permissions to its users. Two types of accounts are provided:

• users, which are accounts for human access.
• service accounts, which are accounts for programmatic access.

Service accounts are what get used when a pod needs access to the Kubernetes API. Containers within that pod have a volume mounted with certificates and credentials that the applications within can use to authenticate and communicate with the API server.

Each namespace gets created with a default service account (named default). By default, pods created under a namespace will be assigned the default service account for that namespace. A pod can be configured to use a custom service account that has more or less access rights than the default service account. A pod can also forgo volume mounting the credentials of a service account entirely if the applications within it don't need access to the Kubernetes API.

How access rights are defined depends on how Kubernetes has been set up. By default, Kubernetes is set up to use role-based access control (RBAC), which tightly maps to the REST semantics of the Kubernetes API server. RBAC limits what actions can be performed on which objects: Objects map to REST resources (paths on the REST server) and manipulations of objects map to REST actions (verbs such as DELETE, GET, PUT, etc.. on those REST server paths).

The subsections below detail RBAC as well as other API security related topics.

Role-based Access Control

RBAC tightly maps to the REST semantics of the Kubernetes API server by limiting what actions can be performed on which objects: Objects map to REST resources (paths on the REST server) and manipulations of objects map to REST actions (verbs such as DELETE, GET, PUT, etc.. on those REST server paths). RBAC is configured using two sets of kinds:

• Role and ClusterRole specify which actions can be performed on which kinds / in which namespace.
• RoleBinding and ClusterRoleBinding specify which roles bind to which users, groups, or service accounts.

A role binding always maps a single role to many users, groups, and service accounts.

ClusterRole and ClusterRoleBinding are cluster-level kinds while Role and RoleBinding are namespace-level kinds. RBAC provides different permissions based on which role variant (ClusterRole vs Role) gets used with which role binding variant (ClusterRoleBinding vs RoleBinding):