Kubernetes & Azure Kubernetes Service (AKS)

Executive Summary

Kubernetes is a declarative control-plane system for scheduling and reconciling containerized workloads across a fleet of machines. AKS is Microsoft Azure's managed distribution of Kubernetes: Azure operates and pays the SLA on the control plane (API server, etcd, scheduler, controller-manager), while you manage — or, with AKS Automatic, largely delegate — node pools, networking, and add-ons.

Core Concepts

The declarative reconciliation loop

Everything in Kubernetes reduces to one pattern: you write desired state as an object in etcd (via the API server); a controller continuously observes actual state and drives it toward desired state. There is no imperative "do this now" — only convergence loops running forever.

Pods, not containers

The Pod — not the container — is the atomic scheduling unit. A Pod is one or more containers sharing a network namespace (one IP) and optionally storage volumes. Containers in the same Pod always co-locate on the same node.

Architecture Overview

A Kubernetes cluster splits into a control plane (brains) and a data plane (worker nodes running your workloads). In AKS, Microsoft hosts and manages the control plane in an Azure-managed subscription; you only see and pay for (in Standard/Premium tiers) the API endpoint and SLA.

Fig 1. Kubernetes control plane vs. data plane, mapped to AKS's managed/customer-managed split.

Internal Working

Scheduling: filter, then score

The scheduler runs two phases per unscheduled pod:

1. Filtering — eliminate nodes that cannot run the pod (insufficient CPU/memory, taints without matching tolerations, node selectors/affinity not satisfied, port conflicts).
2. Scoring — rank remaining nodes (spread pods across nodes/zones, prefer nodes with the image already cached, balance resource utilization) and bind to the top-scoring node.

kubelet's control loop

Each node's kubelet watches the API server for pods bound to it, then drives the container runtime (via the CRI — Container Runtime Interface) to start/stop containers, reports node and pod status back, and runs liveness/readiness/startup probes.

Networking model

• Every pod gets its own IP; pods can reach all other pods without NAT (the "flat network" requirement).
• kube-proxy (or eBPF dataplanes like Cilium) programs iptables/IPVS/eBPF rules so a Service's virtual IP load-balances to healthy pod IPs.
• CoreDNS resolves in-cluster DNS names (svc-name.namespace.svc.cluster.local).

etcd: the only stateful thing that matters

Key Components

Request / Execution Flow

Trace of kubectl apply -f deployment.yaml from client to running containers:

Fig 2. End-to-end reconciliation path for a Deployment rollout.

AKS Networking

This is the single most consequential decision at cluster creation time — hard to change later.

Example: create a cluster with Overlay + Cilium

# az CLI — network-plugin-mode=overlay + Cilium dataplane
az aks create \
  --resource-group rg-platform-prod \
  --name aks-prod-eastus \
  --network-plugin azure \
  --network-plugin-mode overlay \
  --network-dataplane cilium \
  --pod-cidr 192.168.0.0/16 \
  --service-cidr 10.0.0.0/16 \
  --dns-service-ip 10.0.0.10 \
  --node-vm-size Standard_D4s_v5 \
  --enable-managed-identity \
  --generate-ssh-keys

Explanation: --network-plugin-mode overlay decouples pod IPs from the VNet; --network-dataplane cilium swaps kube-proxy's iptables rules for Cilium's eBPF programs, which also unlocks L3/L4/L7 network policies and better observability.

Identity & Access (AKS)

Two separate identity problems

Example: federate a Kubernetes ServiceAccount with an Entra Workload Identity

# 1. Create a user-assigned managed identity and federated credential
az identity create -g rg-platform-prod -n wi-orders-api

az identity federated-credential create \
  --name orders-api-fic \
  --identity-name wi-orders-api \
  --resource-group rg-platform-prod \
  --issuer "$(az aks show -g rg-platform-prod -n aks-prod-eastus --query oidcIssuerProfile.issuerUrl -o tsv)" \
  --subject system:serviceaccount:orders:orders-api-sa \
  --audience api://AzureADTokenExchange

apiVersion: v1
kind: ServiceAccount
metadata:
  name: orders-api-sa
  namespace: orders
  annotations:
    azure.workload.identity/client-id: "<managed-identity-client-id>"
  labels:
    azure.workload.identity/use: "true"

Explanation: the annotation tells the AKS-managed webhook which Entra app to mint tokens for; the label opts the ServiceAccount's pods into token projection. No client secret ever touches the cluster.

AKS Automatic vs Standard

Autoscaling Model

Important APIs & Objects

Command Cheat Sheets

kubectl — everyday operations

az aks — cluster lifecycle

helm — package management

Configuration Examples

Deployment with resource requests/limits + probes

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
  namespace: orders
spec:
  replicas: 3
  selector:
    matchLabels: { app: orders-api }
  template:
    metadata:
      labels: { app: orders-api }
    spec:
      serviceAccountName: orders-api-sa
      containers:
        - name: api
          image: myacr.azurecr.io/orders-api:1.4.2
          resources:
            requests: { cpu: "250m", memory: "256Mi" }
            limits:   { memory: "512Mi" }
          readinessProbe:
            httpGet: { path: /healthz, port: 8080 }
            initialDelaySeconds: 5
          livenessProbe:
            httpGet: { path: /livez, port: 8080 }
            periodSeconds: 10

Explanation: CPU requests influence scheduling; only a memory limit is set, since exceeding a CPU limit throttles (usually fine) while exceeding a memory limit OOM-kills the pod — many teams intentionally omit CPU limits to avoid artificial throttling.

PodDisruptionBudget — protect availability during node drains/upgrades

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: orders-api-pdb
  namespace: orders
spec:
  minAvailable: 2
  selector:
    matchLabels: { app: orders-api }

Explanation: guarantees at least 2 of 3 replicas stay up during voluntary disruptions (node drains, AKS node-image upgrades) — without this, an upgrade can legally evict all replicas at once.

HorizontalPodAutoscaler

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: orders-api-hpa
  namespace: orders
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: orders-api
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target: { type: Utilization, averageUtilization: 70 }

NetworkPolicy — default-deny then allow

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: orders
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-gateway
  namespace: orders
spec:
  podSelector: { matchLabels: { app: orders-api } }
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: ingress } }

Terraform — minimal AKS Automatic-style cluster

resource "azurerm_kubernetes_cluster" "aks" {
  name                = "aks-prod-eastus"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  dns_prefix          = "aksprod"
  sku_tier            = "Standard"  # financially backed SLA

  default_node_pool {
    name       = "system"
    vm_size    = "Standard_D4s_v5"
    auto_scaling_enabled = true
    min_count  = 3
    max_count  = 6
  }

  identity { type = "SystemAssigned" }

  network_profile {
    network_plugin      = "azure"
    network_plugin_mode = "overlay"
    network_data_plane  = "cilium"
  }

  oidc_issuer_enabled       = true
  workload_identity_enabled = true
}

Best Practices

• Always set resource requests; be deliberate about whether you set CPU limits (throttling risk) vs leaving CPU unbounded.
• Define PodDisruptionBudgets for every multi-replica workload before your first cluster upgrade.
• Use readiness probes to gate traffic and liveness probes only to catch true deadlocks — an overly aggressive liveness probe causes needless restarts.
• Prefer Microsoft Entra Workload Identity over any static credential/secret for pod-to-Azure auth.
• Default new clusters to Azure CNI Overlay + Cilium; reserve VNet-mode CNI for workloads needing direct VNet routability or Application Gateway ingress.
• Pin and stagger Kubernetes upgrades: AKS supports 3 GA minor versions at a time — don't let clusters drift onto a version approaching End of Life.
• Separate system and user node pools (or use AKS Automatic, which does this for you) so app workloads never crowd out CoreDNS/metrics-server.
• Enable Azure Policy / Deployment Safeguards to prevent common misconfigurations (privileged containers, missing resource limits) before they reach production.
• Use namespaces + ResourceQuotas + NetworkPolicies as your default multi-tenant isolation boundary, not just RBAC.

Common Mistakes

Performance Considerations

• Networking dataplane: Azure CNI Overlay performs within 2-3% of VNET-mode CNI for inter-pod traffic, whereas Kubenet shows a 10-15% latency penalty from the extra routing hop and NAT complexity.
• Node provisioning latency: managed Karpenter-based Node Auto Provisioning provisions right-sized nodes far faster than legacy VM Scale Set-based Cluster Autoscaler paths — matters directly for burst-scaling SLAs.
• Image pulls: cold-start latency is dominated by image pull time on new nodes; use smaller base images and, where available, artifact streaming/pre-caching.
• etcd health: although Azure-managed in AKS, etcd write latency (driven by object churn — excessive ConfigMap/Secret updates, high-frequency status writes) is still the ultimate ceiling on API responsiveness.
• DNS: CoreDNS can become a bottleneck at scale; AKS Automatic's LocalDNS caches resolutions per-node to cut latency and query volume.

Security Considerations

• Disable local cluster admin accounts; enforce Microsoft Entra ID for API authentication, with Azure RBAC or Kubernetes RBAC for authorization.
• Use Microsoft Entra Workload ID (not the deprecated Pod Identity add-on) for all pod-to-Azure authentication — no long-lived secrets in the cluster.
• Default-deny NetworkPolicies per namespace, opening only required paths — trivial with a Cilium dataplane, which enforces policy natively.
• Enable Image Cleaner to automatically remove images with known vulnerabilities from nodes (on by default in AKS Automatic).
• Use Azure Policy / Deployment Safeguards to block privileged containers, hostPath mounts, and missing resource limits at admission time.
• Encrypt etcd secrets at rest — AKS supports customer-managed keys for an additional layer beyond the platform default.
• Scope Azure Container Registry pulls via managed identity, not static registry credentials embedded in imagePullSecrets.

Troubleshooting Guide

Interview Questions

Cheat Sheet

Glossary