Kubernetes Networking Deep Dive: Intro
A Four-Part Series on the Life of a Packet within Kubernetes
I was having a fairly random tech conversation with a friend of mine and out of curiosity, he asked me some questions about Kubernetes networking, knowing I had spent time in that universe. As I explained to him the high-level basics, he dug deeper (as I am happy he always does) detailed questions about the movement of the packet into, out of, and through a cluster. I soon came to realize some gaps in my knowledge.
I went back home and started looking up answers. I found my efforts snowballing into a lot of information. Then it hit me: why not start off my first Substack posts with a series on Kubernetes networking, so that others could benefit from this work.
I decided to do this as a “Life of a Packet” through a Kubernetes cluster. In this series of posts, I will trace out the path from an external user through a load balancer, into the cluster, to a container in a pod, and back out again. I will then cover pod-to-pod communication within the cluster and touch upon encryption.
My goal is to illustrate what not only within Kubernetes constructs, but also what happens at the network level, below the abstractions of Kubernetes. I’ll take a look at iptables rules, network namespaces, veth pairs, and CNI plugins. I will map a packet’s journey including OSI layers: Layer 2 switching, Layer 3 routing, and Layer 4 load balancing.
I purposefully kept this cloud services agnostic, and I do not cover the concept of the Service Meshes since that warrants posts of its own. This will be standard, simple, Kubernetes networking infrastructure, providing foundational knowledge to anyone growing their Kubernetes skills.
Here’s a breakdown of the series:
Part 1: Foundations
This is where I will go over core concepts:
The Kubernetes networking model
Pod and service CIDR allocation
Linux network namespaces
VETH pairs
The Container Network Interface (CNI).
I will also cover kube-proxy’s role and compare iptables vs IPVS from a high-level.
Part 2: Pod-to-Pod Communication (East-West Traffic)
I’ll then trace a packet from one pod to another otherwise known as “East-West” traffic.
This will go over two scenarios:
same-node communication (through a bridge)
cross-node communication
For cross-node traffic, I will explain overlay networking (VXLAN encapsulation) and routed networking (BGP). Don’t get too scared. I’ll have diagrams to show the packet structure at each hop, including what gets encapsulated and where. Of course, if it’s too in-the-weeds you can always skip this bit. Weeds can be fun, though.
Part 3: External Traffic to Pods (North-South Traffic)
Once I’m finished going over traffic within a cluster, I’ll pull back a bit and discuss traffic into and out of a cluster, called “North-South” traffic. This will begin its journey at an external user, go through a Load Balancer service, then land on a pod.
I will dig into iptables: PREROUTING, the KUBE-SERVICES chain, service endpoint selection, DNAT, and SNAT. (Wow. Say that 10 times fast.)
I’ll also show the return path and how connection tracking enables it. And I’ll touch upon externalTrafficPolicy and its tradeoffs for client IP preservation.
Part 4: Encryption In-Flight
Finally, I get into encryption. Oh, yeah, this is always fun, “TLS and Certificates and Keys... on my”.
I will cover:
TLS termination at the load balancer
TLS passthrough to ingress controllers
backend TLS to pods
CNI-level encryption for pod-to-pod traffic.
I will also touch upon the tradeoffs among the different termination points and briefly cover network policies as a complementary security layer.
What You Will Need
If you haven’t been scared off already, note that for these posts I will assume some familiarity with basic networking concepts (TCP/IP) and core Kubernetes concepts (pods, services, nodes, the kubelet). It couldn’t hurt to review the networking OSI model but deep expertise is NOT required. I will include some command examples you can try out for yourself.
Next post: Kubernetes Networking Deep Dive, Part 1: Foundations