In my previous posts, I discussed how I used MetalLB to implement load balancing in my on-premises Kubernetes cluster. While MetalLB served its purpose by providing Layer 2 load balancing, I heard Cilium and discovered its powerful networking capabilities. In this post, I’ll explain what Cilium is, why it’s beneficial, and how I replaced both my Container Network Interface (CNI) and MetalLB with Cilium.

Understanding Cilium and Its Advantages

What Is Cilium?

Cilium is an open-source networking, security, and load-balancing solution for Kubernetes that leverages eBPF (extended Berkeley Packet Filter) technology in the Linux kernel. eBPF allows programs to run in the kernel space safely, providing high-performance networking and security features without the need for kernel module modifications.

Why Choose Cilium?

  • Advanced Networking: Cilium provides networking capabilities, including transparent encryption, load balancing, and network policies.
  • eBPF-Powered Performance: By utilizing eBPF, Cilium achieves high performance and low latency.
  • Rich Network Policies: Offers extensive network policy support, enabling fine-grained security controls.
  • Integration: Seamlessly integrates with Kubernetes and replaces components like kube-proxy for better performance.
  • Simplified Load Balancing: Can replace MetalLB by providing load balancing through its own mechanisms.

Replacing MetalLB and Default CNI with Cilium

The Need for a Better Solution

While MetalLB provided a way to implement load balancing, using multiple tools for networking and load balancing added complexity. Cilium offers a unified solution that handles both networking and load balancing efficiently.

Benefits of Using Cilium’s CNI

  • Unified Networking Stack: Eliminates the need for separate CNI plugins and load balancers.
  • Enhanced Performance: Reduces latency and overhead by leveraging eBPF.
  • Simplified Management: Easier to manage and troubleshoot with a single tool.
  • Advanced Features: Supports features like direct server return, session affinity, and more.

Installing Kubernetes Without a Default CNI

To use Cilium as the CNI, we need to install Kubernetes without any default CNI plugins. Since I’m using K3s, I’ll demonstrate how to set it up accordingly.

Prerequisites

  • Fresh Kubernetes Cluster: Ensure that no other CNI is installed.
  • Access to All Nodes: SSH access to the master and worker nodes.
  • Firewall Configuration: Proper firewall settings to allow necessary traffic.

Step 1: Install K3s Without a Default CNI

On the master node, run:

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="v1.31.1+k3s1" sh -s - server \
  --cluster-init \
  --disable=servicelb \
  --disable=traefik \
  --flannel-backend=none \
  --disable-network-policy \
  --disable-kube-proxy

Explanation:

  • -disable=servicelb: Disables the built-in service load balancer.
  • -disable=traefik: Disables the default Traefik ingress controller.
  • -flannel-backend=none: Disables the default Flannel CNI.
  • -disable-network-policy: Disables the default network policy controller.
  • -disable-kube-proxy: Disables kube-proxy since Cilium will replace it.

On worker nodes, run:

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="v1.31.1+k3s1" K3S_TOKEN=<token> sh -s - server \
  --server https://<master_ip>:6443 \
  --disable=servicelb \
  --disable=traefik \
  --flannel-backend=none \
  --disable-network-policy \
  --disable-kube-proxy

Note: Replace <token> with the node token from the master node and <master_ip> with the master node’s IP address.

Step 2: Install Cilium Using Helm

We’ll use Helm to install Cilium into the cluster.

Add the Cilium Helm Repository

helm repo add cilium https://helm.cilium.io/
helm repo update

Install Cilium

First, set environment variables for the API server:

export API_SERVER_IP=<master_ip>
export API_SERVER_PORT=6443

Basic Installation:

helm install cilium cilium/cilium \
  --namespace kube-system \
  --set kubeProxyReplacement=strict \
  --set k8sServiceHost=${API_SERVER_IP} \
  --set k8sServicePort=${API_SERVER_PORT}

Explanation:

  • kubeProxyReplacement=strict: Replaces kube-proxy entirely with Cilium’s implementation.
  • k8sServiceHost and k8sServicePort: Specifies the API server’s host and port.

Configuring Cilium for Layer 2 Load Balancing

Since we’re replacing MetalLB, we need to configure Cilium to handle load balancing at Layer 2.

Enable Layer 2 Announcements

Install Cilium with Layer 2 (L2) announcements enabled:

helm install cilium cilium/cilium --version 1.16.3 \
  --namespace kube-system \
  --set operator.replicas=1 \
  --set l2announcements.enabled=true \
  --set externalIPs.enabled=true \
  --set kubeProxyReplacement=strict \
  --set k8sServiceHost=${API_SERVER_IP} \
  --set k8sServicePort=${API_SERVER_PORT} \
  --set k8sClientRateLimit.qps=50 \
  --set k8sClientRateLimit.burst=100

Explanation:

  • l2announcements.enabled=true: Enables Layer 2 load balancing.
  • externalIPs.enabled=true: Allows the use of external IPs.

Define L2 Announcement Policy

Create a CiliumL2AnnouncementPolicy to specify how L2 announcements should be handled:

apiVersion: cilium.io/v2alpha1
kind: CiliumL2AnnouncementPolicy
metadata:
  name: default-l2-announcement-policy
  namespace: kube-system
spec:
  nodeSelector: {}
  interfaces:
    - ens192
    - '^eth[0-9]+'
  externalIPs: true
  loadBalancerIPs: true

Explanation:

  • interfaces: Specifies the network interfaces to use for announcements.
    • ens192: Example interface name.
    • '^eth[0-9]+': Regex to match interfaces like eth0, eth1, etc.
  • externalIPs and loadBalancerIPs: Enables handling of external and load balancer IPs.

Apply the policy:

kubectl apply -f cilium-l2-announcement-policy.yaml

Define Load Balancer IP Pool

Create a CiliumLoadBalancerIPPool to specify the IP address pool for load balancer services:

apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
  name: default-pool
spec:
  blocks:
    - cidr: 10.20.30.100/30

Explanation:

  • blocks: Defines the CIDR blocks from which Cilium can assign IPs for load balancer services.

Apply the pool configuration:

kubectl apply -f cilium-load-balancer-ip-pool.yaml

Updating Cilium Configuration

If you need to enable additional features like Hubble (Cilium’s observability platform), you can update the installation:

helm upgrade cilium cilium/cilium \
  --namespace kube-system \
  --reuse-values \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true

Explanation:

  • -reuse-values: Reuses the existing Helm values.
  • hubble.relay.enabled=true: Enables Hubble Relay.
  • hubble.ui.enabled=true: Enables the Hubble UI.

Why Cilium’s CNI Is Better

Enhanced Performance with eBPF

  • Low Overhead: eBPF allows Cilium to run networking functions in the kernel space, reducing context switches and improving performance.
  • Efficient Packet Processing: Directly manipulates packets without relying on iptables, leading to faster networking.

Replacing kube-proxy

  • Simplified Networking: By replacing kube-proxy, Cilium streamlines the networking stack.
  • Consistent Policies: Applies network policies consistently across Layers 3-7.

Advanced Network Policies

  • Identity-Based Security: Uses identities for endpoints, enabling more granular security policies.

Built-In Load Balancing

  • Service Load Balancing: Handles service load balancing without additional components.
  • External Load Balancing: Manages external IPs and load balancer IPs, eliminating the need for MetalLB.

Observability with Hubble

  • Deep Visibility: Hubble provides real-time monitoring and troubleshooting.
  • Flow Monitoring: Visualizes network flows and policy enforcement.

Future Exploration

Cilium offers many features beyond basic networking and load balancing:

  • Encryption: Transparent encryption of network traffic.
  • Cluster Mesh: Connects multiple clusters together.
  • Ingress and Egress Policies: Controls traffic entering and leaving the cluster.
  • Service Mesh Integration: Works alongside service meshes like Istio.

I plan to explore these features in future posts to further enhance the networking capabilities of my Kubernetes cluster.

Conclusion

By replacing MetalLB and the default CNI with Cilium, I achieved a more efficient, secure, and manageable networking solution for my Kubernetes cluster.