When deploying applications on cloud platforms like AWS, Google Cloud, or Azure, load balancing is often a seamless experience. Cloud providers offer managed load balancer services (e.g., AWS Application Load Balancer) that integrate effortlessly with Kubernetes clusters. However, when running Kubernetes on-premises, things get a bit more complicated. You need to handle load balancing yourself, which raises questions:

  • How do you distribute incoming traffic to your services?
  • How can you ensure high availability (HA) of your load balancer?
  • How do you automate updates when new services are deployed?

In this post, I’ll share how I tackled these challenges using MetalLB, a load balancer implementation for on-premises Kubernetes clusters.

The Challenge of Load Balancing On-Premises

In a cloud environment, creating a Kubernetes Service of type LoadBalancer provisions a cloud load balancer that distributes traffic to your service pods. On-premises, Kubernetes doesn’t provide an implementation for the LoadBalancer service type by default. This means you need to find an alternative solution.

Possible Approaches

  1. Deploying a Dedicated Load Balancer: You could set up a load balancer like NGINX or HAProxy on a separate server, exposing your services via NodePort. However, this approach has drawbacks:
    • Single Point of Failure: If the load balancer server fails, your services become unavailable.
    • Manual Updates: Every time you deploy or update a service, you need to manually adjust the load balancer configuration.
    • Scalability Concerns: Handling increased traffic might require scaling the load balancer, adding more complexity.
  2. Using Ingress Controllers: While ingress controllers manage HTTP/HTTPS traffic and can route requests to different services based on hostnames or paths, they don’t handle non-HTTP traffic and still require an external IP address.

To address these challenges, I turned to MetalLB.

Introducing MetalLB

MetalLB is a load-balancer implementation for bare-metal Kubernetes clusters, using standard routing protocols. It allows you to create services of type LoadBalancer in environments that don’t natively support this feature.

Key Features

  • Protocols: Supports both Layer 2 (ARP/NDP) and BGP modes.
  • Easy Integration: Works with existing Kubernetes services without requiring changes to your applications.
  • High Availability: Distributes traffic across multiple nodes, and can be configured for redundancy.

Understanding MetalLB Layer 2 Mode

MetalLB operates in two modes:

  1. Layer 2 Mode: MetalLB uses Address Resolution Protocol (ARP) on IPv4 or Neighbor Discovery Protocol (NDP) on IPv6 to make the services reachable on the local network. It’s suitable for simple network setups and is easier to configure.
  2. BGP Mode: MetalLB uses Border Gateway Protocol (BGP) to announce service IPs to upstream routers. This mode is more complex but offers greater control and scalability, especially in larger networks.

For my small cluster, I chose Layer 2 mode because:

  • Simplicity: Easier to set up without needing to configure BGP peers.
  • Compatibility: Works well in a flat Layer 2 network, which is common in small on-premises setups.

Deploying MetalLB in Layer 2 Mode

Prerequisites

  • A functioning Kubernetes cluster (e.g., K3s, which I used).
  • Kubernetes version 1.13 or newer.
  • Access to a range of IP addresses within your network that are not used by other devices.

Step 1: Disable ServiceLB in K3s (if applicable)

If you’re using K3s, it comes with a built-in service load balancer called ServiceLB. To avoid conflicts with MetalLB, disable ServiceLB when installing K3s:

curl -sfL https://get.k3s.io | sh -s - server --disable servicelb

Step 2: Install MetalLB

Apply the MetalLB manifest to your cluster:

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.10/config/manifests/metallb-native.yaml

This command deploys MetalLB in the metallb-system namespace.

Step 3: Configure MetalLB

Create a configuration file named metallb.yaml:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: pool
  namespace: metallb-system
spec:
  addresses:
    - 10.20.30.100-10.20.30.105
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: l2-advertisement
  namespace: metallb-system
spec:
  ipAddressPools:
    - pool

Explanation:

  • IPAddressPool:
    • metadata.name: The name of the IP address pool (pool in this case).
    • spec.addresses: A list of IP addresses or CIDR ranges that MetalLB can use to assign to LoadBalancer services. Ensure these IPs are:
      • Within your local network subnet.
      • Not in use by any other devices.
      • Outside of your DHCP server’s allocation range to avoid conflicts.
  • L2Advertisement:
    • metadata.name: Name of the Layer 2 advertisement configuration.
    • spec.ipAddressPools: References the IP address pool created earlier.

Apply the Configuration

kubectl apply -f metallb.yaml

Step 4: Verify MetalLB Deployment

Check that MetalLB pods are running:

kubectl get pods -n metallb-system

You should see the controller and speaker pods running.

Using MetalLB with Services

Now, when you create a Kubernetes Service of type LoadBalancer, MetalLB will assign it an IP address from the pool you configured.

Example Service

Here’s an example of a service manifest:

apiVersion: v1
kind: Service
metadata:
  name: my-service
  namespace: my-namespace
spec:
  selector:
    app: my-app
  type: LoadBalancer
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080

Deploy the Service

kubectl apply -f my-service.yaml

Verify the Assigned IP

After deploying the service, check its status:

kubectl get service my-service -n my-namespace

You should see an external IP assigned from the range you specified (e.g., 10.20.30.100).

How It Works

  • Traffic Flow:
    • When a request is sent to the external IP (10.20.30.100), MetalLB handles the ARP requests and directs the traffic to one of the nodes running the MetalLB speaker.
    • The request is then forwarded to the service’s pods based on the service configuration.
  • High Availability:
    • If a node fails, MetalLB will ensure that the IP is re-announced from a healthy node, maintaining service availability.

Important Considerations

Network Configuration

  • Layer 2 Network: Ensure all your Kubernetes nodes are on the same Layer 2 network segment. MetalLB’s Layer 2 mode relies on ARP/NDP broadcasts, which don’t cross routers.

IP Address Management

  • Avoid IP Conflicts: Double-check that the IP addresses assigned to MetalLB are not used elsewhere in your network.
  • DHCP Range: Exclude MetalLB’s IP range from your DHCP server’s allocation pool.

Security

  • Firewall Rules: Update your firewall settings to allow traffic to the assigned IP addresses and necessary ports.
  • Access Control: Limit access to the MetalLB-configured IPs as needed to prevent unauthorized access.

Troubleshooting

Service Not Getting an External IP

  • Check MetalLB Pods: Ensure the MetalLB controller and speaker pods are running without errors.
  • Validate Configuration: Confirm that the IPAddressPool and L2Advertisement configurations are correct.
  • Logs: Examine logs from MetalLB pods for any error messages.

Connectivity Issues

  • Network Reachability: Verify that the assigned IP addresses are reachable from your client machines.
  • Port Accessibility: Ensure the service ports are correctly exposed and that there are no network policies blocking traffic.

Conclusion

MetalLB provides a powerful and flexible solution for implementing load balancing in on-premises Kubernetes clusters. By simulating the functionality of cloud provider load balancers, it simplifies the deployment and management of services requiring external access.

Key Takeaways

  • Ease of Use: MetalLB integrates seamlessly with Kubernetes, requiring minimal configuration.
  • Flexibility: Supports both Layer 2 and BGP modes to fit different network environments.
  • Scalability: Easily handles multiple services and can be expanded by adjusting the IP address pool.

By leveraging MetalLB in Layer 2 mode, I was able to:

  • Automate the assignment of external IPs to services.
  • Maintain high availability without setting up additional load balancer hardware.
  • Simplify the management of network resources in my on-premises cluster.