Skip to content

Network Topology

This document explains the 5G overlay networking stack from first principles: why it exists, how each technology fits together, and what the actual bridge/tunnel topology looks like. Read Virtualization Layers first for context on why a secondary CNI is needed.

The Problem: 5G Needs Multiple Isolated Interfaces

Standard Kubernetes gives every pod one network interface (eth0) via the primary CNI (Flannel in this cluster). That is sufficient for web services. It is not sufficient for 5G NFs.

A real 5G AMF must have:

  • an N1 interface (NAS signalling to UEs)
  • an N2 interface (NGAP control plane to gNBs)
  • an SBI interface (Service-Based Interface to other NFs)

A UPF must have N3 (GTP-U from gNBs), N4 (PFCP from SMF), and N6 (towards the data network), all on different subnets, all expected to be isolated from each other.

Putting everything on one Flannel interface is not acceptable: it breaks the 3GPP reference point architecture, mixes traffic from different planes on the same IP, and prevents per-interface traffic policies. The rules for which planes may cross, and where, are in Plane Isolation.

The Solution Stack

graph TB
    APP["5G NF Pod  (e.g. AMF)
    eth0 = Flannel  ·  n1 = N1 net  ·  n2 = N2 net"]

    subgraph CNI["CNI Layer"]
        MULTUS["Multus  (meta-CNI)
        reads k8s.v1.cni.cncf.io/networks annotation
        delegates to primary + secondary CNIs"]
        FLANNEL["Flannel CNI
        → eth0  (cluster traffic, K8s services)"]
        OVSCNI["OVS CNI plugin
        → nX  (one interface per N-reference-point)"]
    end

    subgraph IPAM["IP Management"]
        WA["Whereabouts
        cluster-wide IPAM
        allocation stored in K8s CRDs"]
    end

    subgraph Dataplane["Data Plane"]
        OVS["Open vSwitch bridge
        br-n1 · br-n2 · br-n3 · br-n4 · br-n6e · br-n6c · br-n6m"]
        VXLAN["VXLAN tunnel
        UDP 4789 · per-bridge VNI key"]
    end

    APP --> MULTUS
    MULTUS --> FLANNEL
    MULTUS --> OVSCNI
    OVSCNI --> WA
    OVSCNI --> OVS
    OVS --> VXLAN

Multus (meta-CNI)

Multus is not a CNI itself: it is a shim that delegates to other CNIs. When a pod is created, Multus reads the k8s.v1.cni.cncf.io/networks annotation, calls the primary CNI first (Flannel → eth0), then calls each secondary CNI listed in the annotation (OVS → n1, n2, etc.).

# Example pod annotation
metadata:
  annotations:
    k8s.v1.cni.cncf.io/networks: |
      [
        {"name": "n2-net", "interface": "n2"},
        {"name": "n3-net", "interface": "n3"}
      ]

Secondary interfaces are defined by NetworkAttachmentDefinitions (NADs), Kubernetes CRDs that specify the CNI plugin and IPAM configuration for each network.

OVS CNI plugin

The OVS CNI plugin connects a pod's network namespace to a named OVS bridge by creating a veth pair: one end in the pod netns (the n2 interface), the other end plugged into the OVS bridge as a port. The pod can then send frames that traverse the OVS bridge and reach any other port on that bridge, including the VXLAN tunnel port to the peer node.

Whereabouts IPAM

host-local IPAM (the Kubernetes default) allocates IPs from a per-node pool. Two pods on different nodes can get the same IP, fine for Flannel, catastrophic for the 5G overlays where AMF on worker and gNB on edge must reach each other by IP.

Whereabouts stores IP allocations in Kubernetes CRDs (IPAllocation objects), so it has cluster-wide visibility and ensures uniqueness across nodes. It serves the pool NADs only; fixed endpoints use static IPAM (see below), and their addresses are excluded from the pool.

Open vSwitch

OVS is a programmable software switch. Unlike Linux bridge, it supports OpenFlow, QoS, traffic statistics, and per-flow rules. In this testbed it serves as the data plane for the 5G overlays:

  • One OVS bridge per 5G interface (6 bridges total per node)
  • VXLAN tunnel ports connect the worker and edge bridges for each network
  • With a physical gNB, a separate bridge br-ran holds the worker's RAN NIC; the AMF attaches to it and the worker routes N3 from it to br-n3

OVS Bridge Topology

Six bridges per node

graph LR
    subgraph W["Worker  192.168.56.11"]
        direction TB

        BN1W["br-n1
        VNI 101
        10.201.0.0/24
        N1 · NAS"]

        BN2W["br-n2
        VNI 102
        10.202.0.0/24
        N2 · NGAP"]

        BN3W["br-n3
        VNI 103
        10.203.0.0/24
        N3 · GTP-U"]

        BN4W["br-n4
        VNI 104
        10.204.0.0/24
        N4 · PFCP"]

        BN6EW["br-n6e
        VNI 106
        10.206.0.0/24
        N6 edge · MEC"]

        BN6CW["br-n6c
        VNI 107
        10.207.0.0/24
        N6 cloud · DN"]

        BN6MW["br-n6m
        VNI 108
        10.208.0.0/24
        N6 MEC · services"]
    end

    subgraph E["Edge  192.168.56.12"]
        direction TB

        BN1E["br-n1
        VNI 101"]

        BN2E["br-n2
        VNI 102"]

        BN3E["br-n3
        VNI 103"]

        BN4E["br-n4
        VNI 104"]

        BN6EE["br-n6e
        VNI 106"]

        BN6CE["br-n6c
        VNI 107"]
    end

    BN1W <-->|"VXLAN
    UDP 4789"| BN1E

    BN2W <-->|"VXLAN
    UDP 4789"| BN2E

    BN3W <-->|"VXLAN
    UDP 4789"| BN3E

    BN4W <-->|"VXLAN
    UDP 4789"| BN4E

    BN6EW <-->|"VXLAN
    UDP 4789"| BN6EE

    BN6CW <-->|"VXLAN
    UDP 4789"| BN6CE

Bridge reference table

Bridge VNI Subnet Gateway Purpose
br-n1 101 10.201.0.0/24 10.201.0.1 N1 — NAS UE↔AMF
br-n2 102 10.202.0.0/24 10.202.0.1 N2 — NGAP gNB↔AMF
br-n3 103 10.203.0.0/24 10.203.0.1 N3 — GTP-U gNB↔UPF
br-n4 104 10.204.0.0/24 10.204.0.1 N4 — PFCP SMF↔UPF
br-n6e 106 10.206.0.0/24 10.206.0.1 N6 edge — MEC local breakout
br-n6c 107 10.207.0.0/24 10.207.0.1 N6 cloud — internet data network
br-n6m 108 10.208.0.0/24 10.208.0.1 N6 MEC — UPF-Cloud ↔ MEC service pods

VXLAN tunnel configuration

Each VXLAN tunnel uses a dedicated VNI key so that frames from different 5G interfaces cannot cross-contaminate:

# Example: VXLAN port on br-n2 (worker side)
ovs-vsctl add-port br-n2 vxlan-n2 -- \
  set interface vxlan-n2 type=vxlan \
  options:key=102 \
  options:remote_ip=192.168.56.12 \
  options:local_ip=192.168.56.11

The VXLAN tunnel runs over the management NIC (eth1, 192.168.56.0/24) between worker and edge. All six bridge tunnels share this single physical link.

Local switching versus inter-node tunnels

A VXLAN tunnel only carries overlay traffic between nodes. Two pods on the same node and the same overlay communicate through local OVS switching: the bridge forwards frames between their veth ports at L2, with no encapsulation. The VXLAN port is used only to reach a pod on a different node, for example AMF on the worker and a gNB or UPF on the edge.

VXLAN is therefore edge-conditional. ovs-setup.sh adds the per-interface VXLAN ports only when the edge node is enabled. With edge disabled (the default server profile without edge, or any single-node deployment) the bridges are created without VXLAN ports and every NF runs on the worker, so all overlay traffic stays local. In that case sudo ovs-vsctl show lists no vxlan-* interfaces, which is expected rather than a fault.

The flannel.1 VXLAN device present on every node belongs to the primary cluster CNI (Flannel pod network) and is unrelated to the 5G overlays.

MTU sizing and GTP-U encapsulation

VXLAN adds 50 bytes of header (14 outer Ethernet + 20 IP + 8 UDP + 8 VXLAN) to every frame, so overlay interfaces (n1…n6m) use overlay_mtu: 1450 (defined in ansible/group_vars/all.yml) instead of the 1500 default of the underlying eth1.

The host-only adapter between the host and the VMs is a separate case: its MSS is right, but the host's segmentation offload bypasses it on the first burst of a connection and the VM drops the oversized frame. See ../known-issues/virtualbox-hostonly-tso.md.

The N6 cloud bridge (br-n6c) and NAD n6c-net use n6_data_mtu: 1400 so host and pod TCP stacks on the decapsulated data network match the UPF ogstun MTU (1400). N3 bridges stay at overlay_mtu (1450); that outer budget must still fit GTP-U encapsulation on the RAN path.

The user plane adds a second layer of encapsulation on top of that. Packets leaving the UPF on ogstun are re-encapsulated by open5gs-upfd in GTP-U (~40 bytes of outer IP + UDP + GTP header) before being emitted on n3 towards the gNB. Without MTU adjustment the chain is:

UE payload  →  ogstun (1500)  →  UPF encaps GTP-U (+40)  →  n3 (1450)  =  fragmentation or drop

To keep UE traffic inside the overlay budget without requiring any UE-side configuration, the UPF init scripts:

  1. Set ogstun and ogstun2 MTU to 1400 (leaves 10 bytes of headroom after GTP-U over the 1450 overlay).
  2. Apply TCP MSS clamping to 1360 (= 1400 − 20 IP − 20 TCP) on both directions of the FORWARD chain for ogstun/ogstun2, so remote peers negotiate a segment size that survives GTP-U encapsulation regardless of what the UE advertises.

This keeps the overlay MTU (used by Multus/OVS for VXLAN framing) decoupled from the UE-visible MTU (constrained by GTP-U overhead). GTP-U encapsulation is present on the N3 user plane in every topology, so the ogstun MTU reduction and MSS clamping apply whether or not VXLAN is in use. VXLAN adds its 50-byte overhead only on the inter-node hop when edge is enabled; the GTP-U overhead, not VXLAN, is the binding constraint on the UE-visible MTU. Implemented in:

  • ansible/phases/05-5g-core/scripts/upf_cloud_init.sh
  • ansible/phases/05-5g-core/scripts/upf_edge_init.sh

Multus Deployment Details

Why two DaemonSets

K3s and standalone containerd use different host paths for CNI configuration. A single DaemonSet cannot conditionally mount different paths on different nodes. Two DaemonSets solve this:

graph TD
    DS_W["DaemonSet: multus-worker
    nodeSelector: kubernetes.io/hostname=worker
    CNI config dir: /var/lib/rancher/k3s/agent/etc/cni/net.d
    mode: auto (reads k3s primary conflist)"]

    DS_E["DaemonSet: multus-edge
    nodeSelector: kubernetes.io/hostname=edge
    CNI config dir: /etc/cni/net.d
    mode: static conflist + external kubeconfig"]

    W["worker node"]
    E["edge node"]

    DS_W --> W
    DS_E --> E

The same split applies to the OVS DaemonSets (one per node, same reason).

Edge Multus: static conflist

On the edge node, Multus cannot use auto-mode because KubeEdge injects empty KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT environment variables. When Multus auto-generates its conflist, it reads these env vars to build the K8s API URL, and empty values cause Multus to crash.

Solution: install a pre-written static conflist that hard-codes the API server URL and points to an external kubeconfig at /var/lib/multus/multus.kubeconfig.

See known-issues/kubeedge-multus-env-injection.md for the full root cause analysis.

NetworkAttachmentDefinitions (NADs)

Each plane has two NADs on the same OVS bridge, created by phase 04 from the network plan in ansible/group_vars/all.yml:

  • <plane>-net: Whereabouts pool, for attachments that need no fixed address.
  • <plane>-static: static IPAM, for fixed endpoints. The pod gets exactly the address in its ips annotation and nothing else.

The pool NAD:

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: n2-net
  namespace: 5g
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "type": "ovs",
      "bridge": "br-n2",
      "ipam": {
        "type": "whereabouts",
        "range": "10.202.0.0/24",
        "range_start": "10.202.0.100",
        "range_end": "10.202.0.250",
        "exclude": ["10.202.0.100/32"]
      }
    }

The exclude list keeps the fixed endpoints (like the AMF's 10.202.0.100) out of the pool. The AMF attaches to n2-static with "ips": ["10.202.0.100/24"]. A fixed endpoint must not request its address on the pool NAD: Whereabouts would allocate a pool address first and Multus would add the requested one on top, leaving the pool address as the interface's primary. Which endpoints are fixed, and why, is in 5G Interfaces: Fixed Endpoints.

Per-Cell Network Scaling

Every deployment has one shared N2 and one shared N3 bridge. With UERANSIM on, each simulated cell also gets its own N2 and N3 bridge, and its own VXLAN tunnel when an edge node exists:

graph LR
    subgraph Default["Default (1 cell)"]
        BN2["br-n2
        10.202.0.0/24
        shared"]

        BN3["br-n3
        10.203.0.0/24
        shared"]
    end

    subgraph MultiCell["Multi-cell (N cells)"]
        BN2C1["br-n2-cell-1
        10.202.1.0/24
        VNI 1021"]

        BN3C1["br-n3-cell-1
        10.203.1.0/24
        VNI 1031"]

        BN2CN["br-n2-cell-N
        10.202.N.0/24
        VNI 102N"]

        BN3CN["br-n3-cell-N
        10.203.N.0/24
        VNI 103N"]
    end

The cells and their IDs are listed in ansible/phases/06-ueransim-mec/vars/topology.yml, and they are active only while ueransim_enabled is true (ueransim_active_cells). Phase 4 creates the bridges and NADs of the active cells, and phase 5 gives the AMF one interface per active cell. When UERANSIM is off, phase 5 removes the cells' NADs and bridges once the AMF no longer uses them.

Physical RAN Integration

When a physical gNB is connected, the worker has one more OVS bridge, br-ran, on the RAN transport network. The AMF attaches to it with a second interface, n2ran:

graph LR
    GNB["Physical gNB
    RAN transport"]

    BRAN["br-ran
    worker OVS bridge"]

    BN2["br-n2"]

    BN3["br-n3"]

    AMF["AMF pod
    n2, n2ran"]

    UPF["UPF-Cloud pod
    n3"]

    GNB -->|"L2 bridged NIC"| BRAN
    BRAN -->|"N2 / NGAP to n2ran"| AMF
    BRAN -->|"N3, routed by the worker"| BN3
    BN2 --> AMF
    BN3 --> UPF

The gNB's NGAP association terminates on the AMF's n2ran, on the RAN subnet. GTP-U to the UPF is routed by the worker from br-ran to br-n3, since the RAN subnet and the N3 subnet differ. br-ran has no layer-2 link to any plane bridge; make ran checks it ("No Layer-2 Link to the Planes"). The addresses are in 5G Interfaces.

See Physical RAN Integration for the step-by-step setup guide.

Verification Commands

Context: OVS commands run on worker or edge (via vagrant ssh worker). kubectl commands run from master using sudo k3s kubectl.

Check OVS bridges

vagrant ssh worker   # or edge
sudo ovs-vsctl show

Check VXLAN tunnels

sudo ovs-vsctl list interface | grep -A5 vxlan

Check NADs

sudo k3s kubectl get net-attach-def -A

Check pod network interfaces

# Verify AMF has n1 and n2 interfaces
sudo k3s kubectl -n 5g exec deploy/amf -- ip -o -4 addr

# Check Multus network-status annotation
sudo k3s kubectl -n 5g get pod -l app=amf -o json \
  | jq -r '.items[0].metadata.annotations["k8s.v1.cni.cncf.io/network-status"]' | jq .

Check Whereabouts IP pools

sudo k3s kubectl get ippools.whereabouts.cni.cncf.io -A

Troubleshooting

VXLAN connectivity issues

# Test VXLAN UDP port reachability from worker to edge
nc -vzu 192.168.56.12 4789

# Check OVS bridge ports
sudo ovs-vsctl list-ports br-n2

# Check VXLAN tunnel state
sudo ovs-vsctl list interface vxlan-n2

IP assignment issues

# Check Whereabouts IP pool status
sudo k3s kubectl get ippools.whereabouts.cni.cncf.io -A -o yaml

# Verify pod got expected IPs
sudo k3s kubectl -n 5g get pod <pod> \
  -o jsonpath='{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/network-status}' | jq .

Pod stuck in ContainerCreating

Usually a Multus or IPAM issue. Check:

sudo k3s kubectl -n 5g describe pod <pod>   # look at Events section
sudo k3s kubectl -n kube-system logs -l app=multus --tail=50

See runbooks/multus-nad-ipam.md and runbooks/ovs-vxlan-health.md for detailed diagnostics.