Plane Isolation and Observation¶

This document owns the rules that keep each 5G plane's traffic on that plane, and how the testbed is observed without breaking them. The addresses themselves are in 5G Interfaces. How the overlays are built (OVS, VXLAN, Multus) is in Network Topology.

Why it matters: a measurement is only meaningful if the path it measured is the path the architecture describes. A route that works by accident (for example GTP-U downlink leaving the UPF on N6 because a return route is missing) gives correct-looking results over the wrong path. The rules below make such a path impossible, or at least visible, instead of silent.

The three rules¶

1. Planes are separate networks. The worker routes only where 3GPP has a transport.

N1, N2, N3, N4 and the N6 data networks each connect their own NFs and nothing else. The worker VM owns the gateway address of every overlay bridge, which makes it able to route between any two of them; that ability is used only for:

Crossing	Carries
physical RAN ↔ N3	GTP-U between a physical gNB and the UPF
N6c → external network (NAT)	internet breakout for UEs

NGAP from a physical gNB crosses no plane: the AMF has its own interface on the RAN bridge (n2ran). The RAN bridge has no layer-2 link to any plane bridge, so the only path between the RAN and a plane is the routed one above.

UE traffic toward the data networks (for example a UE reaching an N6m app) is routed by the UPF, by the session's DNN and the destination, never by the worker: see 5G Interfaces: Data Networks. Every other crossing between planes is not a transport the architecture has, and is to be dropped.

In particular a UE never reaches the management network (the VM host-only network and the NodePorts on it) or the physical RAN network, and UE addresses never appear outside the plane bridges: traffic to or from the UE pools on the pod network (for example an app answering a UE through eth0 instead of N6m) is a leak too. A UE that needs a service reaches it the way the architecture provides: on N6m if the service is a MEC app, or through N6c and the public front-door like any other client.

The filter is the nftables table inet kelt_planes on the worker, generated by the OVS setup script (phase 04). Its forward hook runs before the iptables rules and only acts on traffic entering or leaving a plane bridge (br-*), so Kubernetes pod traffic is untouched. It is a table of its own because the iptables filter table is rewritten periodically by the k3s network components, which would reset its counters. plane_filter_mode in ansible/group_vars/all.yml selects observe (count every crossing, change nothing) or enforce (drop every crossing not listed above).

Every rule's counters are exported to Prometheus through the node-exporter textfile collector: kelt_plane_crossing_packets_total and kelt_plane_crossing_bytes_total, labelled verdict (allowed, not_allowed) and rule, plus kelt_plane_filter_enforced. A leak shows as increase(kelt_plane_crossing_packets_total{verdict="not_allowed"}[5m]) > 0.

2. Observing never requires routing.

Passive: capture on the overlay bridges from the worker (tcpdump -i br-n3). It sees every frame of the plane and changes nothing, like a tap or a mirror port in a physical network.
Active: the probe starts inside the plane. Either run the tool in the NF (kubectl exec into the UPF, as the dashboard Network Health checks do) or use a probe pod attached to the plane (the netshoot pod in 5g).

A probe pod attached to more than one plane is itself a link between them if it forwards packets. A pod's IPv4 settings start from the host's, and the worker forwards, so probe pods turn IP forwarding off explicitly at start. They take their address from the plane's pool NAD, never a fixed one. The netshoot probe is attached to every plane, so no check needs a route through the worker.

3. Management stays on its own network.

The VM host-only network, the Kubernetes pod network (eth0, Flannel), the SBI between NFs, and metrics endpoints are the management side. They are reached through Kubernetes Services and the pod network, never through an overlay. The dashboard's diagnostics use kubectl exec and SSH to the worker for captures, so none of them needs a route from the management network into a plane.

Addressing follows the same logic¶

An address on a plane is fixed only when a peer must know it before the NF exists (the gNB is configured with the AMF address; Open5GS lists PFCP peers by address). Everything else takes one address from the plane's pool. Each plane therefore has two NADs on the same bridge, <plane>-net (pool) and <plane>-static (fixed endpoints), so every interface carries exactly one address and that address is the one documented. The fixed endpoints and the reason for each are listed in 5G Interfaces: Fixed Endpoints.

Replicas of an NF follow the same rule: the entry point a peer is configured with stays fixed, the replicas behind it take pool addresses.

Routes follow the interface they use: a route over a plane is declared on that plane's NAD, so the pod has it from the moment the interface is attached. The UPF's route back to the physical RAN subnet was once added by its init script, a few seconds after the pod started. In that window the gNB was still sending to the previous UPF's sessions, the new pod answered through its default route on the pod network, and the filter dropped the answers on their way into br-ran. The route now lives on 5g/n3-upf-static.

Current state¶

Part	State
One source for every address (`ansible/group_vars/all.yml`)	done
One address per interface (pool and static NADs)	done
UPF return route to the physical RAN over N3	done, checked by `make ran`
Worker forwards only the crossings in rule 1	done, `enforce`
Crossings counted and exported to Prometheus, every drop sampled in the kernel log	done
MEC apps answer UEs on N6m, not through `eth0`	done, checked by `make e2e` (MEC Data Network)
Probe pod on every plane, forwarding off	done

The filter ran in observe mode first, under real traffic (physical RAN, UE internet, a UE video stream to an N6m app): the only not-allowed crossings it counted came from a bug it exposed (the RAN gateway address left on the bridged NIC, see Physical RAN), and none after the fix, before it was switched to enforce.

Verification¶

The dashboard's Isolation page (Network group) shows the filter as a from/to matrix with the packets of the last 24 h per rule, and the latest logged samples. The same data from a shell: crossings seen by the filter, per rule (packet and byte counters), and a sample of the not-allowed ones with source and destination:

sudo nft list table inet kelt_planes      # on the worker
sudo journalctl -k | grep KELT-PLANES

A probe that tries to cross planes shows the drop, for example from the netshoot pod toward the SMF N4 address through the worker instead of its own N4 interface:

sudo k3s kubectl -n 5g exec deploy/netshoot -- ip route add 10.204.0.100/32 via 10.203.0.1 dev n3
sudo k3s kubectl -n 5g exec deploy/netshoot -- ping -c3 -W1 10.204.0.100   # 100% loss
sudo k3s kubectl -n 5g exec deploy/netshoot -- ip route del 10.204.0.100/32 via 10.203.0.1 dev n3

The packets appear in the not allowed: br-n3 -> br-n4 counter and in the kernel log with the prefix KELT-PLANES enforce br-n3_->_br-n4.

Not-allowed traffic is counted where it was headed: one counter per ordered pair of plane bridges, from <bridge> for a plane toward anything else (the host, the pod network), into a plane for the reverse, and the two UE pool rules. Each of these rules logs its own sample of up to 6 packets a minute, so a burst on one rule never hides the others.

cd tests && make ran

Among the physical RAN checks, two test this document directly:

UPF Downlink Route via N3: the UPF kernel table routes the physical RAN subnet over n3.
No GTP-U on N6 Bridges: while the tunnel is active on br-n3, a capture on every br-n6* finds no GTP-U. The test is skipped, not passed, when no tunnel traffic is present, because an empty capture proves nothing then.