Documentation¶
KELT (Kubernetes-Edge Lightweight Testbed) is a lightweight, reproducible, cloud-native 5G core and cloud-edge testbed. This documentation covers its full lifecycle: from first deployment to day-2 operations, network debugging, physical RAN integration, and contributing new features. It is written for researchers, developers, and operators working with the testbed directly.
Where to Start¶
| Goal | Path |
|---|---|
| Deploy the testbed for the first time | Requirements → Getting Started |
| Understand the system design | Architecture Overview → Virtualization Layers → Network Topology |
| Debug a running testbed | Troubleshooting → relevant Runbook |
| Contribute code or tests | Contributing → Testing Guide |
| Check what is validated vs experimental | Feature Maturity |
Documentation Map¶
Every topic has one document that owns it. Other documents link to the owner and never restate its content, especially concrete values such as IPs, VNIs, ports, and counts. This is what keeps the same fact from drifting in two places. When documenting something, write it at its owner and link from elsewhere.
Entry points, one per audience. They link out, they do not duplicate each other.
| Document | Audience | Owns |
|---|---|---|
| README | Anyone arriving | Pitch, status summary, stack, links out |
| QUICKSTART | Operators, agents | Install, deploy, daily ops, testbed subcommands |
| AGENTS | Agents, new contributors | Hard constraints and where to look first |
| This index | Anyone navigating | Links to every document, grouped by area |
Topic owners. Single source of truth; everyone else links, never restates.
| Topic | Owner |
|---|---|
| Interface matrix (subnets, static IPs, VXLAN VNIs) | architecture/5g-interfaces.md |
| Node/VM topology and IPs | architecture/overview.md |
| System design (layers, network, NFs, positioning) | architecture/ |
| Deployment phases and how to run them | deployment/phases.md |
| External exposure and tunnels | security/external-access.md, deployment/external-tunnel.md |
| Dashboard architecture, access, security summary | dashboard/overview.md |
| Dashboard modules | dashboard/modules.md |
| Dashboard REST/WS endpoints | dashboard/api-reference.md |
| IAM: roles, OIDC clients, per-route matrix | security/iam.md |
| Diagnostics and troubleshooting | operations/troubleshooting.md, runbooks/ |
| Feature maturity | status.md |
| Future direction | roadmap.md |
| Verified gaps and open bugs | gaps.md |
| Platform limitations and workarounds | known-issues/ |
| Per-phase implementation notes | ansible/phases/0X/README.md |
| Test suites and how to run them | development/testing.md |
| Coding standards and contribution workflow | development/contributing.md |
| Operator quick-reference (consolidated IPs, ports, commands) | operations/handbook.md, cheat-sheet only |
Architecture¶
These documents explain how the system is designed. Read them in order; each builds on the previous.
| Document | Description |
|---|---|
| Overview | Node roles, component placement, deployment flow |
| Virtualization Layers | The 5 abstraction layers: host → VMs → K8s → overlay → 5G NFs |
| Network Topology | OVS bridges, VXLAN tunnels, Multus CNI — explained from first principles |
| 5G Interfaces | N1/N2/N3/N4/N6 subnets, static IPs, protocols, and verification commands |
| Subscriber Persistence | MongoDB PVC and subscribers-snapshot ConfigMap — how UE records survive restarts |
| Positioning Adapters | Pluggable positioning engine adapter model behind the CAMARA Location API |
| Edge Apps Platform | Operator-deployed app pods: local registry, deploy-from-image console, dynamic <name>.<base> front-door route |
| NF Platform | Companion 5g-nf-platform repo design: per-NF image builds, patches, versioning (dev plan) |
Deployment¶
Instructions for deploying and configuring the testbed in different environments and modes.
| Document | Description |
|---|---|
| Deployment Phases | What each phase does and how to run them individually |
| Server / NUC Deployment | Deploy on a headless server with optimized profiles and remote access |
| Physical RAN Integration | Connect a real femtocell instead of, or alongside, UERANSIM |
| RAN Mode Switching | Switch between physical and simulated RAN using the dashboard |
Each deployment phase also has an implementation-focused README in its Ansible source directory:
| Phase | Notes |
|---|---|
| Phase 1: Infrastructure | VM provisioning, networking bootstrap |
| Phase 2: Kubernetes | K3s cluster bring-up |
| Phase 3: KubeEdge | CloudCore and EdgeCore configuration |
| Phase 4: Overlay Network | OVS bridges, VXLAN tunnel setup |
| Phase 5: 5G Core | Open5GS NF deployment and configuration |
| Phase 6: UERANSIM + MEC | gNB, UE simulators, MEC workloads |
| Phase 7: Observability | Prometheus, node-exporter, Grafana |
| Phase 8: IAM | Keycloak realm with shared clients for dashboard and CAMARA |
| Phase 9: Dashboard | Dashboard deployment and access |
| Phase 10: Northbound | Northbound platform: CAMARA Location API + positioning engine + demo (optional addon) |
Dashboard¶
The testbed includes an out-of-band operations dashboard (FastAPI + React) that provides real-time visibility and control without depending on the 5G core being healthy.
| Document | Description |
|---|---|
| Overview | Architecture, access URLs, security model, deployment |
| Modules | Every module, grouped by area: cluster, 5G, network, services, settings |
| API Reference | Full REST and WebSocket endpoint listing |
Operations¶
Reference material and diagnostic procedures for running the testbed day-to-day.
| Document | Description |
|---|---|
| Troubleshooting | Common issues and solutions — start here when something is wrong |
| Handbook | Operator cheat-sheet: consolidated IPs, ports, and commands, linking the canonical references |
Runbooks¶
Detailed step-by-step diagnostics for specific subsystems. Use these when the troubleshooting guide points you to a specific area.
| Runbook | Covers |
|---|---|
| NGAP Diagnostics | N2 control plane — gNB ↔ AMF |
| PFCP Diagnostics | N4 control plane — SMF ↔ UPF |
| GTP-U Path | N3 user plane — gNB ↔ UPF data path |
| OVS VXLAN Health | Overlay network infrastructure |
| Multus NAD IPAM | Network attachment and IP allocation |
Security¶
| Document | Description |
|---|---|
| IAM | Keycloak realm: roles, OIDC clients, per-route authorization matrix, user provisioning |
| External Access | Exposing the dashboard, CAMARA, and demo over public domains |
| External Tunnel Setup | Zero-Trust gateway bypass apps for Keycloak, dashboard WebSockets, and dev HMR |
Tools¶
Host-side utilities that complement the testbed.
| Document | Description |
|---|---|
| 5G UE Probe | Manage and benchmark physical UE dongles from the host |
| testbed-config | Interactive CLI for deployment profiles, edge toggle, and RAN configuration |
Development¶
| Document | Description |
|---|---|
| Testing Guide | Run and write the automated test suites |
| Contributing | Coding standards, workflow, and PR guidelines |
The test suite is documented separately: tests/README.md.
Known Issues¶
Platform-specific limitations with documented workarounds. These are bugs or constraints that have no upstream fix yet.
| Issue | Description |
|---|---|
| KubeEdge Edge Discovery | No CoreDNS + no ConfigMap sync on edge — workarounds for pod DNS and API access |
| KubeEdge Multus Env Injection | Empty K8s env vars injected by KubeEdge break Multus auto-config |
| KubeEdge ServiceAccount Tokens | Token projection bugs — use automountServiceAccountToken: false |
| UPF-Edge CNI Route Conflict | UPF-Edge stuck in ContainerCreating — open issue, investigation roadmap |
| TCP Performance over 5G DRX | TCP throughput degradation on the radio path from DRX latency variance |
| No Network-Based Positioning | 3GPP/LMF positioning is out of scope on current hardware; position comes from non-3GPP adapters |
| subPath mount creates a directory | An empty-PVC subPath file mount becomes a directory; the writable store uses a whole-dir mount + env redirect instead |
Coverage Tracker¶
gaps.md tracks documentation gaps, incomplete features, and areas planned for future work. Check it before opening an issue or starting a new contribution.
Roadmap¶
roadmap.md lists Near Term items tied to the current codebase and longer-term Planned directions, with positioning context against comparable open tools.