What the DST lab runs today, where large P2P experiments stop being reproducible, and a working rig that sizes a run, writes the manifests and reads the results.
A K3s research cluster on a handful of large hosts, configured with Ansible and Rancher Fleet, networked with Kube-OVN and Cilium, observed with VictoriaMetrics, VictoriaLogs and Grafana.
At thousands of peers per host the bottleneck moves to the network layer: CNI limits, neighbour tables and bandwidth that Kubernetes does not schedule. That is where results stop being reproducible.
A rig that plans a run (packing, gossip bandwidth, observability budget), writes hardened manifests, fingerprints the plan and analyses delivery and latency per peer.
| Takeaway | Why it matters for DST |
|---|---|
| Plan before deploying | A 1,000-peer plan at default sizes already needs about 21 machines and 0.6M series. Catching an impossible plan on paper saves a lab day. |
| Bandwidth is the hidden variable | Two gossip-heavy peers on one host change latency results. The lab already builds a bandwidth scheduler plugin; every run should record the placement it got. |
| Every run carries its plan | A fingerprint of the inputs, image tag and lab revision makes a result comparable to the next release. |
| Observability scales with peers | Series grow with peer count times metrics per peer. Dropping unused metrics is cheaper than more storage. |
| Reports in the open | The weekly roadmap PRs are public; regression baselines per release could be too. |
From the public repos and weekly reports. Each row is checkable at the link.
| Layer | What the lab uses | Source |
|---|---|---|
| Cluster | K3s with containerd; static CPU manager core pinning; max-pods 1000; sysctls sized for 40 Gbit/s networking | iftech/vaclab-2 |
| Provisioning | Ansible playbooks and Rancher Fleet GitOps bundles; secrets in ansible-vault | vaclab-2 |
| Network and storage | Kube-OVN with Cilium chained, Hubble, OpenvSwitch offloading, Longhorn | vaclab-2 |
| Policy and access | Kyverno with Policy Reporter; Authentik SSO in front of the lab | vaclab-2 |
| Observability | VictoriaMetrics, VictoriaLogs, Grafana, Grafana Tempo, OpenTelemetry collector | vaclab-local |
| Scheduling | Bandwidth CRD and operator plus scheduler plugins | vaclab-k8s-plugins |
| Experiments | Python toolkit that generates and deploys manifests, scrapes metrics and runs post-run analysis; a workflow_dispatch to run a config | iftech/10ksim |
| Test nodes | nim, go and rust GossipSub nodes for Kubernetes and Shadow | dst-libp2p-test-node |
| When | Milestone |
|---|---|
| Nov 2023 | 35K-node Shadow simulation at low traffic |
| Mar 2024 | About 1,400 Waku nodes per physical host; a CNI issue at 1,200 nodes on an 8-worker cluster |
| Apr 2024 | Move to OpenvSwitch with offloading; first 10K run at 10,847 nodes |
| Dec 2025 | 5,000 nodes deployed in under 8 minutes, stable mesh in about 25 |
| Oct 2026 | Messaging API baselines at 100 and 1,000 nodes |
From the weekly reports in iftech/roadmap.
Other P2P and client testing infrastructure, and how DST differs.
| Tool or team | What it does | DST in comparison |
|---|---|---|
| Testground | P2P test plans on Docker and Kubernetes with network shaping; last activity 2024 | Dormant; DST covers the same ground on its own lab |
| libp2p unified-testing | Interop, perf and hole-punch tests across implementations | Small-n correctness; DST runs 1K to 10K scale |
| Kurtosis, ethpandaops | Packaged private Ethereum devnets and Helm charts | Reproducible devnets as code; DST moved from Kurtosis to raw Kubernetes for scale |
| Hive | Ethereum end-to-end conformance | Conformance; DST does scaling and regression |
| Shadow | Discrete-event simulator running real binaries | Used by DST for determinism and very large N; Kubernetes for realism |
| Chaos Mesh, Attacknet | Fault injection on Kubernetes and devnets | A natural add next to the churn and partition analyzers |
| Client teams (Lighthouse, Lodestar) | In-repo network simulators | Self-testing per client; DST is a shared service across projects |
The constraints the lab has already met, and what each needs from the platform.
| Constraint | What happens | Platform answer |
|---|---|---|
| Bandwidth is not a scheduled resource | Gossip-heavy peers pile onto one host; latency reflects the NIC more than the protocol | Bandwidth requests honoured at placement (the lab's plugin), and the plan checks NIC share before a run |
| Pods per node | The kubelet default of 110 wastes large hosts; 1,000 needs a matching pod CIDR and neighbour-table tuning | Lab-wide defaults in Ansible, checked by a preflight |
| CNI at density | A CNI issue appeared at 1,200 nodes; OpenvSwitch offloading moved the limit | Per-release regression runs on the network layer itself |
| Noisy neighbours | CPU contention shows up as protocol latency | Pack to a target utilisation and pin cores for measured pods |
| Series growth | Series scale with peers times metrics per peer | Per-experiment metric allow-lists and retention |
| Concurrent experiments | The queueing and concurrency tasks were suspended in 2025 | Namespaces per run, quotas and a simple queue |
Things I noticed in the public repos. Each is a small pull request.
| Item | Detail |
|---|---|
| Roadmap site address | The roadmap README points to roadmap.vac.co; the site is served at roadmap.vac.dev |
| 10ksim folder paths | The README describes analysis/ and deployment/; the code now lives under src/ and deployment-utilities/ |
| Licence files | vaclab-2 and vaclab-k8s-plugins show an Apache 2.0 badge without a LICENSE file in the repository |
| Helm chart registry | The plugins README installs from ghcr.io/vacp2p after the move to the iftech organisation |
| Careers page | research.logos.co/join-us shows no openings while this role is open |
| JD duty | How I would do it | Detailed in |
|---|---|---|
| Design and optimise Kubernetes clusters for experiments | Lab defaults (maxPods, CIDR, sysctls, CPU pinning) in Ansible with a preflight that fails a run before it starts | §03 |
| Improve scalability and utilisation | Plan every run first: pods per host, NIC share, series budget; target utilisation instead of maximum packing | demo |
| Monitoring, logging, observability | Per-experiment dashboards and metric allow-lists in VictoriaMetrics; retention by experiment class | §05 |
| Streamline experiment workflows | One config in, one run record out: plan fingerprint, image tags, lab revision, results bundle | §05 |
| Automate infrastructure and deployment | GitOps for the lab (Fleet), experiment manifests generated from config and validated before apply | demo |
| Troubleshoot and tune workloads | Outlier peers by robust statistics, then node, CNI and placement checks for that peer | demo |
| Reproduce and debug distributed issues | Rerun the fingerprinted plan; diff placement and network metrics between runs | §05 |
| Stage | What happens |
|---|---|
| Plan | Config to plan: pods per host, bandwidth per peer against the NIC, series and log budget. Impossible plans fail here |
| Render | Manifests generated from the plan, checked against Pod Security and lab policy (Kyverno), image tags pinned |
| Queue | A namespace and quota per run so experiments can share the lab |
| Run | Deploy, wait for mesh formation, run the traffic profile, scrape at a fixed interval |
| Record | A run record: plan fingerprint, image digests, lab revision, placement map, metrics window |
| Analyse | Delivery ratio, latency percentiles, outlier peers; compared with the last baseline for the same plan |
| Report | A short weekly note and a baseline table per release |
Read vaclab-2, the plugins and 10ksim end to end. Bring up vaclab-local. Shadow a full run and write down every manual step.
Add a plan and preflight step to the run workflow. Pin the run record format. Take one of the open Q4 tasks, such as Logos Blockchain testnet support in the lab.
Namespaces and quotas for concurrent runs. Per-experiment metric allow-lists. Publish a baseline table for one protocol release.
Role: DevOps Engineer (DST), Logos / IFT Technical Services (public job posting, 2026).
Repositories: vaclab-2 · vaclab-k8s-plugins · vaclab-local · 10ksim · dst-libp2p-test-node · roadmap (roadmap.vac.dev/dst) · research posts at research.logos.co/rlog.
Peers: Testground, libp2p unified-testing, ethpandaops, Hive, Shadow, Chaos Mesh, Attacknet, client-team simulators (their public repositories).
Demo: logos-dst.pages.dev
Independent homework for the Logos DevOps Engineer (DST) role · 2026 · edwardtay.com