NVIDIA Open-Sources NVKE and NVKR for Managed Kubernetes on GPU Infrastructure

NVIDIA has released 2 open-source projects for running managed Kubernetes on GPU infrastructure: NVIDIA Kubernetes Engine (NVKE) and NVIDIA Kubernetes Recipes (NVKR). Both were recently announced as components of the NVIDIA DSX AI Factory Platform. NVKE is the control plane for a managed Kubernetes service. NVKR builds the OS images, container images and packages those clusters depend on. Both ship under Apache 2.0 and are in public preview.

TL;DR

  • Size / footprint: 3 CPU servers for the management cluster, 3 CPU servers per workload control plane, plus a pool of GPU worker nodes. A local evaluation needs 6 CPUs, 12 GiB RAM, 100 GiB disk and no GPU.
  • Runs on: your own NVIDIA GPU bare-metal servers, provisioned through NVIDIA Infra Controller (NICo). Node images build for amd64 and arm64.
  • Performance: no throughput or provisioning-time benchmarks are published. Quality signal is certification, not speed.
  • Best: Certified Kubernetes AI Platform and Certified Kubernetes Installer, conformance tested on Kubernetes 1.36.
  • Worst: after install, the NVKE API serves plain HTTP on port 30080 with authentication off, and there is no management-plane upgrade or backup procedure.
  • Bottom line: best thing: open, attested, upstream Kubernetes for GPU clouds. Worst thing: preview-grade security and lifecycle gaps; not for production yet.

What is NVIDIA Kubernetes Engine (NVKE)?

NVKE is open-source software that an AI factory installs to run a managed Kubernetes service on its own NVIDIA GPU infrastructure. The repo calls it ‘a self-hosting factory for Kubernetes clusters,’ not a distribution. It exposes a REST API and controllers on a management cluster. Customers get standard clusters and use kubectl. The operator manages them through the NVKE API or the nvkectl CLI.

NVKE uses upstream Kubernetes, kubeadm and Cluster API. It is written in Go and has no runtime dependency on NVKR.

What does NVIDIA Kubernetes Recipes (NVKR) do?

NVKR is a build system with declarative Definitions. It produces executables, container images, Helm charts and node images for amd64 and arm64. ‘Recipes’ define validated combinations of software versions and GPU generations. Hosted workflows attach signatures, SBOMs and provenance to digest-pinned OCI references. NVKR can also build artifacts for other Kubernetes management systems.

How does the NVKE architecture work?

The NVIDIA technical blog walkthrough describes 5 stages:

  1. Bootstrap: nvkectl bootstrap site starts a temporary single-node kind cluster on a bootstrap host.
  2. Management cluster: NVKE requests 3 servers from NICo, which boots them from the node image. The temporary cluster is then deleted.
  3. Machine provisioning: NICo handles bare-metal lifecycle. CAPNICo, a Cluster API provider, connects NICo to NVKE.
  4. Release sets: operators publish a machine image, a cluster release and a node pool release. Customers pick a release by name.
  5. Workload clusters: each customer cluster has its own control plane, etcd and node pools. NVKE installs OVN-Kubernetes and NodeLocal DNSCache.

A node must present an attested SPIFFE identity before it gets join credentials. The README says SPIRE uses TPM for this. Per-cluster etcd encryption is available. An OpenTelemetry-based Helm chart exports logs, metrics and traces over OTLP.

Inside workload clusters, 3 companion projects plug in: AI Cluster Runtime (AICR) for the GPU runtime, NodeWright for node packages and NVSentinel for break-fix automation.

‘+S[cur][0]+’

‘+S[cur][1]+’

‘;ping();}
function stop(){if(timer){clearInterval(timer);timer=null;R.querySelector(‘#nvplay’).textContent=”Auto-play”}}
R.querySelector(‘#nvnext’).onclick=function(){stop();go(cur+1)};
R.querySelector(‘#nvprev’).onclick=function(){stop();go(cur-1)};
R.querySelector(‘#nvplay’).onclick=function(){if(timer){stop();return}this.textContent=”Pause”;timer=setInterval(function(){go(cur+1)},2600)};
go(0);
R.querySelector(‘#nvbuild’).onclick=function(){var ts=R.querySelectorAll(‘#nvpipe .tile’),bar=R.querySelector(‘#nvbar’);ts.forEach(function(t){t.classList.remove(‘lit’)});bar.style.width=”0″;ts.forEach(function(t,j){setTimeout(function(){t.classList.add(‘lit’);bar.style.width=((j+1)*25)+’%’},300+j*650)});};
R.querySelectorAll(‘.li’).forEach(function(l){l.onclick=function(){l.classList.toggle(‘open’);ping()}});
window.addEventListener(‘load’,ping);window.addEventListener(‘resize’,ping);setTimeout(ping,300);
})();

Source link

Leave a Reply

Your email address will not be published. Required fields are marked *