Nvidia AI Factory Operations Agent: how SRE agents work
Key takeaways
- The agent team investigates faults across compute, network, hardware and scheduling layers through one chat interface
- Nvidia starts agents read-only so engineers can verify findings before any automation
- It runs fully on-premises on Nemotron open models, with per-session sandboxes and Kubernetes RBAC limiting what agents can do
- The framework is experimental open source, so APIs and deployment needs may change between releases
When training performance on a large GPU cluster drops, who works out why? Usually a site reliability engineer (SRE), an operations specialist who keeps systems running, and usually by flicking between dashboards for compute, network, hardware and scheduling. Nvidia has built a team of AI agents to do that correlation through one chat window, and it has released the framework as open source.
What the AI Factory Operations Agent actually is
Nvidia calls it the AI Factory Operations Agent. It is a coordinated set of specialised sub-agents, each owning one operational domain, built to run Nvidia's own DGX SuperPOD clusters. An "AI factory" here means a large GPU cluster that produces trained models and tokens, rather than a conventional data centre serving web traffic.
Engineers talk to it in natural language, switch between investigations, and every action is tracked for auditability. The agents talk to each other and to infrastructure through the Model Context Protocol (MCP), a standard way of exposing tools to AI models. Nvidia says any organisation can adopt this MCP-based communication between sub-agents.
The framework is built on Nvidia's NemoClaw blueprint and uses Nvidia OpenShell, a runtime that governs what agents may do. Inference runs on locally served Nvidia Nemotron open models, and the system runs across internal DGX H100, DGX B200 and DGX GB300 production clusters.
How it works
Here the chat UI sends each request to NemoClaw, which routes it to the right domain agent.
Nvidia lists six:
- Job Recovery Agent: detects training job failures and goodput degradation, and runs root cause analysis across scheduler, application, hardware and network layers.
- Hardware Agent: triages device-level faults such as PCIe errors, ECC errors, NVMe failures and thermal anomalies, tracing symptoms to a specific component.
- Observability Agent: queries historical Prometheus telemetry and Grafana Loki logs, then builds dashboards dynamically in the web UI.
- Research Agent: searches preloaded documentation, including Mission Control manuals and DGX SuperPOD reference architectures. Teams can upload their own runbooks.
- Kubernetes Agent: gives read-only awareness of pods, services, GPU resources, node status and scheduling.
- Cluster Management Agent: talks to Base Command Manager APIs through diagnostic tools for advisory troubleshooting.
A concrete case: a training job slows down. Rather than an engineer checking the scheduler, then the network, then the node hardware, the Job Recovery Agent examines all those layers and recommends recovery steps once the root cause is confirmed. Nvidia's point is that this cross-layer view finds causes that isolated tools miss.
Operationally, it runs as a Kubernetes control-plane application deployed with Helm. For shell execution, OpenShell creates a per-session Kubernetes sandbox rather than running commands in the main gateway pod. Kubernetes inspection goes through a server-side run_kubectl tool using a pinned kubectl binary and registered read-only credentials. OpenShell itself receives neither kubectl nor those credentials, and Kubernetes service accounts and role-based access control (RBAC) set the boundary.
Who uses it and why
The first user is Nvidia, for its own SuperPOD operations, aiming to cut manual effort for its SREs. The design also targets multi-tenant AI factories, where several teams share one cluster. Scoped agents, tools, sandboxes and Kubernetes permissions mean tenant SREs see only their own namespaces, jobs and nodes, while central operations keep broader visibility. OpenShell enforces this with kernel-level sandbox restrictions, and tenants can have their own agents and dedicated inference hardware.
What the limitations are
The framework is experimental open source. Nvidia states that APIs, configuration and deployment requirements may change between releases, which matters for anyone planning to depend on it.
There is also a hardware question. For organisations running AI factories at scale, Nvidia's preferred model is a dedicated GPU-accelerated control plane, isolated from the production cluster so agents do not consume training GPUs. Smaller deployments can use an RTX PRO Workstation or DGX Spark, or a local or cloud inference endpoint, depending on data sovereignty, cluster size and spare GPU capacity.
Trust is the other constraint. Nvidia is measuring root cause analysis accuracy and SRE agreement, but the post gives no accuracy figures. Its own advice is to start read-only and let engineers inspect tool calls, evidence and conclusions before acting.
What this means in practice
For UK and US infrastructure teams running GPU estates, the useful part is the set of five lessons Nvidia drew from production use:
- Give each agent clear boundaries, because overlapping ownership creates noise rather than signal.
- Stay read-only first.
- Correlate across workload, hardware and network signals.
- Define MCP tools precisely, which improves routing and reduces wrong tool selection.
- Show reasoning transparently.
Those apply to any agentic operations project, not only Nvidia hardware. On the UK side, the framework's on-premises option is relevant to teams with data residency worries.
Where this is heading
Nvidia says it will keep refining the framework by measuring RCA accuracy, tracking SRE agreement and expanding the agent ecosystem. Teams can already plug in their own MCP servers and APIs to fit existing schedulers, observability platforms and cluster managers. The practical watch point is whether read-only investigation earns enough trust to graduate to automated recovery, and whether Nvidia publishes accuracy numbers to back that step.
Frequently asked questions
What is the Nvidia AI Factory Operations Agent?
A framework of specialised AI agents that detect failures and correlate root causes across compute, network, hardware and scheduling layers of a DGX SuperPOD, surfacing recommendations through a chat interface. Nvidia built it for its own site reliability engineers.
Does the agent fix problems automatically?
Not by default. Nvidia says it starts read-only, because accurate investigation builds engineer trust before automation. The Kubernetes Agent is read-only, and the Cluster Management Agent is advisory.
Can it run on-premises?
Yes. The framework is fully on-premises capable and uses locally served Nvidia Nemotron open models, which supports data sovereignty. Other deployments can use a local or cloud inference endpoint.
Is the framework open source?
Yes, as experimental open source on GitHub. Nvidia warns that APIs, configuration and deployment requirements may change between releases. Customers can ask their account manager for access to Nvidia-developed cluster operations agents.