Evaluate models, datasets, and agents with the NeMo Evaluator plugin. Use for metric selection, SDK checks, platform jobs, and result retrieval.
Evaluator Plugin
The Plugin CLI entrypoint is uv run nemo evaluator.
Purpose
Use this skill to choose an evaluation interface and metric, validate a minimal
example, submit a NeMo Platform evaluation job, and retrieve its results.
Inputs
Establish these inputs before building an evaluation:
-
Evaluation interface: dataset-driven vs. task-driven agentic evaluation
-
Execution interface: standalone SDK evaluation or a durable NeMo Platform job.
-
Pass/fail dataset examples: the smallest representative pass and failure cases.
-
Metrics: the behaviors to score and the template fields they consume.
-
Target: no target for offline scoring, or the model, agent, runner, or precomputed trials that produce outputs.
