SkillEvaluator: test if an agent skill really helps
Added NewCopyable prompt below
Published Last verified Released
Our verdict
Most people add skills to Claude Code or Codex and hope they help. SkillEvaluator turns that into a check you can run. The free first step needs no model key: skillevaluator validate ./my-skill --tiers 1 scores the SKILL.md, runs security scanners and a harness check that says, in seconds, whether Codex, Claude Code, OpenCode, Hermes, OpenClaw and Copilot CLI will load it (for example, a description cut off at 1,024 characters or a folder that agent does not scan). The deeper tiers flag repeated guidance and run live with-and-without comparisons in a Docker sandbox. NVIDIA says it used it on more than 300 of its own verified skills and reports an average Skill Lift of 41 points for Correctness, with per-product results from +2 to +46 and some skills raising token use; those are NVIDIA's numbers. It has about 3,000 GitHub stars, with a trending tracker reporting +873 in one day on 17 Nov, and builder jatingargiitk ran it on the bundled sample skill without a key and got 88.8/100 with five suggested fixes (one was a 312-character description, against a recommended 50 to 150). Limits: NVIDIA labels it experimental with community support only; the keyless tier checks form, not whether the skill helps; live tests need a model provider, a supported agent and Docker, and can cost tokens; we have not seen an independent Tier 3 run yet.
- Who it's for
- Builders who write or install agent skills for Claude Code, Codex, OpenCode, Hermes or OpenClaw, and teams that keep a shared skill library in a repo
- Leverage
- Catch skills your agent silently ignores, trim bloated or overlapping guidance, and keep only the skills that measurably improve task results
- Effort
- Low
- Date added
What it does
SkillEvaluator (NVIDIA/SkillEvaluator, Apache-2.0, Python 3.12 to 3.14) evaluates folders that follow the Agent Skills specification, plus plugins, in three independent tiers. Tier 1 (Validation) checks schema and frontmatter, scores quality, scans for prompt injection, data exfiltration and secrets (it fetches pinned SkillSpector and Gitleaks releases; Semgrep is optional) and runs the harness check across six agents. Tier 2 (Deduplication) uses embeddings to find repeated guidance inside a skill and, with a catalog file, overlap between skills. Tier 3 (Live Evaluation) generates or reuses evaluation cases, runs your agent twice per case in isolated sandboxes through the open-source Harbor framework, once with the skill and once without, and reports Skill Lift plus token use and run time. skillevaluator ci init opens a pull request that adds the checks to GitHub CI, and the project also ships as a skill you can add to your own agent. NVIDIA Developer Program members can use free NVIDIA Build endpoints for the model-backed checks, within their rate limits.
How builders use it
- Install with pip install skillevaluator (or uv tool install skillevaluator) and run skillevaluator harness-check ./my-skill to see which agents load the skill and why any do not.
- Run skillevaluator validate ./my-skill --tiers 1 for the keyless quality and security score, then fix the findings, such as an over-long description.
- Add skillevaluator ci init to your skills repo so every skill change gets the same checks in a pull request.
- For skills that matter, generate cases with create-eval-dataset, review them, and run the Tier 3 comparison on your agent in Docker; keep the skill only if the lift is worth its extra tokens.
What to expect
A skill library you can trust: skills that load in the agents you use, no duplicated guidance, and live evidence for the ones you keep.
Copy this: Install + first checks
# Official source: github.com/NVIDIA/SkillEvaluator (Apache-2.0). Python 3.12-3.14.
pip install skillevaluator # or: uv tool install skillevaluator
# Will each agent load my skill? (no model, no key)
skillevaluator harness-check ./my-skill
# Static quality + security score (no key; fetches SkillSpector and Gitleaks)
skillevaluator validate ./my-skill --tiers 1
# Add the checks to your GitHub repo (opens a setup PR)
skillevaluator ci init
# Live with/without comparison (needs a model provider, an agent and Docker; costs tokens)
skillevaluator create-eval-dataset ./my-skill --full
skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode docker
# Or ask your agent:
npx skills add NVIDIA/SkillEvaluator
# then: "Is my skill any good? Run the Tier 1 checks on ./my-skill and fix what fails."Related discoveries
See the newest picks →- ToolSmall Tools, Repos & Plugins
mattpocock/skills: make your coding agent ask first
Matt Pocock's small, composable agent skills for real engineering: grill the plan, write a spec, split tickets, then build with TDD.
NewEffort: LowPrompt#Plugin / extension#Open source
- ToolVibe-Coding & Prompt-to-App
Impeccable: give your coding agent a design vocabulary and an A.I.-slop detector
An Apache-2.0 design skill with 24 commands and a no-LLM detector that flags A.I.-generated UI tells in files, URLs or CI.
Effort: LowPrompt#Plugin / extension#Open source
- WorkflowVibe-Coding & Prompt-to-App
Give your agent a spec-to-ship lifecycle with agent-skills
Install a set of open-source engineering skills so your agent specs, plans, builds, tests, reviews and ships in order.
Effort: LowPrompt#Open source
- ToolSmall Tools, Repos & Plugins
Repomix: pack a repo into one A.I.-ready file
One command turns a local or remote repo into a single file you can hand to any LLM, with token counts.
Effort: LowPrompt#Plugin / extension#MCP
Get the best new A.I. picks in your inbox
The best new agent workflows and tools we’ve vetted, with a clear verdict on each. Short, occasional, useful.