AI engineering: AI agents, skills & prompts

Head-to-head comparison of coding agents (Claude Code, Aider, Codex, etc.) on custom tasks with pass rate, cost, time, and consistency metrics. Use when choosing between coding agents, or when a…

Benchmark CodeGraph retrieval quality on a real codebase by comparing agent behavior with vs without CodeGraph. Use when the user runs /agent-eval or asks to test, benchmark, audit, or validate a…

Teaches AI coding agents to create and maintain promptfoo eval suites. Encodes best practices like deterministic assertions first, file-based test organization, correct environment variable syntax,…

Create and run evaluation suites, watch live benchmark progress, view scorecards, compare model performance, and integrate eval runs with CI workflows from the CLI.

Add or extend a Paperclip full-stack runner E2E workflow, fixture, matcher, or report evidence path for local or Daytona execution.

Use when retrieving from or asking questions against a WeKnora knowledge base via the `weknora` CLI — and especially when unsure whether to use `chat`, `session ask`, or `search chunks` for a given…

This skill should be used when the user asks to "test the harness", "run integration tests", "validate features with real API", "test with real model calls", "run agent loop tests", "verify…

Use when changing local eval capture, gold labels, diversified and tune sets, matching, replay storage, or eval CLI behavior.

Run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware. Use for backend selection, local GPU evals, and choosing between vLLM / Transformers / accelerate. Not…

Fine-tune a Kev decision model (open Jev-style System One model) on a user's own workload and serve it with calibrated probabilities. Interviews the user, finds existing Jev/TypeSafe questions or…

This skill should be used when the user wants to "run an evaluation",

Run, monitor, analyze, and debug LLM evaluations via nemo-evaluator-launcher. Covers running evaluations, checking status and live progress, debugging failed runs, exporting artifacts and logs, and…

Compare extracts from two Tika builds over a corpus to detect regressions

Condensed tika-eval pattern for charset-detector regression hunts ("A picks

Query the tika-eval H2 database directly for counts and joins the canned

Compare MCP tool behavior between target and baseline versions using pre-built and custom stories with diff-based triage.

Searches, saves, and maintains a local document index through a local RAG MCP server. Use when user says "search my docs", "save this page", "read around that chunk", "sync my index", or invokes…

Provides context about the CoStrict evals system structure in this monorepo. Use when tasks mention "evals", "evaluation", "eval runs", "eval exercises", or working with the evals infrastructure.…

Fine-tune public CodonFM Encodon checkpoints on labeled coding-sequence or coding-variant data using LoRA, head-only, or full fine-tuning. Use when a user explicitly asks to fine-tune CodonFM or…

Fine-tune a DPA3 model in DeePMD-kit using the PyTorch backend. Use when the user wants to adapt a pre-trained DPA3 model to a new downstream dataset. Supports fine-tuning from a self-trained DPA3…

Fine-tune a DPA4 model in DeePMD-kit. Use for standard or LoRA fine-tuning from a DPA4/SeZM .pt checkpoint, validation and .pt2 export.

Evaluate Expo skills in this repo end-to-end - trigger accuracy, generated code quality, and runtime screenshots on iOS simulator and Android emulator via Expo Go (web optional). Use when the user…

Create a OpenBitFun Canvas for single-case agent evaluation and trajectory

Load the repository's AutomationBench OpenAPI catalogue and run evaluations against the local simulator.

Implement RAG (Retrieval-Augmented Generation) with Neuron AI including vector stores, embeddings providers, document loaders, retrieval strategies, and metadata filtering. Use this skill whenever…

NVIDIA RAG Blueprint — deploy, configure, troubleshoot, and manage. Handles any RAG action: deploy, install, start, enable, disable, toggle, change, configure, troubleshoot, debug, fix, shutdown,…

Filesystem RAG benchmarks: corpus/, train.json, evaluate_rag.py (RAGAS quality). Not for prod

Performance benchmarking for a deployed NVIDIA RAG Blueprint server: profiling pass + aiperf

Adopt the per-agent eval scorecard for a GAIA hub agent: write the harness→payload adapter, run the eval to produce a REAL scorecard, link + surface it from the agent's README, wire the release…

Inspect an agent repository and optional traces, interview the user, write reviewed Task Specs, build and audit Harbor tasks, and bootstrap reusable project World Knowledge Skills. Use for agent…

INVOKE THIS SKILL when setting up a new project or when asked about package versions, installation, or dependency management for LangChain, LangGraph, LangSmith, or Deep Agents. Covers required…

Create LangChain agents with create_agent, define tools, and use middleware for human-in-the-loop and error handling.

Retroactive audit of an implemented AI phase's evaluation coverage. Checks implementation against the AI-SPEC.md evaluation plan. Scores each eval dimension as COVERED/PARTIAL/MISSING. Produces a…

Designs a structured evaluation strategy for an AI phase. Identifies critical failure modes, selects eval dimensions with rubrics, recommends tooling, and specifies the reference dataset. Writes the…

Eval and improve a skill against golden cases — run the target skill blind in a fresh, context-free subagent on each example input, grade the artifact against the expected outcome, and let the gaps…

Analyzes Guidance evaluation results to identify patterns and causes of low pass rates.

Downloads and analyzes the latest three distinct nightly evaluation runs (Claude Code, Codex CLI, and Jetski CLI)

Audit an LLM eval pipeline and surface problems: missing error analysis,

Entry point for evals. Use when the user asks for help with evals,

Guides evaluation of RAG pipeline retrieval and generation quality.

Integrate Arcjet security into a LangChain JS createAgent using @arcjet/guard — wrap tool() / StructuredTool, put guardMiddleware on createAgent({ middleware }) for MCP / unwrapped tools, and read…

Integrate Arcjet security into a LangGraph Graph API agent using @arcjet/guard — wrap tool() / StructuredTool, wrap ToolNode for unwrapped MCP tools, and read thread_id for correlation. Use when…

Use when marketing numbers look bad and the user needs to know WHY — layered root-cause diagnosis across tracking, delivery, creative, landing page, offer, and audience quality, 5-Whys, regional…

Improves Agent Skills via validate → rule-based eval cases → plugin-eval → prompt evals → bounded edits with held-out gates. Use when tuning skill quality, routing, or adopting Chrome/Microsoft…

Local-first semantic search for your codebase and conversation history. Hybrid vector + BM25 search with cross-encoder reranking, AST-aware chunking for 6 languages, smart chunking for 30+ file…

Use when measuring whether an LLM or agent system actually got better and gating merges on it: golden sets, fixing an inflated LLM-as-judge, scoring RAG (faithfulness, contextual recall) or agent…

This skill should be used when picking or diagnosing a training move (SFT, LoRA, DPO/KTO/ORPO, RFT, GRPO/PPO/RLOO, RLHF), or when the user mentions fine-tuning, post-training, training recipe,…

Fine-tune a DPA3 model in DeePMD-kit using the PyTorch backend. Use when the user wants to adapt a pre-trained DPA3 model to a new downstream dataset. Supports fine-tuning from a self-trained DPA3…

Inspect, query, compare, or explicitly export GitHits agent-eval history in Braintrust using the repository's verified workflow.

Measure how much a cloned voice sounds like the real character, and whether a change really helped. Covers speaker-embedding similarity (ECAPA) against the centroid of held-out real clips, word…

Summarize a contributor's Git history, a provided work log, or both into a concise, review-friendly self-evaluation for quarterly, semi-annual, or promotion cycles.

Audit whether a paper's EVALUATION DESIGN actually measures what it claims and whether its reporting is complete — the validity layer family D (experiment-forensics) cannot reach. Three patterns:…

Design, run, review, or release framework- and vendor-neutral evaluations and

Implements document chunking, embedding generation, vector storage, and retrieval pipelines for Retrieval-Augmented Generation systems. Use when building RAG applications, creating document Q&A…

Audit, score, and compare repositories containing portable Agent Plugins against the official Agent Plugins specification. Use when asked to review a plugin repo, check plugin.json or mcp.json…

Spawns the decision-scorer agent after architect proposes 2+ variants in an ADR. Produces a weighted scoring table and recommended choice saved to docs/decisions/.

LLM evaluation specialist who runs structured eval datasets, computes quality metrics using DeepEval/RAGAS, tracks regression across model versions, and reports to Langfuse for tracing and scoring.

Generic isolated-context runner — executes the analysis spec named in its dispatch and returns the structured output that spec defines. Reusable by any skill (shipped or operator-authored) that…

Help users build robust infrastructure for measuring, monitoring, and iterating on AI product performance using human, code-based, and LLM-as-a-judge methodologies.

Isolated evaluator in the eval loop — scores generator outputs with strict isolation; never sees generator context or chain-of-thought