Home / Skills / pydantic-evals
pydantic-evals
Fuenfgeld/pydantic-ai-skillsSkill
Test and evaluate AI agents and LLM outputs using code-first evaluation framework with strong typing. Use when the user wants to: (1) Create evaluation datasets with test cases for AI agents, (2) Define evaluators (deterministic, LLM-as-Judge, custom, or span-based), (3) Run evaluations and generate reports, (4) Compare model performance across experiments, (5) Integrate evaluations with Pydantic AI agents, (6) Set up observability with Logfire, (7) Generate test datasets using LLMs, (8) Implement regression testing for AI systems.
Why this score
This repo hosts 2 skills, so its 10 stars are split between them (about 5 each) for scoring.
Popularity score: 56/100 from public GitHub signals only. It shows how widely used and maintained this is, not how well it works. Not included yet: the fake-star check. The security scan is shown separately and doesn't change the score. How we rank
Security
Scanned Oct 20, 2026 at commit baf50b57e870, with Cisco's open-source skill-scanner.
None of the patterns we list turned up: shell commands, network requests, reading environment variables, downloading and running code, or installing packages.
A scan lists patterns in the code. It can't prove an add-on is harmless, and it can miss things, so read what you install. How we scan
Alternatives
Other makers' add-ons that do the same job, next to this one.
| Name | Score | Stars | Context | Updated |
|---|---|---|---|---|
| This onepydantic-evals Fuenfgeld/pydantic-ai-skills · Skill | 56 | ★ 10 | ≈138 always | 18d ago |
| eval-code Calibre-Labs/reforge-ai-evals · Skill Same job: ai engineering | 54 | ★ 76 | ≈26 always | 16d ago |
| write-code-eval ai-evals-course/evals-skills · Skill Same job: ai engineering | 66 | ★ 1.5k | ≈23 always | 1mo ago |
| automationbench-eval bionic-gpt/bionic-gpt · Skill Same job: ai engineering | 73 | ★ 2.4k | ≈31 always | 28d ago |
| harness-eval HKUDS/OpenHarness · Skill Same job: ai engineering | 80 | ★ 16k | ≈74 always | 5mo ago |
| agentic-eval sandbaseai/sandbase-skills · Skill Same job: ai engineering | 52 | ★ 202 | ≈49 always | 1mo ago |