Claude Skill Creator: Tested Review
skill-creator is meant to help you write, test and improve other skills. We gave Claude one small job and then tested the skill it wrote.
- Source
- anthropics/skills, folder skills/skill-creator
- Source commit tested
- e4bb5d9
- Tested on
- 2026-11-16 with Claude Code 2.1.325
- Runs
- One run with the skill, one without
Our verdict
With a simple request, the skill produced a more detailed SKILL.md (908 words against 655) with a richer description and an extra rule for reporting omitted commits. When we ran both finished skills on the same 12 commits, both gave correct changelogs. We did not see an advantage in the final result for a task this small.
What it says it does
Create new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.
From the skill's own SKILL.md description, source: anthropics/skills at commit e4bb5d9.
What you need before it works
- Claude Code. The skill folder is large (about 5,200 words in SKILL.md plus scripts, agents, references and an eval viewer).
- Its testing and benchmarking features are interactive and use scripts. We did not run those in this test, see the caveats.
Real tests
Each test ran twice with the same prompt: once with the skill installed, once without. Single run each.
Create a changelog-writer skill
Prompt we gave Claude
Create a new Claude skill called changelog-writer. It should turn a list of git commit messages into a short, user-facing changelog grouped by Added, Changed and Fixed. Put the skill in a folder named changelog-writer in the current directory.
| With skill | Without skill | |
|---|---|---|
| Skill used by Claude | Yes, on its own (skill-creator) | Not installed |
| Effort | 5 agent turns, 41 s, about $0.28 of usage | 3 agent turns, 34 s, about $0.16 of usage |
| Length of SKILL.md | 908 words | 655 words |
| Tested its own result? | No, it said it had not tested it on any prompts | No, it said it had not tried it on real commits |
Both produced a valid SKILL.md with a name, a description that says when to use it, and a worked example.
Does the skill it wrote actually work?
Prompt we gave Claude
Here are the git commits in commits.txt for our next release, version 2.0.0, dated 2026-11-16. Write the changelog. (12 commits, including one labelled "fix:" that really adds a new option, one breaking rename, and several internal commits.)
| With skill | Without skill | |
|---|---|---|
| Skill used | changelog-writer from the with-skill run | changelog-writer from the without-skill run |
| Result | Correct: new option listed under Added, breaking change flagged, 6 internal commits left out, and it told us how many were omitted | Correct: same grouping, same breaking flag, and a note explaining why the mislabelled commit went under Added |
Both outputs passed our checklist. No difference in correctness.
Caveats
- We gave it one short prompt in a non-interactive run. The skill describes a longer loop of test prompts, reviews and benchmarks. We did not run that loop, so we cannot say how well it works.
- One run per setting.
How we installed it for the test
We copied the skill folder into .claude/skills/skill-creator. For the second test, each generated changelog-writer folder was copied into a fresh project the same way.