Awesomejev Skills

Claude Skill Creator: Tested Review

skill-creator is meant to help you write, test and improve other skills. We gave Claude one small job and then tested the skill it wrote.

Source
anthropics/skills, folder skills/skill-creator
Source commit tested
e4bb5d9
Tested on
2026-11-16 with Claude Code 2.1.325
Runs
One run with the skill, one without

Our verdict

With a simple request, the skill produced a more detailed SKILL.md (908 words against 655) with a richer description and an extra rule for reporting omitted commits. When we ran both finished skills on the same 12 commits, both gave correct changelogs. We did not see an advantage in the final result for a task this small.

What it says it does

Create new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.

From the skill's own SKILL.md description, source: anthropics/skills at commit e4bb5d9.

What you need before it works

Real tests

Each test ran twice with the same prompt: once with the skill installed, once without. Single run each.

Create a changelog-writer skill

Prompt we gave Claude

Create a new Claude skill called changelog-writer. It should turn a list of git commit messages into a short, user-facing changelog grouped by Added, Changed and Fixed. Put the skill in a folder named changelog-writer in the current directory.
With skillWithout skill
Skill used by ClaudeYes, on its own (skill-creator)Not installed
Effort5 agent turns, 41 s, about $0.28 of usage3 agent turns, 34 s, about $0.16 of usage
Length of SKILL.md908 words655 words
Tested its own result?No, it said it had not tested it on any promptsNo, it said it had not tried it on real commits

Both produced a valid SKILL.md with a name, a description that says when to use it, and a worked example.

Top of the SKILL.md written with skill-creator
The SKILL.md written with skill-creator (top part).
Top of the SKILL.md written without skill-creator
The SKILL.md written without it (top part).

Does the skill it wrote actually work?

Prompt we gave Claude

Here are the git commits in commits.txt for our next release, version 2.0.0, dated 2026-11-16. Write the changelog. (12 commits, including one labelled "fix:" that really adds a new option, one breaking rename, and several internal commits.)
With skillWithout skill
Skill usedchangelog-writer from the with-skill runchangelog-writer from the without-skill run
ResultCorrect: new option listed under Added, breaking change flagged, 6 internal commits left out, and it told us how many were omittedCorrect: same grouping, same breaking flag, and a note explaining why the mislabelled commit went under Added

Both outputs passed our checklist. No difference in correctness.

Caveats

How we installed it for the test

We copied the skill folder into .claude/skills/skill-creator. For the second test, each generated changelog-writer folder was copied into a fresh project the same way.

More tested skills