Engineering Quality as an AI Skill
Engineering quality reviews are one of those things everyone agrees matter and almost no one does consistently. The checklist isn’t the problem - most staff engineers could write it from memory. The problem is sitting down to run it, every quarter, for every service you own.
I’ve been using a Claude Code skill to handle this. You invoke it, answer questions category by category, and get a status report with P0/P1 actions. More importantly, you can run updates in under ten minutes - revisiting only what was previously incomplete or unknown.
What it does
The skill walks through a structured checklist across 14 categories, records where each aspect is measured, and writes a status report to eng-quality-<service>.md. On subsequent runs it loads the existing report and only asks about the gaps.
Invoke it with /eng-quality. The full skill definition is on GitHub.
Modes
Three modes:
- Full review: every category; for a first run or annual audit
- Update: revisit only ❌ and ❓ items from the last report; target under 10 minutes
- Spot check: focus on one or two specific categories
The checklist
For each item the skill records two things: status (Yes / No / Partial / Unknown) and where it lives - a URL, a tool name, or a one-line description. Questions are batched by category so the conversation moves fast.
| Category | Items |
|---|---|
| Access | Source control (e.g. GitLab/GitHub), Code quality tool (e.g. SonarQube), Observability platform (e.g. Grafana) |
| Developer Documentation | Public developer docs, API reference (Postman / Swagger / Playground), Onboarding docs, Internal runbooks, API versioning policy |
| Coding Standards | Static types, Unit tests, Coverage target, TDD practice, Semantic versioning |
| Development Tooling | Pre-commit hooks, Local debugging setup, Dependency updates (e.g. Dependabot), Sonar / quality gating, Git workflow, Changelog |
| Dependencies | Language version, Database version, Web framework, Base container image, Message queues, Queue encryption |
| Planning & Documentation | Issue tracker (e.g. JIRA/Linear), ADRs, PRDs, Internal release notes, External release notes / changelog, Code review process |
| Release Tooling | CI pipeline, Merge approvals, Deployment approvals, Automated regression tests, Rollback strategy, Golden image / immutable artifact |
| Environments | QA environment, Sandbox / staging environment, Production environment, VPC / network isolation |
| Security | Container image scanning (e.g. Trivy), SAST (e.g. Semgrep), OWASP Top 10, Secrets management, Config management, Payload encryption |
| Resilience | Global rate limits, Per-client rate limits, Tunable timeouts, Load testing |
| Monitoring & Alerting | Alerting (PagerDuty / Rootly / OpsGenie), Metrics (e.g. Prometheus), Status page (e.g. InStatus), Error tracking (e.g. Sentry), Distributed tracing, On-call rotation, Post-mortem process, MTTD |
| Analytics | Business metrics dashboard, Engineering metrics dashboard, Unit cost dashboard, Revenue attribution |
| Logging & SLOs | P90 / P99 latency, SLI, SLO, Error budget, Integration tests, Smoke tests |
| Outcomes (DORA) | Deployment frequency, Lead time for changes, Change failure rate, MTTR, MTBF, Cycle time, First time right |
The report
After collecting answers, the skill writes a status report in this format:
# Engineering Quality - <Service Name>
Last updated: <date>
## Summary
<2-3 sentence health assessment: what's strong, what's missing, top 3 priorities>
## Status by Category
### <Category>
| Item | Status | Where / Notes |
|------|--------|---------------|
| ... | ✅ Yes | ... |
| ... | ⚠️ Partial | ... |
| ... | ❌ No | ... |
| ... | ❓ Unknown | ... |
## P0 Actions
Items that are missing or unknown with highest impact:
- [ ] <item> - <why it matters>
## P1 Actions
- [ ] <item>
## Next Review
Suggested: <3 months from now>The update loop is where it pays off
A full review takes time. The update doesn’t.
Load the existing report, go through only the ❌ and ❓ items, update statuses, re-rank the action list. Under ten minutes if you’re prepared. The report is durable: readable by someone who wasn’t in the conversation, useful as a handoff artifact, and there when someone asks “where are we on observability?”
The value isn’t the first audit. It’s that the second one takes ten minutes, and the third one takes five.