Chat Customizations Evaluations installs statistics

Chat Customizations Evaluations

An extension for analyzing and improving AI prompt files. Works with SKILL.md, .agent.md, .instructions.md, and .prompt.md files — providing LLM-powered semantic analysis directly in VS Code.

Features

LLM-Powered Analysis (via GitHub Copilot)

• Contradiction Detection — Finds logical, behavioral, and format conflicts • Semantic Ambiguity — Ambiguity analysis with rewrite suggestions • Persona Consistency — Detects conflicting personality traits and tone drift • Cognitive Load Assessment — Warns about overly complex prompts with too many nested conditions • Semantic Coverage — Identifies gaps in intent handling and missing error paths • Composition Conflict Analysis — Detects conflicts between a prompt and other prompt files it imports via markdown links

Waza Integration

• Create Eval Scaffold — Generates eval files for a skill via waza new eval • Run Evaluation — Executes skill evaluation via waza run --context-dir • Automatic Local Fallback — If waza is not on PATH, commands attempt a local fallback via go run ./cmd/waza when a sibling waza repo is available

Editor Integration

• Editor Title Bar — Analyze Prompt button appears when editing prompt files • Command Palette — Chat Customizations Evaluations: Analyze Prompt command • Problems Panel — All diagnostics appear in the standard VS Code Problems panel with precise line and column locations

Supported File Types

| Pattern | Type |

| SKILL.md | Skill | | *.agent.md | Agent | | *.instructions.md | Instructions | | *.prompt.md | Prompt (VS Code only) |

Usage

1. Open any supported prompt file in VS Code 2. Run Chat Customizations Evaluations: Analyze Prompt from the command palette or click the beaker icon in the editor title bar 3. View results in the Problems panel (Ctrl+Shift+M / Cmd+Shift+M)

LLM analysis requires GitHub Copilot — no API keys needed. Just sign in to GitHub Copilot in VS Code.

You can also trigger analysis from Copilot Chat with the slash command /analyze-prompt.

Commands

| Command | Description |

| Chat Customizations Evaluations: Analyze Prompt | Run full LLM-powered analysis on the active file | | /analyze-prompt | Run analysis for the active customization file and summarize diagnostics in chat | | Chat Customizations Evaluations: Create Waza Eval Scaffold | Create wazaEval.yaml and task files for the active skill | | Chat Customizations Evaluations: Run Waza Evaluation | Run the skill's eval suite | | Chat Customizations Evaluations: Download Waza Binary | Download the latest platform-specific Waza binary and configure the extension to use it | | Chat Customizations Evaluations: Open Analysis and Fix User Guide | Open the analysis and fix workflow guide |

Guides

• Analysis and Fix User Guide — Learn how to run analysis, interpret diagnostics, implement suggestions, and optionally run Waza evaluations. • Waza User Guide — Set up and run Waza evaluations, configure graders, and validate your customizations.

Configuration

| Setting | Default | Description |

| chatCustomizationsEvaluations.enable | true | Enable/disable the extension | | chatCustomizationsEvaluations.trace.server | off | Trace communication between VS Code and the language server | | chatCustomizationsEvaluations.customDiagnostics | [] | Array of custom diagnostic objects with name and description fields | | chatCustomizationsEvaluations.waza.command | waza | Command used to run Waza (for example /usr/local/bin/waza) |

Telemetry

The extension now emits usage telemetry events for activation, analysis commands, diagnostics fixes, and Waza workflows.

• Telemetry is gated by VS Code telemetry settings. • Events only include coarse metadata (for example counts, durations, and success/failure outcomes). • File contents, prompt text, and file paths are not sent in telemetry event payloads.

To export telemetry to your own collector, set these environment variables before launching VS Code:

• CHAT_CUSTOMIZATIONS_EVALUATIONS_TELEMETRY_ENDPOINT: HTTPS endpoint that accepts JSON POST payloads. • CHAT_CUSTOMIZATIONS_EVALUATIONS_TELEMETRY_AUTH_TOKEN: Optional bearer token added as the Authorization header.

License

MIT