@lushly-dev/afd-testing weekly downloads statistics

@lushly-dev/afd-testing

Testing utilities for Agent-First Development.

Installation

Overview

This package provides utilities for testing AFD commands:

• Validators: Validate command results and definitions • Surface Validation: Cross-command semantic quality analysis (similarity, schema overlap, naming, injection detection) • Test Helpers: Easy command testing with validation • Assertions: Custom assertions for command results • Mock Server: In-memory MCP server for testing • JTBD Scenario Runner: Jobs-to-be-Done scenario testing with YAML files • Fixtures: Pre-seeded test data with inheritance and overrides • Step References: Dynamic references between scenario steps • Scenario Commands: List, evaluate, coverage, create, and suggest scenarios • MCP Agent Integration: Expose commands as MCP tools with agent hints

Subprocess Scenario Execution

ScenarioExecutor can run each step through the installed afd CLI. The server URL and transport are passed to each afd call without changing the developer's saved CLI connection:

The wrapper uses the public CLI contract: positional JSON input plus --format json. Malformed CLI output is reported as a scenario parse error.

Surface Validation (Semantic Quality)

Cross-command analysis that detects semantic collisions, naming ambiguities, schema overlaps, and prompt injection risks. Designed for command sets of 50+ where agents struggle to pick the right tool.

Basic Usage

Input Types

validateCommandSurface() accepts both ZodCommandDefinition[] (from @lushly-dev/afd-server) and CommandDefinition[] (from @lushly-dev/afd-core). Input is auto-detected via duck-typing.

Validation Rules

| Rule | Severity | Description |

| similar-descriptions | Warning | Command pairs with highly similar descriptions (cosine similarity) | | schema-overlap | Warning | Command pairs sharing a high percentage of input fields | | naming-convention | Error | Command names not matching kebab-case domain-action pattern | | naming-collision | Error | Command names that collide when separators are normalized | | missing-category | Info | Commands without a category field | | description-injection | Error | Descriptions containing prompt injection patterns | | description-quality | Warning | Descriptions that are too short or missing action verbs | | orphaned-category | Info | Categories containing only one command | | schema-complexity | Warning/Info | Input schemas too complex for agents (unions, nesting, constraints) | | unresolved-prerequisite | Error | requires entry references a command not in the surface | | circular-prerequisite | Error | Circular dependency cycle in the requires graph | | missing-output-schema | Info | Command does not declare an output schema | | missing-context | Info | Command has no contexts array (when server has configured contexts) |

Options

Suppressions

Suppress findings at the rule level or for specific command pairs:

CLI Integration

afd validate --surface validates the server's commands whatever its toolStrategy: grouped tools are expanded into the commands listed in their _meta.actions, lazy servers are enumerated with afd-discover and afd-detail, and the tools AFD provides itself (afd-call, afd-batch, afd-pipe, afd-discover, afd-detail, afd-help, afd-docs, afd-schema, afd-context-*; AFD_BUILTIN_TOOL_NAMES in @lushly-dev/afd-core) are skipped.

Agent Integration (Phase 3)

MCP server and tools for AI agent integration.

MCP Server

Start an MCP server exposing all scenario commands:

commandHandler runs the commands that scenario-evaluate executes. Every path argument (directory, scenarios, output, and the fixture files that scenarios name) is resolved against cwd; a path that leaves it, including through a symlinked directory, returns PATH_OUTSIDE_WORKSPACE. Tool arguments are checked against each tool's inputSchema (types, enums, required fields, no unknown fields) and invalid input returns VALIDATION_ERROR. Notifications (messages without an id) get no response.

MCP Tools

All scenario commands are exposed as MCP tools:

| Tool | Description |

| scenario-list | List and filter scenarios, including files that failed to parse | | scenario-evaluate | Run scenarios and return a report (failFast, timeout, concurrency, format, output) | | scenario-coverage | Calculate command, error and job coverage | | scenario-create | Generate a scenario file from a template | | scenario-suggest | Heuristic suggestions from keyword and file-name rules (no AI model) |

Agent Hints

All results include _agentHints for AI interpretation:

scenario-suggest

Heuristic scenario suggestions based on context. Each strategy applies keyword and file-name rules; no AI model is called, and each suggestion's confidence is the fixed score of the rule that produced it:

App Adapters (Phase 4)

Adapters enable the framework to work with different AFD applications.

Adapter Interface

Each app provides an adapter implementing:

Using the Registry

Built-in Adapters

| Adapter | App | Description |

| todoAdapter | Todo | The AFD Todo example app | | genericAdapter | Generic | Fallback for unknown apps |

Creating a Custom Adapter

fixture.apply receives context.handler, which returns each command's real CommandResult. Stop at the first failed result: applyFixture fails the fixture on any failed command (naming it and its error code) whether or not the adapter stops, and it returns the adapter's warnings to the caller.

Scenario Commands (Phase 2)

Batch operations and management commands for JTBD scenarios.

scenario-list

List and filter scenarios in a directory.

Files that fail to parse are returned in parseErrors (and as PARSE_ERROR warnings) instead of being dropped. recursive: false searches only the top directory, pattern matches file names (default *.scenario.yaml), and format ('table' | 'json' | 'names') adds a formattedOutput. status: 'passed' or 'failed' returns UNSUPPORTED_FILTER, because scenario-list keeps no run history.

scenario-evaluate…