CLI Testing Framework

Comprehensive testing framework for claude-force CLI commands with enhanced helpers, fixtures, and utilities.

Table of Contents

Overview
Installation
Quick Start
Core Components
Usage Guide
Best Practices
Examples

Overview

The CLI Testing Framework provides a robust set of tools for testing command-line interfaces:

CLITestCase: Enhanced base test class with comprehensive assertions
CLITestTemplate: Pre-configured test class with temporary project setup
CLIFixtures: Reusable test data and project generators
CLIMockHelpers: Utilities for mocking external dependencies
Assertion Helpers: CLI-specific validation methods

Benefits

✅ Comprehensive - Covers exit codes, output, JSON, errors, and file system ✅ Reusable - Pre-built fixtures and templates reduce boilerplate ✅ Isolated - Mock helpers prevent actual API calls ✅ Readable - Descriptive assertion methods improve test clarity ✅ Maintainable - Centralized utilities simplify updates

Installation

The framework is included in the test suite. No additional installation required.

from tests.cli_test_framework import CLITestCase, CLIFixtures, CLIMockHelpers

Quick Start

Basic Test

from tests.cli_test_framework import CLITestCase

class TestMyCommand(CLITestCase):
    def test_help_command(self):
        result = self.run_cli("--help")
        self.assert_success(result)
        self.assert_in_output(result, "usage:")

Test with Temporary Project

from tests.cli_test_framework import CLITestTemplate, CLIFixtures

class TestListCommand(CLITestTemplate):
    def test_list_agents(self):
        # self.temp_dir and self.claude_dir are automatically set up
        CLIFixtures.create_full_project(self.temp_dir, num_agents=3)

        result = self.run_cli("list", "agents")
        self.assert_success(result)
        self.assert_in_output(result, "Total: 3 agents")

Core Components

CLITestCase

Base test class providing CLI execution and assertion methods.

Running Commands

# Basic command
result = self.run_cli("list", "agents")

# With input (for interactive commands)
result = self.run_cli("init", input_text="myproject\n")

# With environment variables
result = self.run_cli("list", "agents", env={"DEBUG": "1"})

# With timeout
result = self.run_cli("long-command", timeout=60)

Exit Code Assertions

# Assert success (exit code 0)
self.assert_success(result)

# Assert failure (non-zero exit code)
self.assert_failure(result)

# Assert specific exit code
self.assert_exit_code(result, 2)

Output Assertions

# Assert text in stdout
self.assert_in_output(result, "Expected text")

# Assert text in stderr
self.assert_in_output(result, "Error message", check_stderr=True)

# Assert text NOT in output
self.assert_not_in_output(result, "Unexpected text")

# Assert multiple texts present
self.assert_output_contains_all(result, ["text1", "text2", "text3"])

# Assert regex pattern
self.assert_output_matches_regex(result, r"Total: \d+ agents")

JSON Assertions

# Parse and validate JSON output
data = self.assert_json_output(result)

# Assert JSON has specific keys
self.assert_json_has_keys(result, ["name", "version", "agents"])

# Assert specific JSON value
self.assert_json_value(result, "name", "my-project")

Error Assertions

# Assert error message in stderr
self.assert_error_message(result, "API key not found")

# Assert helpful error with keywords
self.assert_helpful_error(result, ["API key", "export", "ANTHROPIC_API_KEY"])

File System Assertions

# Assert file exists
self.assert_file_exists(Path(".claude/claude.json"))

# Assert file doesn't exist
self.assert_file_not_exists(Path(".claude/old.json"))

# Assert directory structure
self.assert_directory_structure(temp_dir, [
    ".claude/claude.json",
    ".claude/agents",
    ".claude/workflows"
])

# Assert valid JSON file
config = self.assert_valid_json_file(Path(".claude/claude.json"))

CLITestTemplate

Pre-configured test class with automatic setup/teardown.

Automatically provides:

self.temp_dir - Temporary project directory
self.claude_dir - .claude subdirectory
self.original_cwd - Original working directory
Automatic cleanup after each test

class TestWithTemplate(CLITestTemplate):
    def test_something(self):
        # temp_dir is ready to use
        CLIFixtures.create_minimal_config(self.claude_dir)
        result = self.run_cli("list", "agents")
        self.assert_success(result)

CLIFixtures

Reusable test data generators.

Create Temporary Project

temp_dir = CLIFixtures.create_temp_project("my-project")

Create Minimal Config

claude_dir = Path(".claude")
config = CLIFixtures.create_minimal_config(
    claude_dir,
    name="test-project",
    version="1.0.0"
)

Create Test Agent

agent_path = CLIFixtures.create_test_agent(
    claude_dir,
    "my-agent",
    domains=["domain1", "domain2"]
)

Create Full Project

# Creates project with agents, workflows, and config
config = CLIFixtures.create_full_project(temp_dir, num_agents=5)

CLIMockHelpers

Utilities for mocking external dependencies.

Mock Anthropic Client

with CLIMockHelpers.mock_anthropic_client():
    result = self.run_cli("run", "agent", "test-agent", "--task", "test")
    # No actual API call made

Mock Environment Variables

with CLIMockHelpers.mock_env_vars(ANTHROPIC_API_KEY="test-key", DEBUG="1"):
    result = self.run_cli("list", "agents")

Remove API Key

with CLIMockHelpers.no_api_key():
    result = self.run_cli("run", "agent", "test")
    self.assert_error_message(result, "API key")

Usage Guide

Testing a New Command

Create test file (e.g., test_my_command.py)
Choose base class:
- Use CLITestCase for simple tests
- Use CLITestTemplate if you need a temp project
Write tests using assertion helpers
Run tests: pytest tests/test_my_command.py

Example:

from tests.cli_test_framework import CLITestTemplate, CLIFixtures

class TestAnalyzeCommand(CLITestTemplate):
    def setUp(self):
        super().setUp()
        # Create project with test data
        CLIFixtures.create_full_project(self.temp_dir, num_agents=3)

    def test_analyze_agents(self):
        result = self.run_cli("analyze", "agents")
        self.assert_success(result)
        self.assert_in_output(result, "Analysis")

    def test_analyze_json_output(self):
        result = self.run_cli("analyze", "agents", "--json")
        data = self.assert_json_output(result)
        self.assertIn("summary", data)

Testing Error Scenarios

class TestErrorHandling(CLITestTemplate):
    def test_missing_required_arg(self):
        result = self.run_cli("command-without-args")
        self.assert_failure(result)
        self.assert_in_output(result, "required", check_stderr=True)

    def test_invalid_input(self):
        CLIFixtures.create_minimal_config(self.claude_dir)
        result = self.run_cli("run", "agent", "nonexistent")
        self.assert_error_message(result, "Agent not found")

    def test_helpful_suggestions(self):
        CLIFixtures.create_minimal_config(self.claude_dir)
        result = self.run_cli("run", "agent", "cod-reviewer")  # typo
        self.assert_helpful_error(result, ["not found", "Did you mean", "code-reviewer"])

Testing JSON Output

class TestJSONOutput(CLITestTemplate):
    def test_agents_json_format(self):
        CLIFixtures.create_full_project(self.temp_dir, num_agents=3)

        result = self.run_cli("list", "agents", "--json")
        self.assert_success(result)

        # Validate JSON structure
        data = self.assert_json_output(result)
        self.assertIsInstance(data, list)
        self.assertEqual(len(data), 3)

        # Check required keys
        self.assert_json_has_keys(result, ["name", "priority", "domains"])

        # Validate specific values
        for agent in data:
            self.assertIn("name", agent)
            self.assertIn("test-agent", agent["name"])

Testing Interactive Commands

class TestInteractiveCommands(CLITestTemplate):
    def test_init_interactive(self):
        # Simulate user input
        user_input = "myproject\nMy Description\nPython, FastAPI\n1\n"

        result = self.run_cli(
            "init",
            str(self.temp_dir),
            "--interactive",
            input_text=user_input
        )

        self.assert_success(result)
        self.assert_in_output(result, "Project name")
        self.assert_file_exists(self.claude_dir / "claude.json")

Testing with Mocks

class TestWithMocks(CLITestTemplate):
    def test_no_api_key_error(self):
        CLIFixtures.create_full_project(self.temp_dir, num_agents=1)

        with CLIMockHelpers.no_api_key():
            result = self.run_cli("run", "agent", "test-agent-1", "--task", "test")
            self.assert_helpful_error(result, [
                "API key",
                "ANTHROPIC_API_KEY",
                "https://console.anthropic.com"
            ])

    def test_with_test_api_key(self):
        CLIFixtures.create_minimal_config(self.claude_dir)

        with CLIMockHelpers.mock_env_vars(ANTHROPIC_API_KEY="sk-test-key"):
            result = self.run_cli("list", "agents")
            self.assert_success(result)

Best Practices

1. Use Appropriate Base Class

# ✅ Good: Simple test, no project needed
class TestHelpCommand(CLITestCase):
    def test_help(self):
        result = self.run_cli("--help")
        self.assert_success(result)

# ✅ Good: Needs temp project
class TestProjectCommands(CLITestTemplate):
    def test_list(self):
        CLIFixtures.create_minimal_config(self.claude_dir)
        result = self.run_cli("list", "agents")
        self.assert_success(result)

# ❌ Avoid: Manual setup when template exists
class TestBad(unittest.TestCase):
    def setUp(self):
        self.temp_dir = tempfile.mkdtemp()  # Use CLITestTemplate instead!

2. Use Descriptive Assertion Methods

# ✅ Good: Clear and specific
self.assert_json_has_keys(result, ["name", "version"])
self.assert_helpful_error(result, ["API key", "required"])

# ❌ Avoid: Generic assertions
self.assertIn("name", json.loads(result.stdout))
self.assertTrue("API key" in result.stderr)

3. Test Both Success and Failure

class TestCommand(CLITestTemplate):
    def test_success_case(self):
        CLIFixtures.create_full_project(self.temp_dir, num_agents=1)
        result = self.run_cli("analyze", "agents")
        self.assert_success(result)

    def test_failure_no_agents(self):
        CLIFixtures.create_minimal_config(self.claude_dir)
        result = self.run_cli("analyze", "agents")
        self.assert_failure(result)
        self.assert_error_message(result, "No agents found")

4. Use Fixtures for Test Data

# ✅ Good: Reusable fixtures
CLIFixtures.create_full_project(self.temp_dir, num_agents=5)

# ❌ Avoid: Manual test data creation in each test
agents_dir = self.claude_dir / "agents"
agents_dir.mkdir()
(agents_dir / "agent1.md").write_text("...")
(agents_dir / "agent2.md").write_text("...")
# ... repetitive code

5. Mock External Dependencies

# ✅ Good: No actual API calls
with CLIMockHelpers.mock_anthropic_client():
    result = self.run_cli("run", "agent", "test", "--task", "test")

# ❌ Avoid: Real API calls in tests
result = self.run_cli("run", "agent", "test", "--task", "test")
# Slow, expensive, requires API key

Examples

See tests/test_cli_framework_examples.py for comprehensive examples including:

Basic command testing
JSON output validation
Error message testing
Interactive command testing
Mock usage
Advanced assertions
Directory structure validation

Running Tests

# Run all CLI tests
pytest tests/integration/

# Run specific test file
pytest tests/test_my_command.py

# Run with verbose output
pytest tests/test_my_command.py -v

# Run specific test
pytest tests/test_my_command.py::TestMyCommand::test_specific

Contributing

When adding new CLI commands:

Create corresponding test file in tests/
Use the CLI testing framework
Test success, failure, and edge cases
Test both normal and JSON output
Test error messages are helpful
Add examples to documentation

Support

For issues or questions:

Check existing tests in tests/integration/
Review examples in tests/test_cli_framework_examples.py
See framework source in tests/cli_test_framework.py

Version: 1.0.0 Last Updated: 2025-11-14 Maintained By: Claude Force Team