# TeDS Tutorial — Test-Driven Schema Development

Learn TeDS through hands-on examples. This tutorial builds from basic verification to advanced schema testing patterns using real examples from the project.

## 1. Prerequisites

- Python 3.10+

- Basic understanding of JSON Schema

- Familiarity with YAML

## 2. Setup

### 2.1. Standard Installation (Recommended)

    # Install from PyPI
    pip install teds

    # Verify installation
    teds --version

Expected output: `teds X.Y.Z (spec supported: 1.0-1.0; recommended: 1.0)`

### 2.2. Development Installation (Alternative)

    # Clone and setup for development
    git clone https://github.com/yaccob/teds.git
    cd teds
    python3 -m venv .venv && . .venv/bin/activate
    pip install -r requirements.txt -r requirements-dev.txt

    # Verify installation
    ./teds.py --version

## 3. Chapter 1: Your First Test Specification

### 3.1. Understanding the Problem

Consider this simple schema for a user email:

    # user_email.yaml
    type: string
    format: email

How do you know it actually validates emails correctly? Let’s test it.

### 3.2. Create Your First Testspec

Create `user_email.tests.yaml`:

    version: "1.0.0"
    tests:
      user_email.yaml#/:
        valid:
          simple_email:
            description: "Basic valid email"
            payload: "alice@example.com"
          email_with_subdomain:
            description: "Email with subdomain"
            payload: "bob@mail.company.com"
        invalid:
          missing_at:
            description: "Email without @ symbol"
            payload: "alice.example.com"
          missing_domain:
            description: "Email without domain"
            payload: "alice@"

### 3.3. Key-as-Payload: Quick Testing without Payload Field

TeDS offers a powerful shortcut: when the `payload` field is missing, the test case key is automatically parsed as YAML/JSON and used as the payload:

    version: "1.0.0"
    tests:
      user_email.yaml#/:
        valid:
          '"alice@example.com"':    # Key becomes payload: "alice@example.com"
            description: "Valid email from key"
          '"bob@company.com"':      # Key becomes payload: "bob@company.com"
            description: "Another valid email"
        invalid:
          '"alice.example.com"':    # Key becomes payload: "alice.example.com"
            description: "Invalid: missing @"
          '"@example.com"':         # Key becomes payload: "@example.com"
            description: "Invalid: missing user"

Key advantages:

- Compact: Perfect for testing primitive values

- Quick: No need to write `payload:` for simple cases

- Clear: Test key shows exactly what’s being tested

Parsing rules:

- `"42"` → number 42

- `"null"` → null value

- `'"hello"'` → string "hello" (note the nested quotes)

- `'{"name": "test"}'` → object with name property

### 3.4. Run Your First Verification

    teds verify user_email.tests.yaml

Note: Use `./teds.py` instead of `teds` if you’re using the development installation.

You’ll see output like:

    version: 1.0.0
    tests:
      user_email.yaml#/:
        valid:
          simple_email:
            payload: alice@example.com
            result: SUCCESS
          email_with_subdomain:
            payload: bob@mail.company.com
            result: SUCCESS
        invalid:
          missing_at:
            payload: alice.example.com
            result: WARNING
            message: |
              UNEXPECTEDLY VALID
              A validator that *ignores* 'format' accepted this instance...

### 3.5. Understanding the Warning

The `WARNING` tells us that `format: email` isn’t enforced by all validators. Some accept `alice.example.com` as valid, others reject it. This is a real-world issue!

Fix it by tightening the schema:

    # user_email.yaml (improved)
    type: string
    format: email
    pattern: '^[^@]+@[^@]+\.[^@]+$'  # Basic email pattern

Run the test again - the warning should disappear.

## 4. Chapter 2: Generating Tests from Existing Schemas

### 4.1. Working with the Demo Schema

Explore the demo schema:

    cat demo/sample_schemas.yaml

This contains multiple schema definitions with examples. Let’s generate tests from them.

### 4.2. Generate Tests from Schema Examples

The `generate` command creates test cases from schema definitions using two different addressing methods:

#### 4.2.1. JSON Pointer (Default Method)

JSON Pointer uses the `#/path/to/element` format and points to a specific location in the document. TeDS generates tests for the schemas found directly under the specified path.

    # Generate tests for schemas directly under components/schemas
    # This will find and process schema definitions at this level
    teds generate sample_schemas.yaml#/components/schemas

    # Generate tests for properties directly under User schema
    # This processes the properties defined at this level
    teds generate sample_schemas.yaml#/components/schemas/User/properties

How JSON Pointer works:

- Points to exactly one location in the document

- TeDS processes schemas found directly at that location

- No wildcards needed - the tool looks at the direct children of the specified path

| 🔍 Critical Difference: JSON-Pointer vs JSON-Path Reference Behavior JSON-Pointer (`#/path/to/element`) references the children of the specified node: `#/components/schemas/User` → Returns `type`, `properties`, `required` (schema internals) ❌ Problem: You get schema properties, not the schema itself JSON-Path (`$.path.to.element`) references the node directly: `$.components.schemas.User` → Returns the complete User schema object ✅ Correct: You get the actual schema for test generation Key Rule: Use JSON-Path syntax (`$.components.schemas.User`) when you want to reference a schema directly. JSON-Pointer syntax (`#/components/schemas/User`) will give you the schema’s internal properties instead.

#### 4.2.2. JSON Path (Alternative Method)

JSON Path uses CSS-like selector syntax and requires explicit wildcards to select multiple elements. TeDS supports JSON Path through YAML/JSON configuration objects in three ways:

##### Method 1: Configuration Files

    # Create a configuration file
    cat > config.yaml << EOF
    sample_schemas.yaml:
      paths: ["$.components.schemas.*"]
    EOF

    # Generate using @file syntax
    teds generate @config.yaml

##### Method 2: Direct YAML/JSON Arguments

    # Pass configuration directly as YAML/JSON string
    teds generate '{"sample_schemas.yaml": {"paths": ["$.components.schemas.*"]}}'

    # Multiple paths in one schema
    teds generate '{"api.yaml": {"paths": ["$.components.schemas.Product", "$.components.schemas.Order"]}}'

    # With custom target file
    teds generate '{"schema.yaml": {"paths": ["$[\"$defs\"].*"], "target": "custom_tests.yaml"}}'

##### Method 3: Simple List Format

    # Simplified syntax for multiple paths
    teds generate '{"schema.yaml": ["$.components.schemas.*", "$[\"$defs\"].*"]}'

JSON Path Examples:

    # Configuration examples (can be in files or direct arguments)

    # Select ALL schemas under components/schemas
    sample_schemas.yaml:
      paths: ["$.components.schemas.*"]

    # Select specific schemas by name
    api.yaml:
      paths: ["$.components.schemas.Product", "$.components.schemas.Email"]

    # Array indices and wildcards
    complex.yaml:
      paths: ["$[\"$defs\"].*", "$.allOf[0]", "$.items[1]"]
      target: "complex_tests.yaml"

    # Nested selections
    nested.yaml:
      paths: ["$.components.schemas.*.properties.*"]

Key differences:

- JSON Pointer: `#/components/schemas` → finds schemas directly at this location

- JSON Path: `$.components.schemas.*` → requires `*` wildcard to select multiple items

- JSON Path usage: Via YAML/JSON config objects using `@file`, direct strings, or simple lists

#### 4.2.3. Practical Examples

    # GOOD: Generate from schema container - finds schemas directly under this path
    teds generate api_spec.yaml#/components/schemas

    # GOOD: Generate from specific properties - processes properties at this level
    teds generate api_spec.yaml#/components/schemas/User/properties

    # AVOID: Single schema without properties - limited test generation
    # teds generate api_spec.yaml#/components/schemas/User

    # Using configuration file with JSON Path
    echo 'api_spec.yaml: ["$.components.schemas.*"]' > config.yaml
    teds generate @config.yaml

This creates `api_spec.components~1schemas.tests.yaml` with test cases derived from the `examples` in each schema found at the specified location.

### 4.3. Inspect Generated Tests

    cat api_spec.components~1schemas.tests.yaml

Notice:

- Valid cases are created from schema `examples`

- Test cases are marked with `from_examples: true`

- No invalid cases are generated (you add these manually)

### 4.4. Extend with Manual Test Cases

Edit the generated file to add negative cases:

    # Add to existing generated file
    tests:
      api_spec.yaml#/components/schemas/Email:
        valid:
          # ... generated cases here ...
        invalid:
          not_an_email:
            description: "String without email format"
            payload: "not-an-email"
          empty_string:
            description: "Empty string"
            payload: ""

## 5. Chapter 3: Advanced Schema Testing Patterns

### 5.1. Testing Object Schemas with additionalProperties

    # schemas/user.yaml
    type: object
    additionalProperties: false
    required: [id, name, email]
    properties:
      id:
        type: string
        format: uuid
      name:
        type: string
        minLength: 1
      email:
        type: string
        format: email

Test it thoroughly:

    # user.tests.yaml
    version: "1.0.0"
    tests:
      schemas/user.yaml#/:
        valid:
          complete_user:
            description: "User with all required fields"
            payload:
              id: "3fa85f64-5717-4562-b3fc-2c963f66afa6"
              name: "Alice Example"
              email: "alice@example.com"
        invalid:
          missing_email:
            description: "Missing required email field"
            payload:
              id: "3fa85f64-5717-4562-b3fc-2c963f66afa6"
              name: "Alice Example"
          extra_field:
            description: "Additional property not allowed"
            payload:
              id: "3fa85f64-5717-4562-b3fc-2c963f66afa6"
              name: "Alice Example"
              email: "alice@example.com"
              age: 25  # This should be rejected
          invalid_uuid:
            description: "Invalid UUID format"
            payload:
              id: "not-a-uuid"
              name: "Alice Example"
              email: "alice@example.com"

### 5.2. Testing Boundary Conditions

For numeric constraints, test the boundaries:

    # schemas/age.yaml
    type: integer
    minimum: 0
    maximum: 150

    # age.tests.yaml
    version: "1.0.0"
    tests:
      schemas/age.yaml#/:
        valid:
          minimum_age:
            description: "Minimum valid age"
            payload: 0
          maximum_age:
            description: "Maximum valid age"
            payload: 150
          typical_age:
            description: "Typical age"
            payload: 25
        invalid:
          negative_age:
            description: "Below minimum"
            payload: -1
          too_old:
            description: "Above maximum"
            payload: 151
          not_integer:
            description: "Not an integer"
            payload: 25.5

### 5.3. Testing Enums

    # schemas/status.yaml
    type: string
    enum: ["draft", "published", "archived"]

    # status.tests.yaml
    version: "1.0.0"
    tests:
      schemas/status.yaml#/:
        valid:
          draft_status:
            payload: "draft"
          published_status:
            payload: "published"
          archived_status:
            payload: "archived"
        invalid:
          wrong_case:
            description: "Wrong case should be rejected"
            payload: "Draft"
          unknown_status:
            description: "Status not in enum"
            payload: "deleted"
          empty_string:
            description: "Empty string not in enum"
            payload: ""

## 6. Chapter 4: Working with Complex Schemas

### 6.1. Testing oneOf Compositions

    # schemas/contact.yaml
    oneOf:
      - type: object
        required: [email]
        properties:
          email:
            type: string
            format: email
      - type: object
        required: [phone]
        properties:
          phone:
            type: string
            pattern: '^\+[1-9]\d{1,14}$'  # E.164 format

    # contact.tests.yaml
    version: "1.0.0"
    tests:
      schemas/contact.yaml#/:
        valid:
          email_contact:
            description: "Contact with email only"
            payload:
              email: "alice@example.com"
          phone_contact:
            description: "Contact with phone only"
            payload:
              phone: "+1234567890"
        invalid:
          both_fields:
            description: "Both email and phone (should fail oneOf)"
            payload:
              email: "alice@example.com"
              phone: "+1234567890"
          neither_field:
            description: "Neither email nor phone"
            payload:
              name: "Alice"
          invalid_email:
            description: "Invalid email format"
            payload:
              email: "not-an-email"
          invalid_phone:
            description: "Invalid phone format"
            payload:
              phone: "123"  # Too short for E.164

## 7. Chapter 5: Report Generation and CI Integration

### 7.1. Available Report Templates

    # List all available templates
    teds verify tests.yaml --list-templates

    # Available built-in templates:
    # - default.html: Full HTML with syntax highlighting
    # - default.md: Markdown with emoji indicators
    # - default.adoc: AsciiDoc with color coding
    # - summary.html: Compact HTML overview
    # - summary.md: Brief Markdown summary

### 7.2. Generate Professional Reports

    # Generate HTML report
    teds verify sample_tests.yaml --report default.html

    # Generate Markdown report
    teds verify sample_tests.yaml --report default.md

    # Generate AsciiDoc report
    teds verify sample_tests.yaml --report default.adoc

    # Generate compact summary
    teds verify sample_tests.yaml --report summary.md

### 7.3. Custom Output Paths

    # Specify custom output filenames
    teds verify tests.yaml --report default.html=my_report.html
    teds verify tests.yaml --report summary.md=project_summary.md

### 7.4. Understanding Report Content

Reports include:

- Executive Summary: High-level test results with counts

- Schema Coverage Warnings: Schemas missing valid or invalid tests

- Detailed Results: Complete breakdown by schema with YAML payloads

- Color-coded Status: Visual distinction between SUCCESS, WARNING, ERROR

### 7.5. Output Level Filtering

    # Show only errors (CI-friendly)
    teds verify tests.yaml --output-level error

    # Show warnings and errors (default)
    teds verify tests.yaml --output-level warning

    # Show everything including successes
    teds verify tests.yaml --output-level all

### 7.6. CI Integration

    # CI-friendly: only show errors
    teds verify tests/**/*.yaml --output-level error

    # Exit code handling
    if teds verify tests/**/*.yaml --output-level error; then
      echo "All schema tests passed!"
    else
      case $? in
        1) echo "Some tests failed - review ERROR cases" ;;
        2) echo "Hard failure - check configuration/schemas" ;;
      esac
    fi

### 7.7. In-Place Updates

Keep test files clean and normalized:

    # Update test files with results
    teds verify my_tests.yaml --in-place

    # This updates only the 'tests' section, preserving version and comments

## 8. Chapter 6: Advanced Features

### 8.1. Using Multiple Test Files

    # Verify multiple specifications
    teds verify user.tests.yaml product.tests.yaml order.tests.yaml

    # Generate reports for multiple files
    teds verify tests/*.yaml --report default.html

### 8.2. Network Access for Remote Schemas

    # Enable network access for remote $ref resolution
    teds verify spec.yaml --allow-network

    # With custom timeouts
    TEDS_NETWORK_TIMEOUT=10 teds verify spec.yaml --allow-network

### 8.3. Parse Payload: String-to-Object Conversion

Use `parse_payload: true` to parse string payloads as YAML/JSON before validation:

    tests:
      schema.yaml#/User:
        valid:
          complex_from_string:
            description: "User from JSON string"
            parse_payload: true
            payload: '{"id":"123","name":"Alice","email":"alice@example.com"}'

          api_response_format:
            description: "Testing JSON strings within YAML (common in API responses)"
            parse_payload: true
            payload: '{"user": {"profile": {"name": "Bob", "settings": {"theme": "dark"}}}}'

When to use:

- Complex objects as JSON strings (e.g., from API responses)

- Testing JSON strings that come from external systems

- When you need to test string-encoded JSON data

Important: When `parse_payload: true`, the payload must be a string that contains valid YAML/JSON.

### 8.4. Template Variables for Custom Output Paths

Control where generated test files are created using template variables:

    # JSON Pointer syntax with template variables
    teds generate schema.yaml#/components/schemas={base}.{pointer}.custom.yaml

    # JSON Path syntax with template variables
    teds generate '{"schema.yaml": {"paths": ["$.components.schemas.*"], "target": "{base}_{ext}_tests.yaml"}}'

#### 8.4.1. Available Template Variables

File-based variables (work with both JSON Pointer and JSON Path syntax):

- `{file}` - schema.yaml

- `{base}` - schema

- `{ext}` - yaml

- `{dir}` - directory path

Pointer-based variables (work only with JSON Pointer syntax):

- `{pointer}` - components~1schemas (RFC 6901 escaped)

- `{pointer_raw}` - components/schemas (unescaped)

- `{pointer_strict}` - components%2Fschemas (URL encoded)

Why pointer variables don’t work with JSON Path syntax: JSON Path expressions can match multiple locations in a schema (e.g., `$.components.schemas.*` matches multiple schemas), so there’s no single pointer value to use in templates. JSON Pointer syntax references exactly one location, making pointer-based templates meaningful.

    # JSON Pointer: All variables available
    teds generate api.yaml#/components/schemas={dir}/{base}_{pointer}.yaml

    # JSON Path: Only file-based variables available
    teds generate '{"api.yaml": {"paths": ["$.components.schemas.*"], "target": "{dir}/{base}_tests.yaml"}}'

#### 8.4.2. JSON Pointer RFC 6901 Escaping

The `{pointer}` variable automatically applies RFC 6901 escaping for safe filenames:

    # JSON Pointer: #/components/schemas
    # RFC 6901 escaped: components~1schemas
    teds generate api.yaml#/components/schemas
    # Creates: api.components~1schemas.tests.yaml

    # JSON Pointer: #/$defs/Address
    # RFC 6901 escaped: $defs~1Address
    teds generate schema.yaml#/$defs/Address
    # Creates: schema.$defs~1Address.tests.yaml

Why RFC 6901 escaping? Slashes (`/`) are directory separators in file systems, so `components/schemas/User` would create subdirectories. RFC 6901 escaping (`~1` for `/`, `~0` for `~`) ensures the entire pointer becomes part of the filename, keeping all generated files in the same directory while preserving the hierarchical information from the JSON Pointer.

### 8.5. Configuration Files for Complex Generation

Use YAML configuration files for complex generation scenarios:

    # generation-config.yaml
    api_spec.yaml:
      paths: ["$.components.schemas.*", "$.components.responses.*"]
      target: "api_validation.tests.yaml"

    legacy_schema.yaml: ["$.definitions.*"]

    teds generate @generation-config.yaml

### 8.6. Warning System

Tests can include user warnings and TeDS generates system warnings:

    tests:
      schema.yaml#/Email:
        valid:
          deprecated_format:
            payload: "user@company.co.uk"
            warnings: ["This email format will be deprecated in v2.0"]

System warnings appear automatically for format divergence issues (when different validators disagree about `format` constraints).

## 9. Chapter 7: Environment Configuration and Exit Codes

### 9.1. Environment Variables

Configure TeDS behavior via environment variables:

    # Network settings
    export TEDS_NETWORK_TIMEOUT=10        # seconds (default: 5)
    export TEDS_NETWORK_MAX_BYTES=10485760 # bytes (default: 5MB)

    # Use in commands
    teds verify remote_specs.yaml --allow-network

### 9.2. Exit Codes for CI Integration

TeDS uses semantic exit codes for automation:

- 0: All tests passed

- 1: Some tests failed (ERROR results)

- 2: Hard failures (file not found, invalid YAML, etc.)

    # CI script example
    if teds verify tests/*.yaml --output-level error; then
      echo "✅ All schema contracts validated"
    else
      exit_code=$?
      case $exit_code in
        1) echo "❌ Schema validation failures found" ;;
        2) echo "🚨 Configuration or file errors" ;;
      esac
      exit $exit_code
    fi

## 10. Chapter 8: Best Practices and Common Patterns

### 10.1. Test Organization

    project/
    ├── schemas/
    │   ├── user.yaml
    │   ├── product.yaml
    │   └── order.yaml
    ├── tests/
    │   ├── user.tests.yaml
    │   ├── product.tests.yaml
    │   └── order.tests.yaml
    └── docs/
        └── api-validation-report.html

### 10.2. Naming Conventions

- Use descriptive test case names: `valid_email_with_subdomain` vs `test1`

- Include purpose in descriptions: `"Email without @ symbol should be rejected"`

- Group related schemas in the same test file when logical

### 10.3. Version Management

- Always specify `version: "1.0.0"` in testspecs

- Keep testspecs in version control alongside schemas

- Use `--in-place` updates to maintain clean, reviewable diffs

### 10.4. Schema Evolution

When updating schemas:

1. First add new test cases that capture the intended behavior

2. Then update the schema to satisfy the new tests

3. Finally run all tests to ensure no regressions

### 10.5. Common Pitfalls

Format vs Pattern Issues: If you see unexpected `WARNING` or `ERROR` results related to `format` constraints (like `format: email`, `format: date-time`, etc.), this is not a TeDS tool problem. Different JSON Schema validators handle `format` differently - some enforce it strictly, others treat it as advisory. This is a known JSON Schema ecosystem issue. Use `pattern` for strict validation:

    # Weak - format may not be enforced by all validators
    type: string
    format: email

    # Strong - pattern ensures consistent validation across validators
    type: string
    format: email
    pattern: '^[^@]+@[^@]+\.[^@]+$'

additionalProperties: Always be explicit:

    # Ambiguous - might allow extra properties
    type: object
    properties:
      name: {type: string}

    # Clear - extra properties forbidden
    type: object
    additionalProperties: false
    properties:
      name: {type: string}

Boundary Testing: Test edge cases:

    # For minimum: 1, test 0, 1, 2
    # For maximum: 100, test 99, 100, 101
    # For minLength: 3, test "", "ab", "abc", "abcd"

## 11. Chapter 9: Roundtrip Workflow - Edit and Reuse Verification Output

One of TeDS’s powerful features is roundtrip capability: verification output can be edited and reused as input, especially when combined with the `--in-place` flag.

### 11.1. Basic Roundtrip Workflow

    # 1. Run verification and save output
    teds verify my_tests.yaml > verification_results.yaml

    # 2. Edit the results file to add new test cases or modify existing ones
    # (The output format is valid TeDS input format)

    # 3. Use edited results as new input
    teds verify verification_results.yaml --in-place

### 11.2. Practical Example: Expanding Test Coverage

    # Start with basic tests
    teds verify user.tests.yaml --in-place

    # This updates user.tests.yaml with results. Now you can:
    # 1. Add more test cases directly to user.tests.yaml
    # 2. Copy successful patterns to new test sections
    # 3. Incrementally build comprehensive test suites

    # Re-run verification to validate new additions
    teds verify user.tests.yaml --in-place

### 11.3. Advanced Roundtrip Pattern

    # Generate initial tests from schema
    teds generate schemas.yaml#/components/schemas > user_generated.tests.yaml

    # Add manual test cases to generated file
    # (Edit user_generated.tests.yaml to add invalid cases)

    # Verify and clean up with in-place updates
    teds verify user_generated.tests.yaml --in-place

    # The file is now a clean, validated test specification
    # ready for version control and CI integration

### 11.4. Benefits of Roundtrip Workflow

- Iterative Development: Build test suites incrementally

- Clean Output: `--in-place` normalizes formatting and removes temporary fields

- Version Control Friendly: Consistent file format for meaningful diffs

- Team Collaboration: Share test results that others can extend and modify

## 12. Conclusion

You now know how to:

- Create comprehensive test specifications for JSON Schemas

- Generate initial tests from schema examples using JSON Pointer syntax

- Test complex scenarios like oneOf, boundaries, and formats

- Generate professional validation reports

- Integrate TeDS into CI/CD pipelines

- Use roundtrip workflows for iterative test development

- Follow best practices for maintainable schema testing

TeDS helps you catch schema issues early and maintain high-quality API contracts. Use it to build confidence in your schema definitions and create living documentation for your APIs.

## 13. Next Steps

- Explore example directories for more patterns

- Set up TeDS in your CI pipeline

- Contribute test cases for edge cases you discover