SW-SEARCH(1)

NAME
sw-search - build and query semantic search indexes for SignalWire AI agents

SYNOPSIS
sw-search sources [sources...] [options]
sw-search search index_file query [options]
sw-search validate index_file
sw-search remote host query [options]
sw-search migrate index_file [options]

DESCRIPTION
sw-search is a command-line tool for building semantic search indexes from documents and querying them. It creates .swsearch files that can be used by SignalWire AI agents to provide knowledge-based responses through the native_vector_search skill.

The tool supports multiple document formats (Markdown, text, PDF, Word, etc.), various chunking strategies, and multiple storage backends including SQLite and PostgreSQL with pgvector.

COMMANDS

sw-search sources...
    Build a search index from the specified source files or directories. This is the default command when no subcommand is specified.

sw-search search index_file query
    Search within an existing index file for the given query.

sw-search validate index_file
    Validate the structure and integrity of an existing index file.

sw-search remote host query
    Search via a remote search API server.

sw-search migrate index_file
    Migrate an index between storage backends.

BUILD OPTIONS

  Output Options

--output FILE
    Output file name for the search index (default: sources.swsearch). For pgvector backend, this is the collection name.

--output-dir DIR
    Output directory for results. When used with --output-format json, creates one file per source file.

--output-format FORMAT
    Output format: index (create search index) or json (export chunks as JSON for review). Default is index.

  Backend Options

--backend BACKEND
    Storage backend to use: sqlite (default) or pgvector.

--connection-string STRING
    PostgreSQL connection string for pgvector backend. Format: postgresql://user:pass@host:port/database

--overwrite
    Overwrite existing collection (pgvector backend only).

  Chunking Strategies

--chunking-strategy STRATEGY
    Strategy for splitting documents into chunks. Available strategies:

    •   sentence - Split by sentences, grouping up to N sentences per chunk (default)

    •   sliding - Sliding window with configurable size and overlap

    •   paragraph - Split on paragraph boundaries

    •   page - Split on page boundaries (good for PDFs)

    •   semantic - Group semantically similar sentences

    •   topic - Group by topic changes

    •   qa - Optimized for question-answering use cases

    •   json - Read pre-chunked content from JSON files

    •   markdown - Markdown-aware chunking that preserves headers and detects code blocks

--max-sentences-per-chunk N
    Maximum sentences per chunk for sentence strategy (default: 5).

--chunk-size N
    Chunk size in words for sliding window strategy (default: 50).

--overlap-size N
    Overlap size in words for sliding window strategy (default: 10).

--split-newlines N
    Split on N or more consecutive newlines (for sentence strategy).

--semantic-threshold FLOAT
    Similarity threshold for semantic chunking (default: 0.5).

--topic-threshold FLOAT
    Similarity threshold for topic chunking (default: 0.3).

  Source Options

--file-types TYPES
    Comma-separated file extensions to include when processing directories (default: md,txt,rst).

--exclude PATTERNS
    Comma-separated glob patterns to exclude. Example: "**/test/**,**/__pycache__/**"

--languages CODES
    Comma-separated language codes (default: en).

  Model Options

--model MODEL
    Sentence transformer model name or alias. Aliases:

    •   mini - sentence-transformers/all-MiniLM-L6-v2 (fastest, 384 dims)

    •   base - sentence-transformers/all-mpnet-base-v2 (balanced, 768 dims)

    •   large - Same as base (best quality)

Or specify full model name from Hugging Face.

--index-nlp-backend BACKEND
    NLP backend for document processing: nltk (fast, default) or spacy (better quality, slower).

  Metadata Options

--tags TAGS
    Comma-separated tags to add to all chunks.

--verbose
    Enable verbose output showing processing details.

--validate
    Validate the created index after building.

SEARCH OPTIONS

--count N
    Number of results to return (default: 5).

--tags TAGS
    Filter results by tags.

--json
    Output results as JSON.

--verbose
    Show detailed result information including scores.

REMOTE OPTIONS

--index-name NAME
    Index name on the remote server.

--count N
    Number of results to return.

--verbose
    Show detailed results.

MIGRATE OPTIONS

--info
    Show information about the index without migrating.

--to-pgvector
    Migrate to PostgreSQL pgvector backend.

--connection-string STRING
    PostgreSQL connection string for migration target.

--collection-name NAME
    Collection name for pgvector migration.

EXAMPLES

  Building Indexes
Basic usage with a directory:

    sw-search ./docs

Multiple sources with file type filter:

    sw-search ./docs ./examples README.md --file-types md,txt,py

Custom output file:

    sw-search ./docs --output knowledge.swsearch

  Chunking Strategies
Sentence-based chunking with custom parameters:

    sw-search ./docs \
      --chunking-strategy sentence \
      --max-sentences-per-chunk 10

Sliding window chunking:

    sw-search ./docs \
      --chunking-strategy sliding \
      --chunk-size 100 \
      --overlap-size 20

Markdown-aware chunking (best for documentation with code):

    sw-search ./docs \
      --chunking-strategy markdown \
      --file-types md

Semantic chunking (groups similar content):

    sw-search ./docs \
      --chunking-strategy semantic \
      --semantic-threshold 0.6

  Model Selection
Using different embedding models:

    sw-search ./docs --model mini    # Fastest, good for most cases
    sw-search ./docs --model base    # Better quality, slower
    sw-search ./docs --model sentence-transformers/all-mpnet-base-v2

  PostgreSQL pgvector Backend
Build directly to PostgreSQL:

    sw-search ./docs \
      --backend pgvector \
      --connection-string "postgresql://user:pass@localhost:5432/db" \
      --output docs_collection

Overwrite existing collection:

    sw-search ./docs \
      --backend pgvector \
      --connection-string "postgresql://user:pass@localhost:5432/db" \
      --output docs_collection \
      --overwrite

  Searching
Search within an index:

    sw-search search ./docs.swsearch "how to create an agent"
    sw-search search ./docs.swsearch "API reference" --count 3 --verbose
    sw-search search ./docs.swsearch "configuration" --tags documentation --json

Search in pgvector collection:

    sw-search search docs_collection "how to create an agent" \
      --backend pgvector \
      --connection-string "postgresql://user:pass@localhost/db"

  Remote Search
Search via remote API:

    sw-search remote http://localhost:8001 "how to create an agent" \
      --index-name docs

  Exporting and Importing Chunks
Export chunks to JSON for review:

    sw-search ./docs \
      --output-format json \
      --output all_chunks.json

Export one JSON file per source:

    sw-search ./docs \
      --output-format json \
      --output-dir ./chunks/

Build index from exported JSON:

    sw-search ./chunks/ \
      --chunking-strategy json \
      --file-types json \
      --output final.swsearch

  Migration
Show index information:

    sw-search migrate --info ./docs.swsearch

Migrate to pgvector:

    sw-search migrate ./docs.swsearch --to-pgvector \
      --connection-string "postgresql://user:pass@localhost/db" \
      --collection-name docs_collection

FILE FORMATS

  .swsearch Files
SQLite-based search index files containing:

•   Document chunks with text content

•   Embedding vectors for semantic search

•   Metadata (source file, tags, headers)

•   Index configuration and model information

  Supported Input Formats

•   Markdown (.md) - Full support including code block detection

•   Text (.txt) - Plain text files

•   reStructuredText (.rst) - Documentation format

•   PDF (.pdf) - Requires pdfplumber (install with [search-full])

•   Word (.docx) - Requires python-docx (install with [search-full])

•   Excel (.xlsx) - Requires openpyxl (install with [search-full])

•   PowerPoint (.pptx) - Requires python-pptx (install with [search-full])

•   RTF (.rtf) - Requires striprtf (install with [search-full])

•   JSON (.json) - Pre-chunked content

INSTALLATION
The search functionality requires optional dependencies:

    # Query-only (search existing indexes)
    pip install signalwire-agents[search-queryonly]
    # Full search (build and query)
    pip install signalwire-agents[search]
    # With document processing (PDF, Word, etc.)
    pip install signalwire-agents[search-full]
    # With pgvector support
    pip install signalwire-agents[pgvector]
    # Everything
    pip install signalwire-agents[search-all]

EXIT STATUS

0   Successful execution.

1   An error occurred. Check stderr for details.

ENVIRONMENT

SIGNALWIRE_LOG_MODE
    Controls logging verbosity. Set to "off" to suppress logs.

SEE ALSO
swaig-test(1), sw-agent-init(1)

SignalWire AI Agents SDK documentation: https://github.com/signalwire/signalwire-agents

AUTHORS
SignalWire Team <info@signalwire.com>

COPYRIGHT
Copyright (c) 2025 SignalWire. Licensed under the MIT License.