# Dyngle

An experimantal, lightweight, easily configurable workflow engine for
automating development, operations, data processing, and content management
tasks.

Technical foundations

- Configuration, task definition, and flow control in YAML
- Operations as system commands using a familiar shell-like syntax
- Expressions and logic in pure Python

## Quick installation (MacOS)

```bash
brew install python@3.11
python3.11 -m pip install pipx
pipx install dyngle
```

## Getting started

Create a file `.dyngle.yml`:

```yaml
dyngle:
  operations:
    hello:
      - echo "Hello world"
```

Run an operation:

```bash
dyngle run hello
```

## Configuration

Dyngle reads configuration from YAML files. Specify the config file location using any of the following (in order of precedence):

1. A `--config` command line option, OR
2. A `DYNGLE_CONFIG` environment variable, OR
3. `.dyngle.yml` in current directory, OR
4. `~/.dyngle.yml` in home directory

## Operations

Operations are defined under `dyngle:` in the configuration. In its simplest form, an Operation is a YAML array defining the Steps, as system commands with space-separated arguments. In that sense, a Dyngle operation looks something akin to a phony Make target, a short Bash script, or a CI/CD job.

As a serious example, consider the `init` operation from the Dyngle configuration delivered with the project's source code.

```yaml
dyngle:
  operations:
    init:
      - rm -rf .venv
      - python3.11 -m venv .venv
      - .venv/bin/pip install --upgrade pip poetry
```

The elements of the YAML array _look_ like lines of Bash, but Dyngle processes them directly as system commands, allowing for template substitution and Python expression evaluation (described below). So shell-specific syntax such as `|`, `>`, and `$VARIABLE` won't work.

### Return Values

Operations can specify a return value using the `return:` key. The returned value can be used by calling code (such as MCP tools or sub-operations) or printed by the `run` command:

```yaml
dyngle:
  operations:
    get-temperature:
      return: temp
      steps:
        - curl -s "https://api.example.com/weather" => weather-data
        - weather-data -> jq -r '.temperature' => temp
```

When you run this operation with `dyngle run get-temperature`, the value of `temp` will be printed to stdout. Dictionary and list values are formatted as YAML, while strings and other simple types are printed as-is.

The `return:` key can reference:
- Data values set via the `=>` operator
- Values from the `values:` section
- Computed values from the `expressions:` section

### Display Options

By default, the `run` command displays each step before executing it. You can control this behavior with the `--display` option:

```bash
# Show steps (default)
dyngle run my-operation --display steps

# Suppress step display
dyngle run my-operation --display none
```

This is useful when you want cleaner output or when step display might interfere with scripted usage.

### Access Control

Operations can specify an `access:` attribute to control their visibility and usage:

- `public` (default): The operation can be run directly via `dyngle run`, exposed through the MCP server, and listed in `dyngle list-operations`
- `private`: The operation can only be called as a sub-operation by other operations

Private operations are useful for creating reusable helper operations that shouldn't be called directly:

```yaml
dyngle:
  operations:
    deploy:
      steps:
        - sub: build
        - sub: upload-artifacts
        - sub: restart-services
    
    build:
      access: private
      steps:
        - npm run build
    
    upload-artifacts:
      access: private
      steps:
        - aws s3 sync ./dist s3://my-bucket/
    
    restart-services:
      access: private
      steps:
        - kubectl rollout restart deployment/my-app
```

In this example:
- `deploy` is public and can be run directly
- `build`, `upload-artifacts`, and `restart-services` are private helpers
- Attempting to run `dyngle run build` will fail with an error
- Private operations don't appear in `dyngle list-operations` output
- Private operations are not exposed as MCP server tools

If no `access:` attribute is specified, operations default to `public`.

## Data and Templates

Dyngle maintains a block of "Live Data" throughout an operation, which is a set of named values (Python `dict`, YAML "mapping"). The values are usually strings but can also be other data types that are valid in both YAML and Python.

The `dyngle run` command feeds the contents of stdin to the Operation as Data, by converting a YAML mapping to named Python values. The values may be substituted into commands or arguments in Steps using double-curly-bracket syntax (`{{` and `}}`) similar to Jinja2.

For example, consider the following configuration:

``` yaml
dyngle:
  operations:
    hello:
      - echo "Hello {{name}}!"
```

Cram some YAML into stdin to try it in your shell:

```bash
echo "name: Francis" | dyngle run hello
```

The output will say:

```text
Hello Francis!
```

### Nested Object Properties

Template syntax supports accessing nested properties in dictionaries using dot notation. This allows you to reference deeply nested data structures directly in your templates:

```yaml
dyngle:
  operations:
    weather-report:
      steps:
        - curl -s "https://api.example.com/weather" => weather
        - echo "Temperature: {{weather.temperature}}"
        - echo "Location: {{weather.location.city}}, {{weather.location.country}}"
```

When using the data input operator (`->`), you can also pass nested values:

```yaml
dyngle:
  operations:
    process-nested:
      steps:
        - curl -s "https://api.example.com/data" => api-response
        - api-response.items.0.name -> echo "First item: {{data}}"
```

This dot notation works anywhere templates are used, including in the `get()` function within expressions (see the Expressions section below).

**Working with Arrays:** Dot notation is designed for accessing named properties in dictionaries, not for array indexing. When you need to work with arrays or more complex data structures, use Python expressions to extract and process the data:

```yaml
dyngle:
  values:
    users:
      - name: Alice
        email: alice@example.com
      - name: Bob
        email: bob@example.com
  operations:
    show-first-user:
      expressions:
        # Get the list, then access elements with Python syntax
        first-user: get('users')[0]
        first-name: get('users')[0]['name']
        first-email: get('users')[0]['email']
        # Use format() to build strings with array elements
        first-user-info: format('User is {name} at {email}').format(name=get('users')[0]['name'], email=get('users')[0]['email'])
        # Or use list comprehensions for more complex operations
        all-names: "[u['name'] for u in get('users')]"
        all-emails: "[u['email'] for u in get('users')]"
      steps:
        - echo "Processing user data"
```

You can then reference these expressions in templates: `{{first-name}}`, `{{first-user-info}}`, etc.

## Expressions

Operations may contain Expressions, written in Python, that can be referenced in Operation Step Templates using the same syntax as for Data. In the case of a naming conflict, an Expression takes precedence over Data with the same name. Expressions can reference names in the Data directly.

Expressions may be defined in either of two ways in the configuration:

1. Global Expressions, under the `dyngle:` mapping, using the `expressions:` key.
2. Local Expressions, within a single Operation, in which case the Steps of the operation require a `steps:` key.

Here's an example of a global Expression

```yaml
dyngle:
  expressions:
    count: len(name)    
  operations:
    say-hello:
      - echo "Hello {{name}}! Your name has {{count}} characters."
```

For completeness, consider the following example using a local Expression for the same purpose.

```yaml
dyngle:
  operations:
    say-hello:
      expressions:
        count: len(name)
      steps:
        - echo "Hello {{name}}! Your name has {{count}} characters."
```

### YAML Structure Syntax

In addition to the traditional string-based Python expression syntax, you can define expressions using native YAML structure (dictionaries and arrays). This provides a more readable and maintainable way to build complex data structures.

```yaml
dyngle:
  operations:
    weather:
      expressions:
        summary:
          location: format("{{location.city}} {{location.state}}")
          temp: format("{{weather-data.temperature}}")
      return: summary
```

The YAML structure syntax is automatically converted to Python dictionaries and lists, with each string value evaluated as a Python expression.

**Supported Structures:**

Dictionaries (YAML mappings):
```yaml
expressions:
  user-info:
    name: get('first-name')
    email: get('email-address')
    age: "int(get('age-string'))"
```

Arrays (YAML sequences):
```yaml
expressions:
  coordinates:
    - get('latitude')
    - get('longitude')
    - get('altitude')
```

Nested structures:
```yaml
expressions:
  api-payload:
    user:
      name: format("{{first-name}} {{last-name}}")
      contact:
        email: get('email')
        phone: get('phone')
    preferences:
      - format("{{pref-1}}")
      - format("{{pref-2}}")
      - format("{{pref-3}}")
```

Mixed with other expression types:
```yaml
expressions:
  # YAML structure for complex data
  response-data:
    timestamp: "datetime.now()"
    items:
      - name: "'Product A'"
        price: "get('price-a') * 1.1"
      - name: "'Product B'"
        price: "get('price-b') * 1.1"
  # Traditional string for simple calculation
  total-price: "sum([item['price'] for item in get('response-data')['items']])"
```

**Important Notes:**

- Each string in a YAML structure is evaluated as a Python expression
- Other literal values (numbers, booleans, None) pass through unchanged
- String literals still require Python string syntax: `"'literal string'"` (or use 1values:`)
- Access nested properties using dot notation in templates: `{{api-payload.user.name}}`
- Access array elements in expressions using Python bracket notation: `get('coordinates')[0]`

Expressions can use a controlled subset of the Python standard library, including:

- Built-in data types such as `str()`
- Essential built-in functions such as `len()`
- The core modules from the `datetime` package (but some methods such as `strftime()` will fail)
- A specialized function called `dtformat()` to perform string formatting operations on a `datetime` object
- A restricted version of `Path()` that only operates within the current working directory
- Various other useful utilities, mostly read-only, such as the `math` module
- A special function called `get()` which retrieves values from the data using the same logic as in templates
- A special function called `format()` which renders a string as a template using the current data context
- An array `args` containing arguments passed to the `dyngle run` command after the Operation name

**NOTE** Some capabilities of the Expression namespace might be limited in the future. The goal is support purely read-only operations within Expressions.

Expressions behave like functions that take no arguments, using the Data as a namespace. So Expressions reference Data directly as local names in Python.

YAML keys can contain hyphens, which are fully supported in Dyngle. To reference a hyphenated key in an Expression, choose:

- Reference the name using underscores instead of hyphens (they are automatically replaced), OR
- Use the built-in special-purpose `get()` function (which can also be used to reference other expressions)

```yaml
dyngle:
  expressions:
    say-hello: >-
        'Hello ' + full_name + '!'
```

... or using the `get()` function, which also allows expressions to essentially call other expressions, using the same underlying data set.

```yaml
dyngle:
  expressions:
    hello: >-
        'Hello ' + get('formal-name') + '!'
    formal-name: >-
        'Ms. ' + full_name
```

Note it's also _possible_ to call other expressions by name as functions, if they only return hard-coded values (i.e. constants).

```yaml
dyngle:
  expressions:
    author-name: Francis Potter
    author-hello: >-
        'Hello ' + author_name()
``` 

Here are some slightly more sophisticated exercises using Expression reference syntax:

```yaml
dyngle:
  operations:
    reference-hyphenated-data-key:
      expressions:
        spaced-name: "' '.join([x for x in first_name])"
        count-name: len(get('first-name'))
        x-name: "'X' * int(get('count-name'))"
      steps:
        - echo "Your name is {{first-name}} with {{count-name}} characters, but I will call you '{{spaced-name}}' or maybe '{{x-name}}'"
    reference-expression-using-function-syntax:
      expressions:
        name: "'George'"
        works: "name()"
        double: "name * 2"
        fails: double()
      steps:
        - echo "It works to call you {{works}}"
        # - echo "I have trouble calling you {{fails}}"
```

### Template Formatting in Expressions

The `format()` function allows you to use template syntax within expressions. This is useful when you want to build complex strings by combining multiple data values or expression results:

```yaml
dyngle:
  values:
    first-name: Alice
    last-name: Smith
  operations:
    greet:
      expressions:
        full-greeting: format('Hello, {{first-name}} {{last-name}}!')
      steps:
        - echo "{{full-greeting}}"
```

The `format()` function takes a template string and renders it using the current data context, including access to data values, expressions, and anything else available in templates. You can combine `format()` with `get()` for more sophisticated string building:

```yaml
dyngle:
  operations:
    weather-report:
      expressions:
        city: get('location.city')
        temp: get('weather.temperature')
        report: format('The temperature in {{location.city}} is {{weather.temperature}} degrees')
      steps:
        - echo "{{report}}"
```

Finally, here's an example using args:

```yaml
dyngle:
  operations:
    name-from-arg:
      expressions:
        name: "args[0]"
      steps:
        - echo "Hello {{name}}"
```

## Passing values between Steps in an Operation

The Steps parser supports two special operators designed to move data between Steps in an explicit way.

- The data assignment operator (`=>`) assigns the contents of stdout from the command to an element in the data
- The data input operator (`->`) assigns the value of an element in the data (or an evaluated expression) to stdin for the command

The operators must appear in order in the step and must be isolated with whitespace, i.e.

```
<input-variable-name> -> <command and arguments> => <output-variable-name>
```

Here we get into more useful functionality, where commands can be strung together in meaningful ways without the need for Bash.

```yaml
dyngle:
  operations:
    weather:
      - curl -s "https://api.open-meteo.com/v1/forecast?latitude=52.52&longitude=13.41&current_weather=true" => weather-data
      - weather-data -> jq -j '.current_weather.temperature' => temperature
      - echo "It's {{temperature}} degrees out there!"
```

If names overlap, data items populated using the data assignment operator take precedence over expressions and data in the original input from the beginning of the Operation.

### Stdout Display Behavior

Dyngle operations have two distinct modes that control stdout display behavior:

**Script Mode (operations without `return:`):**
Operations that don't specify a `return:` key behave like shell scripts, displaying command output naturally:

```yaml
dyngle:
  operations:
    build:
      - echo "Starting build..."
      - npm install
      - npm run build
      - echo "Build complete!"
```

All command stdout is visible, making these operations ideal for builds, deployments, and other tasks where you want to see what's happening.

**Function Mode (operations with `return:`):**
Operations that specify a `return:` key behave like functions, suppressing stdout to produce clean, parseable output:

```yaml
dyngle:
  operations:
    get-temperature:
      return: temp
      steps:
        - echo "Fetching weather..."  # stdout suppressed
        - curl -s "https://api.example.com/weather" => weather-data
        - weather-data -> jq -r '.temperature' => temp
```

When you run this operation, only the return value is displayed (as a simple value, or as YAML for dictionaries and lists). This makes function-mode operations ideal for data queries, transformations, and other operations meant to produce specific outputs.

**Important notes:**
- **stderr is always displayed** in both modes, so error messages and warnings remain visible
- The `=>` operator works in both modes:
  - In script mode: Captures stdout to a variable (suppressing that specific command's output)
  - In function mode: Captures stdout to a variable (already suppressed anyway)
- **Sub-operations inherit behavior:** When an operation calls a sub-operation, the sub-operation inherits the stdout display behavior from the top-level parent operation, ensuring consistent output throughout the call chain

## Sub-operations

Operations can call other operations as steps using the `sub:` key. This allows for composability and reuse of operation logic.

Basic example:

```yaml
dyngle:
  operations:
    greet:
      - echo "Hello!"
    
    greet-twice:
      steps:
        - sub: greet
        - sub: greet
```

Sub-operations can accept arguments using the `args:` key. The called operation can access these via the `args` array in expressions:

```yaml
dyngle:
  operations:
    greet-person:
      expressions:
        person: "args[0]"
      steps:
        - echo "Hello, {{person}}!"
    
    greet-team:
      steps:
        - sub: greet-person
          args: ['Alice']
        - sub: greet-person
          args: ['Bob']
```

### Scoping Rules

Sub-operations follow clear scoping rules that separate **declared values** from **live data**:

**Declared Values are Locally Scoped:**
- Values and expressions declared via `values:` or `expressions:` keys are local to each operation
- A parent operation's declared values are NOT visible to child sub-operations
- A child sub-operation's declared values do NOT leak to the parent operation
- Each operation only sees its own declared values plus global declared values

**Live Data is Globally Shared:**
- Data assigned via the `=>` operator persists across all operations
- Live data populated by a sub-operation IS available to the parent after the sub-operation completes
- This allows operations to communicate results through shared mutable state

Example demonstrating scoping:

```yaml
dyngle:
  values:
    declared-val: global
  
  operations:
    child:
      values:
        declared-val: child-local
      steps:
        - echo {{declared-val}}  # Outputs "child-local"
        - echo "result" => live-data
    
    parent:
      steps:
        - echo {{declared-val}}  # Outputs "global"
        - sub: child
        - echo {{declared-val}}  # Still outputs "global"
        - echo {{live-data}}     # Outputs "result" (persisted from child)
```

## Lifecycle

The lifecycle of an operation is:

1. Load Data if it exists from YAML on stdin (if no tty)
2. Find the named Operation in the configuration
2. Perform template rendering on the first Step, using Data and Expressions
3. Execute the Step in a subprocess, passing in an input value and populating an output value in the Data
4. Continue with the next Step

Note that operations in the config are _not_ full shell lines. They are passed directly to the system.

## Imports

Configuration files can import other configuration files, by providing an entry `imports:` with an array of filepaths. The most obvious example is a Dyngle config in a local directory which imports the user-level configuration.

```yaml
dyngle:
  imports:
    - ~/.dyngle.yml
  expressions:
  operations:
```

In the event of item name conflicts, expressions and operations are loaded from imports in the order specified, so imports lower in the array will override those higher up. The expressions and operations defined in the main file override the imports. Imports are not recursive.

## MCP Server

Dyngle can run as an MCP (Model Context Protocol) server, exposing your configured operations as tools that can be used by AI assistants like Claude. This allows AI assistants to execute your Dyngle operations directly.

### Starting the MCP Server

Run the MCP server using the `mcp` command:

```bash
dyngle mcp
```

By default, this starts a server using the `stdio` transport protocol, which is suitable for integration with clients like Claude Desktop.

The server also supports HTTP and SSE transports:

```bash
# Run with HTTP transport
dyngle mcp --transport http --host 127.0.0.1 --port 8000

# Run with SSE transport
dyngle mcp --transport sse --host 127.0.0.1 --port 8000
```

### How It Works

When you start the MCP server:

1. Each operation defined in your Dyngle configuration becomes an MCP tool
2. AI assistants can discover and call these tools
3. Tools accept:
   - `data`: A dictionary/JSON object (equivalent to data passed via stdin to `dyngle run`)
   - `args`: A list of arguments (equivalent to command-line arguments passed to `dyngle run`)
4. Tools return a JSON response with either:
   - `{"result": <value>}` - On success, where `<value>` is the operation's return value (if specified with `return:` key), or `null` if no return value
   - `{"error": "<message>"}` - On failure, with an error message describing what went wrong

**Note:** Standard output and error from operations flow naturally through the MCP server's logging mechanism rather than being captured in the response. If you need to return data from an operation, use the `return:` key (see the Return Values section above).

### Configuring Claude Desktop

To use your Dyngle operations with Claude Desktop, you need to configure the MCP server in Claude's configuration file.

**macOS:**

Edit or create `~/Library/Application Support/Claude/claude_desktop_config.json`:

```json
{
  "mcpServers": {
    "dyngle": {
      "command": "dyngle",
      "args": ["mcp"]
    }
  }
}
```

**Windows:**

Edit or create `%APPDATA%/Claude/claude_desktop_config.json`:

```json
{
  "mcpServers": {
    "dyngle": {
      "command": "dyngle",
      "args": ["mcp"]
    }
  }
}
```

**Specifying a Configuration File:**

If you want to use a specific Dyngle configuration file (other than `.dyngle.yml` in the current directory or `~/.dyngle.yml`), you can specify it:

```json
{
  "mcpServers": {
    "my-workflows": {
      "command": "dyngle",
      "args": ["--config", "/absolute/path/to/.dyngle.yml", "mcp"]
    }
  }
}
```

**Important Notes:**

- You must use absolute paths in the configuration
- After editing the configuration, fully quit and restart Claude Desktop (not just close the window)
- The MCP tools will appear in Claude's "Search and tools" interface

### Example Usage

Given this Dyngle configuration:

```yaml
dyngle:
  operations:
    greet:
      return: greeting
      steps:
        - echo "Hello {{name}}!" => greeting
    
    weather:
      return: temp
      steps:
        - curl -s "https://api.example.com/weather?city={{city}}" => weather-data
        - weather-data -> jq -r '.temperature' => temp
```

After configuring Claude Desktop with your Dyngle MCP server, you can ask Claude:

- "Use the greet tool with the name Alice" - Returns `{"result": "Hello Alice!"}`
- "Check the weather for San Francisco" - Returns `{"result": "72"}` (or whatever the temperature is)

Claude will execute your Dyngle operations and incorporate the results into its responses. Operations without a `return:` key will return `{"result": null}`.

### Troubleshooting

**Server not showing up in Claude Desktop:**

1. Check that your configuration file has valid JSON syntax
2. Ensure `dyngle` is in your PATH (run `which dyngle` on macOS/Linux or `where dyngle` on Windows)
3. You may need to use the full path to the dyngle executable in the `command` field
4. Fully quit and restart Claude Desktop (use Cmd+Q on macOS or quit from the system tray on Windows)

**Checking logs (macOS):**

Claude Desktop writes MCP-related logs to `~/Library/Logs/Claude/`:

```bash
# View recent logs
tail -n 20 -f ~/Library/Logs/Claude/mcp*.log
```

**Tool execution failures:**

- Check that your Dyngle configuration is valid by running operations manually first
- Ensure any required data or arguments are being passed correctly
- Review the error field in the tool response (the response will be `{"error": "<message>"}` on failure)
- Check the MCP server logs for stdout/stderr output from the operation

## Security

Commands are executed using Python's `subprocess.run()` with arguments split in a shell-like fashion. The shell is not used, which reduces the likelihood of shell injection attacks. However, note that Dyngle is not robust to malicious configuration. Use with caution.

**MCP Server Security:** When running as an MCP server, your operations can be executed by connected clients. Only expose operations that are safe for the intended use case, and be mindful of what operations you make available through the MCP interface.
