# GitHub Issue #1400: files.replace doesn't edit my file (support extended regex)

## Issue Summary

**Reporter**: @drewp
**Status**: Open
**Labels**: bug, operations
**URL**: https://github.com/pyinfra-dev/pyinfra/issues/1400

### Problem Description

The `files.replace` operation fails to match patterns that require extended regex (ERE) syntax. The user attempted to use the pattern `127.0.0.1:[0-9]+` with the `flags='-E'` argument, expecting it to be passed to both `grep` (for checking if the pattern exists) and `sed` (for performing the replacement).

**User's Code:**
```python
files.replace(
    path=path,
    text=r'127.0.0.1:[0-9]+',
    replace=f'127.0.0.1:{serve_port}',
    flags='-E',
)
```

**Observed Behavior:**
- The operation reports: `noop: string "127.0.0.1:[0-9]+" does not exist in /path/to/file`
- Manual grep confirms the pattern matches when using `-E` flag
- The file is not modified because the fact check (using `grep` without `-E`) fails to find the pattern

### Technical Analysis

The issue is that the `flags` parameter is only passed to `sed`, not to `grep`. Looking at the command executed:

```
grep -e '127.0.0.1:[0-9]+' /path/to/file
```

Without the `-E` flag, `grep` uses Basic Regular Expressions (BRE), where `+` is treated as a literal character, not a quantifier. The pattern requires Extended Regular Expressions (ERE) to interpret `[0-9]+` as "one or more digits".

---

## Root Cause Analysis

### Code Flow

1. **`files.replace` operation** (`/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/operations/files.py`, lines 477-549):
   ```python
   @operation()
   def replace(
       path: str,
       text: str | None = None,
       replace: str | None = None,
       flags: list[str] | None = None,  # <-- flags parameter
       ...
   ):
   ```

2. **Fact retrieval** (lines 529-534):
   ```python
   existing_lines = host.get_fact(
       FindInFile,
       path=path,
       pattern=text,
       interpolate_variables=interpolate_variables,
   )
   ```

   **Problem**: The `flags` parameter is NOT passed to `FindInFile`. The fact uses plain `grep -e` without extended regex support.

3. **sed replacement** (lines 538-546):
   ```python
   if existing_lines is None or existing_lines:
       yield sed_replace(
           path,
           text,
           replace,
           flags=flags,  # <-- flags ARE passed to sed
           ...
       )
   ```

   The `flags` are correctly passed to `sed_replace`, but the operation never reaches this point because the fact check fails.

### FindInFile Fact Implementation

Located in `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/facts/files.py`, lines 423-456:

```python
class FindInFile(FactBase):
    @override
    def command(self, path, pattern, interpolate_variables=False):
        # ...
        return make_formatted_string_command(
            (
                "grep -e {0} {1} 2> /dev/null || "
                "( find {1} -type f > /dev/null && echo {2} || true )"
            ),
            pattern,
            QuoteString(path),
            QuoteString(self.exists_flag),
        )
```

The `grep` command is hardcoded without any support for additional flags like `-E` (extended regex) or `-P` (PCRE).

---

## Proposed Fix

Based on the discussion in the issue comments by maintainer @Fizzadar, the recommended solution is to add an `extended_regex` boolean parameter that applies `-E` to both `grep` and `sed`.

### Option 1: Add `extended_regex` Parameter (Recommended)

This is the cleanest approach as `-E` has the same meaning for both `grep` and `sed`.

#### Changes Required:

**1. Modify `FindInFile` fact** (`/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/facts/files.py`):

```python
class FindInFile(FactBase):
    """
    Checks for the existence of text in a file using grep. Returns a list of matching
    lines if the file exists, and ``None`` if the file does not.
    """

    @override
    def command(self, path, pattern, interpolate_variables=False, extended_regex=False):
        self.exists_flag = "__pyinfra_exists_{0}".format(path)

        if interpolate_variables:
            pattern = '"{0}"'.format(pattern.replace('"', '\\"'))
        else:
            pattern = QuoteString(pattern)

        grep_flags = "-E " if extended_regex else ""

        return make_formatted_string_command(
            (
                "grep {3}-e {0} {1} 2> /dev/null || "
                "( find {1} -type f > /dev/null && echo {2} || true )"
            ),
            pattern,
            QuoteString(path),
            QuoteString(self.exists_flag),
            grep_flags,
        )
```

**2. Modify `files.replace` operation** (`/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/operations/files.py`):

```python
@operation()
def replace(
    path: str,
    text: str | None = None,
    replace: str | None = None,
    flags: list[str] | None = None,
    backup=False,
    interpolate_variables=False,
    extended_regex=False,  # NEW PARAMETER
    match=None,  # deprecated
):
    """
    Replace contents of a file using ``sed``.

    + path: target remote file to edit
    + text: text/regex to match against
    + replace: text to replace with
    + flags: list of flags to pass to sed
    + backup: whether to backup the file (see below)
    + interpolate_variables: whether to interpolate variables in ``replace``
    + extended_regex: use extended regular expressions (adds -E to grep and sed)

    ...
    """
    # ... existing validation code ...

    existing_lines = host.get_fact(
        FindInFile,
        path=path,
        pattern=text,
        interpolate_variables=interpolate_variables,
        extended_regex=extended_regex,  # PASS TO FACT
    )

    # Ensure -E is in flags for sed if extended_regex is True
    if extended_regex:
        if flags is None:
            flags = ["-E"]
        elif "-E" not in flags:
            flags = ["-E"] + list(flags)

    # ... rest of existing code ...
```

**3. Similarly update `files.line` operation** (same file, around line 278):

The `files.line` operation also uses `FindInFile` and `sed_replace`, so it should receive the same `extended_regex` parameter for consistency.

### Option 2: Add `grep_flags` Parameter

As mentioned in the issue comments, an alternative is to add a separate `grep_flags` parameter. However, this is less elegant because:
- Users need to remember to pass `-E` to both `flags` and `grep_flags`
- The `-E` flag has identical semantics for both tools

### Additional Consideration: Global Replace (`g` flag)

User @drewp also asked about the global (`g`) sed flag for `s/foo/bar/g`. This is already supported via the `flags` parameter - users can pass `flags=['g']` to enable global replacement. The `g` flag only applies to `sed`, not `grep`, so no changes are needed for this.

---

## Files Requiring Modification

| File | Change |
|------|--------|
| `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/facts/files.py` | Add `extended_regex` parameter to `FindInFile.command()` |
| `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/operations/files.py` | Add `extended_regex` parameter to `replace()` and `line()` operations |
| `tests/` | Add tests for extended regex functionality |
| `docs/` | Update documentation for new parameter |

---

## Test Cases to Add

1. **Test extended regex matching in `FindInFile` fact:**
   - Pattern with `+` quantifier (e.g., `[0-9]+`)
   - Pattern with `?` quantifier
   - Pattern with `{n,m}` quantifiers
   - Pattern with alternation `(a|b)`

2. **Test `files.replace` with extended regex:**
   - Verify pattern matches with `extended_regex=True`
   - Verify replacement occurs correctly
   - Verify backward compatibility (default `extended_regex=False`)

3. **Test `files.line` with extended regex:**
   - Similar test cases as `files.replace`

---

## Example Usage After Fix

```python
from pyinfra.operations import files

# Replace IP:port patterns using extended regex
files.replace(
    path="/home/user/.config/app/config.xml",
    text=r'127\.0\.0\.1:[0-9]+',
    replace='127.0.0.1:8080',
    extended_regex=True,
)

# With global replacement
files.replace(
    path="/etc/hosts",
    text=r'192\.168\.[0-9]+\.[0-9]+',
    replace='10.0.0.1',
    extended_regex=True,
    flags=['g'],  # Replace all occurrences on each line
)
```

---

## Related Issues

- This is a documented limitation, not a regression
- No other related issues found in the repository

---

## Timeline

- **2025-07-16**: Issue reported by @drewp
- **2025-08-06**: Maintainer @Fizzadar confirmed the issue and suggested `extended_regex` parameter
- **2025-08-06**: @drewp agreed with the approach
