# GitHub Issue #533: sed replace quoting (server.crontab fails to update crontab entry sometimes)

## Issue Summary

**Reporter**: @taranlu-houzz
**Status**: Open
**Labels**: bug, operations
**URL**: https://github.com/pyinfra-dev/pyinfra/issues/533
**Assignee**: @Fizzadar

### Problem Description

The `server.crontab` (which delegates to `crontab.crontab`) operation fails to properly update existing crontab entries when the command contains special characters, particularly:

1. **Single quotes** (e.g., `echo 'banana'`)
2. **Ampersands** (e.g., `echo 'true' && echo 'true again'`)

When modifying the `command` parameter for an existing crontab entry (identified by `cron_name`), the operation reports success but the crontab is not actually updated, or worse, the crontab entry becomes corrupted.

### Reproduction Steps

1. Clear existing crontab: `crontab -r`
2. Run deploy with command containing quotes:
   ```python
   server.crontab(
       name="Install test cron task.",
       command="echo 'banana'",
       cron_name="My special task",
       minute="*/10",
   )
   ```
3. Change the command and run again:
   ```python
   server.crontab(
       command="echo 'yummy'",  # Changed
       cron_name="My special task",
       minute="*/10",
   )
   ```
4. **Result**: The crontab is NOT updated, despite the operation reporting success.

### Ampersand Issue (reported by @fbarthez)

When commands contain `&&`, the sed replacement causes corruption:

- **First run**: `45 2 * * * echo 'true' && echo 'true again'`
- **Second run result**: `45 2 * * * echo 'true' 45 2 * * * echo 'true' && echo 'true again'45 2 * * * echo 'true' && echo 'true again' echo 'true again'`

This is because `&` in sed's replacement string has special meaning - it refers to the matched pattern.

---

## Root Cause Analysis

The issue is in the `sed_replace` function in `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/operations/util/files.py`.

### Code Flow

1. **`crontab.crontab` operation** (`/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/operations/crontab.py`):
   - When updating an existing crontab entry, it calls `sed_replace()` to replace the old crontab line with the new one.
   - Line 164-170:
     ```python
     edit_commands.append(
         sed_replace(
             temp_filename,
             existing_crontab_match,
             new_crontab_line,
             interpolate_variables=interpolate_variables,
         ),
     )
     ```

2. **`sed_replace` function** (`/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/operations/util/files.py`):
   - Lines 77-85:
     ```python
     def sed_replace(
         filename: str,
         line: str,
         replace: str,
         flags: list[str] | None = None,
         backup=False,
         interpolate_variables=False,
     ) -> StringCommand:
         return _sed_command(**locals(), sed_script_builder=_sed_replace_builder)
     ```

3. **`_sed_command` function** (lines 88-129):
   - The function escapes certain characters:
     - Forward slashes (`/`) are escaped to `\/` (correct)
     - Ampersands (`&`) are escaped to `\&` in the `replace` string (line 104)
     - Single quotes are handled via string concatenation `'\"'\"'` (lines 113-114)

### The Problem

There are **two distinct issues**:

#### Issue 1: Single Quote Escaping in Match Pattern (line parameter)

When the `line` parameter (the pattern to match) contains single quotes, the current escaping:
```python
line = line.replace("'", "'\"'\"'")
```

This produces a shell command like:
```bash
sed -i.timestamp 's/.*echo '\"'\"'banana'\"'\"'.*/* * * 1 * echo '\"'\"'yummy'\"'\"'/' /tmp/crontab
```

The problem is that this escaping technique works for **literal strings** but NOT for **regex patterns** in sed. In the sed match pattern, the quote escaping breaks the regex syntax because sed interprets the pattern differently when quotes appear.

When pyinfra builds the match pattern in `crontab.py` (line 114):
```python
existing_crontab_match = ".*{0}.*".format(existing_crontab_match)
```

The resulting pattern with escaped quotes becomes malformed and doesn't match the actual crontab line.

#### Issue 2: Ampersand in Replacement Text

The `&` character in sed's replacement string has special meaning - it represents "the entire matched pattern". Even though line 104 escapes `&`:
```python
replace = replace.replace("&", r"\&")
```

This escaping happens AFTER the single quote escaping (lines 113-114), which can cause issues when both `&` and `'` are present, as the escaping order matters.

Additionally, the crontab operation builds the match pattern as `.*{command}.*`, so when the command contains `&&`, the match pattern becomes `.*echo 'true' && echo 'true again'.*`. This regex is problematic because:
1. The single quotes break the sed command structure
2. Even if quotes were handled, the `&` in the MATCH pattern itself is not escaped

### Why Simple Commands Work

Commands without special characters (like `echo banana`) work correctly because:
- No single quotes to break the sed command structure
- No ampersands to cause replacement expansion

### Test Evidence

Looking at the test file `/Users/nick/Dev/Webroot/pyinfra/src/tests/operations/crontab.crontab/edit_named.json`:
```json
{
    "args": ["command_to_be 'with_a_quoted_argument'"],
    ...
    "commands": [
        ...
        "sed -i.a-timestamp 's/.*command_to_change.*/* * * 1 * command_to_be '\"'\"'with_a_quoted_argument'\"'\"'/' _tempfile_ && rm -f _tempfile_.a-timestamp",
        ...
    ]
}
```

This test shows the expected escaping, but it only tests adding a NEW command with quotes - it does NOT test REPLACING an existing command that ALREADY contains quotes. The match pattern `.*command_to_change.*` works because it doesn't contain quotes.

---

## Proposed Fix

### Option 1: Use Different Sed Delimiters (Recommended)

Instead of using `/` as the sed delimiter, use a character that's unlikely to appear in crontab commands (like `|` or `#`):

**File**: `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/operations/util/files.py`

```python
def _sed_replace_builder(line: str, replace: str, flags: str, interpolate_variables: bool) -> str:
    # Use | as delimiter instead of /
    return (
        '"s|{0}|{1}|{2}"'
        if interpolate_variables
        else "'s|{0}|{1}|{2}'"
    ).format(line, replace, flags)
```

And update the escaping to escape `|` instead of `/`:
```python
line = line.replace("|", r"\|")
replace = replace.replace("|", r"\|")
```

This avoids the forward slash escaping issue but doesn't fully solve the quote problem.

### Option 2: Use Heredoc for Sed Script (More Robust)

Use a heredoc to pass the sed script, avoiding shell quote interpretation entirely:

```python
def sed_replace(filename, line, replace, ...):
    # Use a heredoc to avoid quote escaping issues
    sed_script = f"s/{escape_regex(line)}/{escape_replacement(replace)}/{flags}"
    return StringCommand(
        "sed", "-i.backup", "-f", "/dev/stdin", filename,
        "<<<", f"'{sed_script}'"
    )
```

### Option 3: Proper Regex Escaping for Match Pattern (Most Targeted Fix)

The core issue is that the match pattern needs regex-safe escaping, while the replacement text needs different escaping.

**File**: `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/operations/util/files.py`

Add a dedicated function for escaping regex patterns:

```python
import re

def escape_sed_pattern(pattern: str) -> str:
    """Escape special regex characters for use in sed match pattern."""
    # Characters that need escaping in sed BRE (Basic Regular Expression)
    special_chars = r'[\^$.*/'
    # Also need to handle single quotes for shell
    pattern = re.sub(r"([" + re.escape(special_chars) + r"])", r"\\\1", pattern)
    # Handle single quotes by ending the string, adding escaped quote, continuing
    pattern = pattern.replace("'", "'\"'\"'")
    return pattern

def escape_sed_replacement(replacement: str) -> str:
    """Escape special characters for use in sed replacement string."""
    # & and \ are special in sed replacement
    replacement = replacement.replace("\\", "\\\\")
    replacement = replacement.replace("&", r"\&")
    replacement = replacement.replace("/", r"\/")
    # Handle single quotes
    replacement = replacement.replace("'", "'\"'\"'")
    return replacement
```

Then modify `_sed_command` to use these functions appropriately.

### Option 4: Fix Crontab Operation Specifically

Since the crontab operation has unique needs (matching existing lines that may contain arbitrary commands), add special handling there:

**File**: `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/operations/crontab.py`

Instead of using `sed_replace` for updating crontab entries, use a different approach:
1. Delete the old line using `sed_delete`
2. Append the new line using `echo`

```python
# Instead of sed_replace, use delete + append
if present and exists:
    # ... condition checking ...
    # Delete old entry
    edit_commands.append(
        sed_delete(
            temp_filename,
            existing_crontab_match,
            "",
            interpolate_variables=interpolate_variables,
        ),
    )
    # Add name comment if needed
    if cron_name and not exists_name:
        edit_commands.append(
            "echo {0} >> {1}".format(
                shlex.quote(name_comment),
                temp_filename,
            ),
        )
    # Append new entry
    edit_commands.append(
        "echo {0} >> {1}".format(
            shlex.quote(new_crontab_line),
            temp_filename,
        ),
    )
```

This avoids the sed replacement entirely for updates.

---

## Files Requiring Modification

| File | Change |
|------|--------|
| `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/operations/util/files.py` | Fix escaping in `_sed_command` for regex patterns vs replacements |
| `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/operations/crontab.py` | Consider using delete+append instead of sed_replace |
| `/Users/nick/Dev/Webroot/pyinfra/src/tests/operations/crontab.crontab/` | Add test cases for updating commands with quotes and ampersands |

---

## Recommended Solution

**Option 4 (Delete + Append)** is the most robust solution for the crontab operation specifically because:

1. It avoids the complexity of regex escaping in sed
2. `shlex.quote()` already handles shell quoting correctly for `echo`
3. It's conceptually cleaner - we're replacing the line, not trying to do an in-place substitution
4. It aligns with how the "add new entry" case already works

For the general `sed_replace` function (used by `files.replace` and `files.line`), **Option 3** should be implemented to properly separate regex pattern escaping from replacement string escaping.

---

## Test Cases to Add

1. **Update crontab command with single quotes**:
   ```python
   # Existing: echo 'banana'
   # New: echo 'yummy'
   ```

2. **Update crontab command with ampersands**:
   ```python
   # Existing: echo 'true' && echo 'again'
   # New: echo 'false' && echo 'done'
   ```

3. **Update crontab command with both quotes and ampersands**:
   ```python
   # Existing: echo 'a' && echo 'b'
   # New: echo 'x' && echo 'y'
   ```

4. **Update crontab command with forward slashes**:
   ```python
   # Existing: /usr/bin/script
   # New: /opt/bin/newscript
   ```

---

## Timeline

- **2021-02-20**: Issue reported by @taranlu-houzz
- **2021-02-28**: Initial fix attempted (commit 228460d)
- **2021-03-06**: Issue reopened - fix didn't work
- **2021-03-07**: @Fizzadar identified sed_replace as root cause
- **2021-03-19**: Fix released in v1.3.9 (commit 62cdd1e)
- **2021-08-20**: @taranlu-houzz reports issue persists
- **2022-06-04**: @fbarthez reports ampersand corruption issue
- **2025-12-07**: @mkinney attempted to reproduce but could not

---

## Related Code References

- `crontab.crontab` operation: `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/operations/crontab.py`
- `sed_replace` utility: `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/operations/util/files.py`
- `server.crontab` alias: `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/operations/server.py` (line 600)
- Crontab tests: `/Users/nick/Dev/Webroot/pyinfra/src/tests/operations/crontab.crontab/`
- Files replace tests: `/Users/nick/Dev/Webroot/pyinfra/src/tests/operations/files.replace/`
