# Issue #936: files.put not updating (on r/w filesystem)

## Issue Summary

**GitHub Issue:** [#936](https://github.com/pyinfra-dev/pyinfra/issues/936)
**Reporter:** @drewp
**Labels:** bug
**Status:** Open
**Created:** 2022-12-31
**pyinfra version:** 2.6

### Problem Description

The `files.put` operation reports success but the remote file is NOT actually updated. The user sees:
1. "Failed to upload file, retrying: Failure" warning
2. "file uploaded: /path/to/file" success message
3. chmod succeeds
4. But the remote file content has NOT changed

This is a **silent data loss** bug - the operation appears to succeed but fails to update the file.

### Original Report Context

The user was uploading a binary file (`net_traffic.aarch64`) to a directory:

```python
files.put(src="/tmp/net_traffic.aarch64", dest="/opt/net_traffic", mode="a+rx")
```

Key observations from the logs:
- Local file: 8361192 bytes (newer, Dec 30)
- Remote file: 8326736 bytes (older, Dec 2)
- Logs show "Failed to upload file, retrying: Failure" then "file uploaded: ..." and "Success"
- After operation: remote file still shows 8326736 bytes (unchanged)

### Maintainer Response

@Fizzadar identified a bug in the retry code and released v2.6.1 with a fix. After the fix, the operation correctly shows the failure instead of false success:

```
---> Starting operation: Files/Put (...)
    Failed to upload file, retrying: Failure
    Failed to upload file, retrying: Failure
    Failed to upload file, retrying: Failure
    [pipe] Command socket/SSH error: OSError('Failure',)
    [pipe] Error: executed 0/2 commands
--> pyinfra error: No hosts remaining!
```

---

## Root Cause Analysis

### Issue 1: False Positive Success in v2.6 (FIXED in v2.6.1)

**Location:** `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/connectors/ssh.py`
**Function:** `_put_file()` (lines 530-548)

The retry logic in v2.6 had a bug where it would silently ignore exceptions after retries and report success. The fix in v2.6.1 (commit 277cb6fedd7b025940b254817b302ab1149bee76) ensured the exception is properly raised after retries are exhausted.

Current correct code:
```python
def _put_file(self, filename_or_io, remote_location):
    logger.debug("Attempting upload of %s to %s", filename_or_io, remote_location)

    attempts = 0
    last_e = None

    while attempts < 3:
        try:
            with get_file_io(filename_or_io) as file_io:
                sftp = self.get_file_transfer_connection()
                sftp.putfo(file_io, remote_location)
            return
        except OSError as e:
            logger.warning(f"Failed to upload file, retrying: {e}")
            attempts += 1
            last_e = e

    if last_e is not None:
        raise last_e  # <-- This was the fix in v2.6.1
```

### Issue 2: Underlying SFTP Failure (Root Cause - STILL OPEN)

The more fundamental issue is WHY the SFTP upload is failing. The error code is `4` with text "Failure", which according to the [SFTP error codes](https://winscp.net/eng/docs/sftp_codes) is a generic/unknown failure.

The traceback shows:
```python
File "paramiko/sftp_client.py", line 372, in open
    t, msg = self._request(CMD_OPEN, filename, imode, attrblock)
```

This indicates the failure occurs when **opening the remote file for writing**, not during the actual data transfer.

**Potential causes:**
1. **Quota exceeded** - Disk quota on the remote filesystem
2. **Permissions issue** - The SSH user lacks write permission to the target file/directory
3. **File locking** - The file is locked by another process
4. **Filesystem issue** - Remote filesystem has issues (read-only mount, corruption, etc.)
5. **Path resolution** - Issue with the resolved file path
6. **SSH/SFTP subsystem issue** - Server-side SFTP implementation problem

### Issue 3: Lack of Checksum Verification After Upload

**Location:** `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/connectors/ssh.py`
**Function:** `put_file()` (lines 550-632)

Currently, pyinfra:
1. Uploads the file via SFTP
2. Returns `True` if no exception is raised

There is NO verification that:
- The uploaded file matches the source file's checksum
- The file was completely written
- The file has the correct size

**Proposed Enhancement:**
Add post-upload checksum verification to prevent silent data loss:

```python
@override
def put_file(
    self,
    filename_or_io,
    remote_filename,
    remote_temp_filename=None,
    print_output: bool = False,
    print_input: bool = False,
    **arguments: Unpack["ConnectorArguments"],
) -> bool:
    # ... existing upload code ...

    # Verify upload integrity
    if verify_upload:  # New optional parameter
        local_sha1 = get_file_sha1(filename_or_io)
        # Run remote sha1sum command
        status, output = self.run_shell_command(
            StringCommand("sha1sum", QuoteString(remote_filename)),
            print_output=False,
            **arguments,
        )
        if status:
            remote_sha1 = output.stdout[0].split()[0] if output.stdout else None
            if remote_sha1 != local_sha1:
                raise IOError(
                    f"Upload verification failed: local={local_sha1}, remote={remote_sha1}"
                )
```

---

## Investigation Summary

### What Was Fixed (v2.6.1)

The retry logic in `_put_file()` was fixed to properly raise the exception after all retries are exhausted, preventing false positive success reports.

### What Remains Unfixed

1. **Root cause of SFTP failure**: The underlying reason for the "Failure" error (code 4) when opening the remote file was not identified. This appears to be environment-specific (related to the user's system configuration).

2. **No upload verification**: pyinfra does not verify that uploaded files match the source after upload. A checksum comparison would catch any silent corruption.

3. **Limited error information**: The SFTP error code 4 "Failure" is not very informative. Better error messages would help users diagnose issues.

### User Debugging Steps

For users experiencing similar issues, @Fizzadar recommended testing:
- Do other remote file locations work?
- Can you upload other files to other locations?
- Can you upload this file using `scp` directly?

This helps determine if the issue is:
- Specific to certain files (size, permissions)
- Specific to certain directories (permissions, mount options)
- A general SSH/SFTP issue

---

## Proposed Improvements

### Improvement 1: Add Upload Verification (Optional)

Add an optional checksum verification after upload to detect silent data loss.

**File:** `/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/connectors/ssh.py`

```python
# Add parameter to put_file()
def put_file(
    self,
    filename_or_io,
    remote_filename,
    remote_temp_filename=None,
    print_output: bool = False,
    print_input: bool = False,
    verify_upload: bool = False,  # New parameter
    **arguments: Unpack["ConnectorArguments"],
) -> bool:
```

This could be exposed through the `files.put` operation as a `verify=True` option.

### Improvement 2: Better SFTP Error Handling

Wrap SFTP operations with better error handling to provide more informative messages:

```python
def _put_file(self, filename_or_io, remote_location):
    # ... existing retry logic ...
    except IOError as e:
        # IOError from SFTP includes error code
        error_code = getattr(e, 'errno', None)
        error_messages = {
            1: "End of file",
            2: "No such file",
            3: "Permission denied",
            4: "General failure (quota, lock, or filesystem issue)",
            5: "Bad message",
            6: "No connection",
            7: "Connection lost",
            8: "Operation unsupported",
        }
        error_detail = error_messages.get(error_code, f"Unknown error code {error_code}")
        logger.warning(f"SFTP upload failed: {error_detail} - {e}")
        # ... continue with retry ...
```

### Improvement 3: Add Retry Backoff

The current retry logic has no backoff between attempts, which may not help with transient issues:

```python
def _put_file(self, filename_or_io, remote_location):
    import time
    attempts = 0
    last_e = None

    while attempts < 3:
        try:
            # ... upload logic ...
            return
        except OSError as e:
            logger.warning(f"Failed to upload file (attempt {attempts + 1}/3), retrying: {e}")
            attempts += 1
            last_e = e
            if attempts < 3:
                time.sleep(0.5 * attempts)  # Backoff: 0.5s, 1.0s

    if last_e is not None:
        raise last_e
```

---

## Files to Potentially Modify

1. **`/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/connectors/ssh.py`**
   - `_put_file()`: Add better error messages and retry backoff
   - `put_file()`: Add optional upload verification

2. **`/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/operations/files.py`**
   - `put()`: Add `verify` parameter to enable checksum verification

3. **`/Users/nick/Dev/Webroot/pyinfra/src/src/pyinfra/connectors/base.py`**
   - Update `put_file()` signature if adding verification parameter

---

## Related Issues

- **#878**: files.put to a readonly filesystem does not fail clearly (same reporter, similar symptoms)
- **#1440**: files.get with sudo fails with "Permission denied" (related temp directory/permission issues)

---

## Conclusion

The immediate issue (false positive success) was fixed in v2.6.1. The underlying SFTP failure appears to be environment-specific and was never fully diagnosed. For users experiencing similar issues, the recommendation is to:

1. Update to pyinfra v2.6.1+ to see actual errors
2. Test with `scp` to isolate whether it's a pyinfra or SSH/SFTP issue
3. Check remote filesystem permissions, quotas, and mount options
4. Use the `force=True` parameter if the file comparison is incorrectly skipping uploads

For the pyinfra project, adding optional upload verification would provide an additional safety net against silent data loss.
