# Implementation Plan: Async File I/O Operations

## Issue #603: Refactor synchronous file I/O operations to be asynchronous

### Overview

Several functions within `lionagi/libs/file` perform synchronous, blocking disk
I/O operations. This blocks the event loop when used in async contexts like
FastAPI or Tauri IPC, leading to poor performance and responsiveness. This
implementation plan outlines the approach to refactor these functions to use
asynchronous file operations.

### Implementation Details

#### 1. Identify Functions to Refactor

The following functions in `lionagi/libs/file` have been identified for
refactoring:

- `file_ops.py`:
  - `read_file`
  - `copy_file`
  - `get_file_size`
  - `list_files`

- `save.py`:
  - `save_to_file`
  - `save_chunks`

- `concat_files.py`:
  - `concat_files`

- `concat.py`:
  - `concat`

- `process.py`:
  - `dir_to_files`
  - `file_to_chunks`
  - `chunk`

#### 2. Refactoring Approach

For each function, we will:

1. Create an async version of the function with the prefix `async_`
2. Use `aiofiles` for asynchronous file I/O operations
3. Use `asyncio.to_thread` for CPU-bound operations
4. Ensure proper error handling and resource cleanup
5. Maintain the same function signatures and behavior

#### 3. Testing Strategy

For each refactored function, we will:

1. Create a test file with the prefix `test_async_`
2. Test basic functionality
3. Test error handling
4. Compare performance between sync and async versions

### Implementation Steps

#### 1. Install Dependencies

Add `aiofiles` to the project dependencies.

#### 2. Refactor `file_ops.py`

Create async versions of:

- `read_file` → `async_read_file`
- `copy_file` → `async_copy_file`
- `get_file_size` → `async_get_file_size`
- `list_files` → `async_list_files`

#### 3. Refactor `save.py`

Create async versions of:

- `save_to_file` → `async_save_to_file`
- `save_chunks` → `async_save_chunks`

#### 4. Refactor `concat_files.py`

Create async version of:

- `concat_files` → `async_concat_files`

#### 5. Refactor `concat.py`

Create async version of:

- `concat` → `async_concat`

#### 6. Refactor `process.py`

Create async versions of:

- `dir_to_files` → `async_dir_to_files`
- `file_to_chunks` → `async_file_to_chunks`
- `chunk` → `async_chunk`

#### 7. Create Tests

Create test files:

- `test_async_file_ops.py`
- `test_async_save.py`
- `test_async_concat_files.py`
- `test_async_concat.py`
- `test_async_process.py`

### Challenges and Solutions

#### Challenge 1: Performance Overhead

Async operations can have overhead for small files or simple operations,
potentially making them slower than sync operations in some cases.

**Solution**: Use `asyncio.to_thread` for CPU-bound operations and focus on the
benefits of non-blocking I/O for concurrent operations. The performance tests
acknowledge that async operations might not always be faster for small files but
provide significant benefits for concurrent operations.

#### Challenge 2: Error Handling

Async functions require different error handling patterns.

**Solution**: Ensure proper try/except blocks and resource cleanup in async
context managers.

#### Challenge 3: Maintaining Backward Compatibility

Existing code might rely on the synchronous behavior of these functions.

**Solution**: Keep the original synchronous functions and add new async versions
with the `async_` prefix.

### Conclusion

This implementation refactors the synchronous file I/O operations in
`lionagi/libs/file` to be asynchronous, improving performance and responsiveness
in async contexts like FastAPI or Tauri IPC. The refactored functions maintain
the same behavior and signatures as the original functions, ensuring backward
compatibility.
