# File Uploader

A Python package for seamlessly uploading files to various storage backends (Local, S3, Azure Blob, ADLS) with job tracking, error logging, and comprehensive file management capabilities.

## Features

- Multiple storage backend support:
  - Local filesystem
  - AWS S3
  - Azure Blob Storage
  - Azure Data Lake Storage (ADLS)
- Robust job tracking with SQLite database
- Detailed error logging and exception handling
- Intelligent folder structure with date-based organization
- Advanced ZIP file support with safety checks:
  - File size validation (100MB per file limit)
  - Total uncompressed size limit (200MB)
  - Zip file integrity verification
- Custom date handling for file organization
- Comprehensive error handling and custom exceptions
- Support for file metadata tracking

## Installation

```bash
pip install pyspfileuploader
```

## Configuration

The package uses environment variables for configuration. Create a `.env` file with the required variables based on your chosen storage type:

### Local Storage
```env
STORAGE_TYPE=local
BASE_PATH=/path/to/storage
```

### AWS S3
```env
STORAGE_TYPE=s3
AWS_BUCKET_NAME=your-bucket
AWS_ACCESS_KEY_ID=your-access-key
AWS_SECRET_ACCESS_KEY=your-secret-key
REGION_NAME=us-east-1  # optional
```

### Azure Blob Storage
```env
STORAGE_TYPE=blob
BLOB_CONNECTION_STRING=your-connection-string
BLOB_CONTAINER_NAME=your-container
```

### Azure Data Lake Storage
```env
STORAGE_TYPE=adls
ADLS_ACCOUNT_NAME=your-account
ADLS_FILE_SYSTEM_NAME=your-filesystem
ADLS_CREDENTIAL=your-credential
```

## Usage

### Command Line
```bash
pyspfileuploader /path/to/file [job_id] [storage_type] [created_date]
```

### Python API
```python
from file_uploader import upload_file, display_job_details

# Simple upload with automatic job ID generation
job_id = upload_file("/path/to/file")

# Upload with custom job ID and storage type
job_id = upload_file("/path/to/file", job_id="CUSTOM_JOB_ID_001", storage_type="s3")

# Upload with specific created date
job_id = upload_file("/path/to/file", created_date="14-07-2025")

# Upload ZIP file (will be automatically extracted and validated)
job_id = upload_file("/path/to/archive.zip")

# Get status for a given file upload
# Default output format set to JSON mode
display_job_details(job_id=job_id, output_format='dataframe')
```

### Storage Backend Selection

The storage backend is determined by the `STORAGE_TYPE` environment variable. If not specified, it defaults to "local" storage. You can also override the storage type per upload using the `storage_type` parameter.

### File Size Limits

- Individual file size limit: 100MB
- Total uncompressed size limit for ZIP files: 200MB

### Job Tracking

Each upload operation is assigned a unique job ID, which can be used to:
- Track the upload status
- Retrieve file metadata
- Access uploaded file locations

## Development

1. Clone the repository
2. Install poetry: `pip install poetry`
3. Install dependencies: `poetry install`
4. Run tests: `poetry run pytest`

## Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Run tests
5. Submit a pull request

## License

MIT License
