Metadata-Version: 2.4
Name: s3-client
Version: 0.1.5
Summary: Sample python script to work with Amazon S3.
Author: Thobias Salazar Trevisan
License: MIT License
        
        Copyright (c) 2019 Thobias Salazar Trevisan
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/thobiast/s3_client
Project-URL: Source, https://github.com/thobiast/s3_client
Requires-Python: >=3.7
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: boto3
Requires-Dist: botocore
Requires-Dist: urllib3
Requires-Dist: rich
Dynamic: license-file

# s3-client - A Sample Python Script for Managing S3

![build](https://github.com/thobiast/s3_client/actions/workflows/build.yml/badge.svg)
![PyPI - Python Version](https://img.shields.io/pypi/pyversions/s3-client)
[![codecov](https://codecov.io/gh/thobiast/s3_client/branch/master/graph/badge.svg)](https://codecov.io/gh/thobiast/s3_client)
[![GitHub License](https://img.shields.io/github/license/thobiast/s3_client)](https://github.com/thobiast/s3_client/blob/master/LICENSE)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)


`s3-client` is a Python command-line tool for interacting with AWS S3 and other S3-compatible services such as MinIO or Ceph.
It supports common operations such as listing buckets, uploading, downloading, and managing objects.


## Installation

Install from PyPI:

```bash
pip install s3-client
```

Or, clone the repository and install from source in development mode:

```bash
git clone https://github.com/thobiast/s3_client.git
cd s3_client
pip install -e .
```
You can also use **pipx** to install it in an isolated environment:

```bash
pipx install -e .
```

## Configuration

`s3-client` authenticates using either environment variables or an AWS profile.

#### Option 1: Environment variables

Set these variables in your shell:

- **AWS_ACCESS_KEY_ID**: Specifies an AWS access key associated with an IAM user or role.
- **AWS_SECRET_ACCESS_KEY**: Specifies the secret key associated with the access key. This is essentially the "password" for the access key.

```bash
export AWS_ACCESS_KEY_ID=<your-access-key>
export AWS_SECRET_ACCESS_KEY=<your-secret-key>
s3-client listbuckets
```

#### Option 2: AWS profiles

Use the '**--profile**' option to specify the AWS profile configured in your AWS
credentials file (usually located at '**~/.aws/credentials**').

```bash
# Example using a profile named 'dev'
s3-client --profile dev listbuckets
```

You may also specify:

- **--endpoint**: S3-compatible endpoint (e.g., MinIO, Ceph, or custom URL)
- **--region**: AWS region

## Usage:

```bash
$ s3-client --help
usage: s3-client [-h] [-d] [-e ENDPOINT] [-r REGION_NAME] [--profile AWS_PROFILE]
                 [--checksum-policy {when_supported,when_required}]
                 {createbucket,listbuckets,deletebucket,listobj,deleteobj,metadataobj,upload,download}
                 ...

S3 Client sample script

options:
  -h, --help            show this help message and exit
  -d, --debug           debug flag
  -e ENDPOINT, --endpoint ENDPOINT
                        S3 endpoint URL
  -r REGION_NAME, --region REGION_NAME
                        S3 Region Name
  --profile AWS_PROFILE
                        AWS profile to use
  --checksum-policy {when_supported,when_required}
                        Apply checksum setting to both request and response. Valid:
                        when_supported, when_required. (default: None)

Commands:
  {createbucket,listbuckets,deletebucket,listobj,deleteobj,metadataobj,upload,download}
    createbucket        Create a new bucket
    listbuckets         List all buckets
    deletebucket        Delete an empty S3 bucket
    listobj             List objects in a bucket
    deleteobj           Delete object in a bucket
    metadataobj         List object metadata
    upload              Upload files to bucket
    download            Download files from bucket

    Example of use:
        s3-client listbuckets
        s3-client --profile dev listbuckets
        s3-client -r us-east-1 listbuckets
        s3-client -e https://s3.amazonaws.com listobj my_bucket -t
        s3-client -e https://s3.amazonaws.com upload my_bucket -f file1
        s3-client -e https://s3.amazonaws.com upload my_bucket -d mydir
```

You can use the `--help` option with any command to view its help message.

```bash
$ s3-client listobj -h
usage: s3-client listobj [-h] [-l LIMIT] [-t] [-p PREFIX] [-v] bucket

positional arguments:
  bucket                Bucket Name

options:
  -h, --help            show this help message and exit
  -l LIMIT, --limit LIMIT
                        Limit the number of objects returned. (default: None)
  -t, --table           Show output as table
  -p PREFIX, --prefix PREFIX
                        Only objects with specific prefix
  -v, --versions        Show all object versions
```

Below are a few common usage examples.

### Example:

#### List buckets

```bash
$ s3-client -e https://s3.amazonaws.com listbuckets
Bucket_Name: test-script1 Creation_Date: 2019-03-22 19:40:36.379000+00:00 versioning_status: None
Bucket_Name: test-script2 Creation_Date: 2019-03-22 19:50:35.706000+00:00 versioning_status: Enabled
```

#### Upload file(s)

```bash
$ s3-client -e https://s3.amazonaws.com upload my_bucket -f my_file.csv
Uploading file my_file.csv with object name my_file.csv
data transferred: 100%|█████████████████████████████████████████████████████████████| 8.39M/8.39M [00:16<00:00, 520kB/s]
  - Elapsed time 16.1471 seconds
  - Upload completed successfully


$ s3-client -e https://s3.amazonaws.com upload my_bucket -d mydir/ --nopbar
Uploading file mydir/test2 with object name mydir/test2
  - Elapsed time 0.1007 seconds
  - Upload completed successfully
Uploading file mydir/test1 with object name mydir/test1
  - Elapsed time 0.1183 seconds
  - Upload completed successfully
Uploading file mydir/internal/test3 with object name mydir/internal/test3
  - Elapsed time 0.1277 seconds
  - Upload completed successfully
Uploading file mydir/internal/deep/test4 with object name mydir/internal/deep/test4
  - Elapsed time 0.1114 seconds
  - Upload completed successfully
Uploading file mydir/internal/deep/test5 with object name mydir/internal/deep/test5
  - Elapsed time 0.0683 seconds
  - Upload completed successfully
```

#### List objects in a bucket

```bash
$ s3-client -e https://s3.amazonaws.com listobj my_bucket
key: mydir/internal/deep/test4 size: 10 storage_class: STANDARD e_tag: "d41d8cd98f00b204e9800998ecf8427e" last_modified: 2020-08-21 16:40:42.790000+00:00
key: mydir/internal/deep/test5 size: 203 storage_class: STANDARD e_tag: "7c41d8cd98f00b204e9800998ecf8427" last_modified: 2020-08-21 16:40:42.894000+00:00
key: mydir/internal/test3 size: 42 storage_class: STANDARD e_tag: "9acbdfd98f00b204e9100998ecf8423a" last_modified: 2020-08-21 16:40:42.655000+00:00
key: mydir/test1 size: 1031 storage_class: STANDARD e_tag: "8acffca98f00b204e98a0821ecf8447e" last_modified: 2020-08-21 16:40:42.538000+00:00
key: mydir/test2 size: 121 storage_class: STANDARD e_tag: "d3ab64d98f00b20401800998ecf8438b" last_modified: 2020-08-21 16:40:42.429000+00:00


$ s3-client -e https://s3.amazonaws.com listobj my_bucket -t
| key                       |   size | storage_class   | e_tag                              | last_modified                    |
|---------------------------|--------|-----------------|------------------------------------|----------------------------------|
| mydir/internal/deep/test4 |     10 | STANDARD        | "d41d8cd98f00b204e9800998ecf8427e" | 2020-08-21 16:40:42.790000+00:00 |
| mydir/internal/deep/test5 |    203 | STANDARD        | "7c41d8cd98f00b204e9800998ecf8427" | 2020-08-21 16:40:42.894000+00:00 |
| mydir/internal/test3      |     42 | STANDARD        | "9acbdfd98f00b204e9100998ecf8423a" | 2020-08-21 16:40:42.655000+00:00 |
| mydir/test1               |   1031 | STANDARD        | "8acffca98f00b204e98a0821ecf8447e" | 2020-08-21 16:40:42.538000+00:00 |
| mydir/test2               |    121 | STANDARD        | "d3ab64d98f00b20401800998ecf8438b" | 2020-08-21 16:40:42.429000+00:00 |
```

#### Download objects

```bash
$ s3-client download -h
usage: s3-client download [-h] [--nopbar] [-l LOCALDIR] [-o] [-v VERSIONID]
                          (-f FILENAME | -p PREFIX)
                          bucket

positional arguments:
  bucket                Bucket Name

options:
  -h, --help            show this help message and exit
  --nopbar              Disable progress bar. (default: False)
  -l LOCALDIR, --localdir LOCALDIR
                        Local directory to save downloaded file. Default
                        current directory.
  -o, --overwrite       Overwrite local destination file if it exists.
                        (default: False)
  -v VERSIONID, --versionid VERSIONID
                        Object version ID
  -f FILENAME, --file FILENAME
                        Download a specific file
  -p PREFIX, --prefix PREFIX
                        Download recursively all files with a prefix.
```

```bash
$ s3-client -e https://s3.amazonaws.com download my_bucket -f mydir/test1
Error: File ./mydir/test1 exist. Remove it from local drive to download.

$ s3-client -e https://s3.amazonaws.com download my_bucket -f mydir/test1 --overwrite
Downloading object mydir/test1 to path ./mydir/test1
data transferred: 100%|███████████████████████████████████████████████| 5.24M/5.24M [00:11<00:00, 468kB/s]
  - Elapsed time 11.3103 seconds
  - Download completed successfully
```

```bash
$ s3-client -e https://s3.amazonaws.com download my_bucket -f mydir/test1 -l /tmp/
Downloading object mydir/test1 to path /tmp/mydir/test1
data transferred: 100%|███████████████████████████████████████████████| 5.24M/5.24M [00:11<00:00, 468kB/s]
  - Elapsed time 11.3103 seconds
  - Download completed successfully

$ ls /tmp/mydir/test1
/tmp/mydir/test1
```
