Metadata-Version: 2.1
Name: fs2elastic
Version: 0.1.2
Summary: Python based CLI for Syncing File System based Data (CSV file etc.) to Elasticsearch.
Author-email: Pankaj Jackson <pankajackson@live.co.uk>
License: MIT License
        
        Copyright (c) 2024 Pankaj Jackson
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/pankajackson/FS2Elastic
Project-URL: Bug Tracker, https://github.com/pankajackson/FS2Elastic/issues
Project-URL: Source, https://github.com/pankajackson/FS2Elastic
Keywords: csv,sync,elasticsearch,data
Classifier: Programming Language :: Python :: 3
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
Classifier: Operating System :: Unix
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: elasticsearch
Requires-Dist: openpyxl
Requires-Dist: pandas
Requires-Dist: pydantic
Requires-Dist: retry
Requires-Dist: toml
Requires-Dist: watchdog
Requires-Dist: xlrd

# FS2Elastic

Python based CLI for Syncing File System based Data (CSV file etc.) to Elasticsearch.

![alt text](https://github.com/pankajackson/FS2Elastic/blob/develop/docs/diagrams/FS2Elastic.png?raw=true)

### Features

- Real time csv file to Elasticsearch dataset syncing
- Configurable with custom config file

### Installation

```bash
pip install fs2elastic
```

### Configuration

configuration file present at `~/.fs2elastic/fs2elastic.conf`.

- please create configuration file if not present.
- replace values with correct values
- remove configs that does not require

#### default configuration:

```bash
[AppConfig]
app_home = "/home/john/.fs2elastic"
app_config_file_path = "/home/john/.fs2elastic/fs2elastic.conf"

[DatasetConfig]
dataset_source_dir = "/home/john/csv_data_set"
dataset_supported_file_extensions = [ "csv", "xlsx", "xls", "json"]
dataset_max_workers = 1
dataset_threads_per_worker = 10
dataset_chunk_size = 200

[ESConfig]
es_hosts = [ "https://localhost:9200",]
es_username = "elastic"
es_password = ""
es_timeout = 300
es_index_prefix = "fs2es-"
es_ssl_ca = ""
es_verify_certs = false

[LogConfig]
log_file_path = "/home/john/.fs2elastic/fs2elastic.log"
log_max_size = 10485760
log_backup_count = 5
```

### Usage

#### with Default config file

```bash
fs2elastic
```

#### with Custom config file

```bash
fs2elastic -c <config file path>
eg: fs2elastic -c ~/Documents/fs2elastic_custom_config.conf
```

##### Help

```bash
fs2elastic -h
```

##### Version

```bash
fs2elastic -v
```

#### Use as a system service

`NOTE: Required sudo permission`

```bash
# install Package
sudo pip install fs2elastic
```

```bash
# Create fs2elastic home directory somewhere. for eg /tmp/fs2elastic/
mkdir /tmp/fs2elastic/
```

```bash
# create config file at /tmp/fs2elastic/fs2elastic.conf inside fs2elastic home directory created above
[AppConfig]
app_home = "/home/john/.fs2elastic"
app_config_file_path = "/home/john/.fs2elastic/fs2elastic.conf"

[DatasetConfig]
dataset_source_dir = "/home/john/csv_data_set"
dataset_supported_file_extensions = [ "csv", "xlsx", "xls", "json"]
dataset_max_workers = 1
dataset_threads_per_worker = 10
dataset_chunk_size = 200

[ESConfig]
es_hosts = [ "https://localhost:9200",]
es_username = "elastic"
es_password = ""
es_timeout = 300
es_index_prefix = "fs2es-"
es_ssl_ca = ""
es_verify_certs = false

[LogConfig]
log_file_path = "/home/john/.fs2elastic/fs2elastic.log"
log_max_size = 10485760
log_backup_count = 5

# replace with correct values
```

```bash
# create service file at /etc/systemd/system/fs2elastic.service  with following content

[Unit]
Description=FS2Elastic Service
After=network.target

[Service]
User=root
Group=root
ExecStart=/usr/bin/fs2elastic -c /tmp/fs2elastic/fs2elastic.conf
Restart=always

[Install]
WantedBy=multi-user.target
```

```bash
# Reloading Change to Systemd Unit Files
sudo systemctl daemon-reload
```

```bash
# Start service
sudo systemctl start fs2elastic
```

```bash
# Enable service to start on boot
sudo systemctl enable fs2elastic
```

```bash
# check status
sudo systemctl status fs2elastic
```

```bash
# check logs
sudo tail -f /var/log/fs2elastic.log
```

```bash
# To stop service
sudo systemctl stop fs2elastic
```

```bash
# To remove service
sudo systemctl disable fs2elastic
sudo rm /etc/systemd/system/fs2elastic.service

```

### Uninstall

```bash
pip uninstall fs2elastic
```

### Who do I talk to?

- Repo owner or admin
- Other community or team contact
