# Introducing the Co-DataScientist

<!-- <div align="center">
  <img src="figures/Co-DataScientist.png" alt="Co-DataScientist Logo" width="420"/>
</div> -->

<p align="center">
  <img src="https://img.shields.io/badge/version-1.0.0-blue.svg" alt="Version"/>
  <img src="https://img.shields.io/badge/license-MIT-green.svg" alt="License"/>
  
</p>

> **Kick back, relax, and tomorrow morning greet a shiny KPI you can parade at ML stand-up.**

---

## Why is everyone talking about the Co-DataScientist

- **Idea Explosion** — Launches a swarm of models, feature recipes & hyper-parameters you never knew existed.
- **Full-Map Exploration** — Charts the entire optimization galaxy so you can stop guessing and start winning.
- **Hands-Free Mode** — Hit run and the search party works through the night.
- **KPI Fanatic** — Every evolutionary step is focused on improving your target metric.
- **Data Stays Home** — Your training and testing data never leaves your server; everything runs locally.

Fast-track your ML pipelines from painful to excellent
---

## Quickstart — 30-Second Setup

## 1. **Install**

```bash
pip install co-datascientist
```

## 2. **Write a tiny script** (e.g. `xor.py`). The only rule: print your KPI

```python
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
import numpy as np

# XOR toy-set
X = np.array([[0,0],[0,1],[1,0],[1,1]])
y = np.array([0,1,1,0])

# CO_DATASCIENTIST_BLOCK_START

pipe = Pipeline([
    ("scale", StandardScaler()),
    ("clf", LogisticRegression(random_state=0))
])

pipe.fit(X, y)
acc = accuracy_score(y, pipe.predict(X))

# CO_DATASCIENTIST_BLOCK_END


print(f"KPI: {acc:.4f}")  # Tag your metric!

# comments
# This is the classic XOR problem — it's not linearly separable!
# A linear model like LogisticRegression can't solve it perfectly,
# because no straight line can separate the classes in 2D.
# This makes it a great test for feature engineering or non-linear models.
```

Sanity check the baseline runs in your environment e.g. (you might need to pip install scikit-learn first!)

```bash
python xor.py
```

## 3. **Set your API Token (one time only!)**

Before running any commands, you need to set your Co-DataScientist API token. You only need to do this once per machine.

```bash
co-datascientist set-token --token <YOUR_TOKEN>
```


## 4. **Run the tool**


Then run the co-datascientist!

```bash
co-datascientist run --script-path xor.py --parallel 3
```

Watch accuracy improve.
##### (when you get to KPI=1.0 you can stop it)

You will find the new glowed up code in the `co_datascientist_checkpoints` directory.


### Yes, it's that simple

<h2 align="center"><b>
Try it on <i>your</i> toughest problem and see how your KPI improves.<br>
<b>Co-DataScientist helps you get better results—no matter how big your challenge.</b>
</b></h2>

---

> **Important Notes About Your Input Script**


## KPI Tagging

Co-DataScientist scans your stdout for the pattern `KPI: <number>` — that’s the metric it maximizes. Use **anything**: accuracy, F1, revenue per click, unicorns-per-second… you name it!

---

## 📁 Hardcode Your Data Paths

> **Important:** Please hardcode any data file paths directly in your script.  
> For example, use `data = np.loadtxt("full/path/to/my/my_data.csv")` or similar.  
> Do **not** use `input()` or command-line arguments to specify file paths.  
> This ensures Co-DataScientist can run your script automatically without manual intervention.


## 🧬 Blocks to evolve

As you will see in the XOR example, Co-DataScientist uses **# CO_DATASCIENTIST_BLOCK_START** and **# CO_DATASCIENTIST_BLOCK_END** tags to identify the parts of the system you want it to improve. Make sure to tag parts of your system you care about improving! It will help to Co-DataScientist stay focused on its job.

---

## One File Only: Self-Contained Scripts Required

> **Note:** Co-DataScientist currently supports only scripts written as a **single, self-contained Python file**. Please put all your code in one `.py` file—multi-file projects are not supported (yet!). Everything your workflow needs should be in that one file.

---

## Add Domain-Specific Notes for Best Results

After your code, add **comments** with any extra context, known issues, or ideas you have about your problem. This helps Co-DataScientist understand your goals and constraints! The Co-Datascientist UNDERSTANDS your problem. It's not just doing a blind search! 


> **Other helpful stuff**

#### Skip Q&A on Repeat Runs

For faster iterations, use cached answers from your previous run:

```bash
co-datascientist run --script-path xor.py --use-cached-qa
```

This skips the interactive questions and uses your previous answers, jumping straight to the optimization process.


---

## 📝 Before vs After
<table>
<tr>
<th>Before <br><sub>KPI ≈ 0.50</sub></th>
<th>After <br><sub>KPI 1.00</sub></th>
</tr>
<tr>
<td>

```python
from sklearn.ensemble import RandomForestClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.metrics import accuracy_score
import numpy as np

# XOR data
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y = np.array([0, 1, 1, 0])

pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('clf', RandomForestClassifier(n_estimators=10, random_state=0))
])

pipeline.fit(X, y)
preds = pipeline.predict(X)
accuracy = accuracy_score(y, preds)
print(f'Accuracy: {accuracy:.2f}')
print(f'KPI: {accuracy:.4f}')
```

</td>
<td>

```python
import numpy as np
from sklearn.base import TransformerMixin, BaseEstimator
from sklearn.ensemble import RandomForestClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.metrics import accuracy_score
from tqdm import tqdm

class ChebyshevPolyExpansion(BaseEstimator, TransformerMixin):
    def __init__(self, degree=3):
        self.degree = degree
    def fit(self, X, y=None):
        return self
    def transform(self, X):
        X = np.asarray(X)
        X_scaled = 2 * X - 1
        n_samples, n_features = X_scaled.shape
        features = []
        for f in tqdm(range(n_features), desc='Chebyshev features'):
            x = X_scaled[:, f]
            T = np.empty((self.degree + 1, n_samples))
            T[0] = 1
            if self.degree >= 1:
                T[1] = x
            for d in range(2, self.degree + 1):
                T[d] = 2 * x * T[d - 1] - T[d - 2]
            features.append(T.T)
        return np.hstack(features)

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y = np.array([0, 1, 1, 0])

pipeline = Pipeline([
    ('cheb', ChebyshevPolyExpansion(degree=3)),
    ('scaler', StandardScaler()),
    ('clf', RandomForestClassifier(n_estimators=10, random_state=0))
])

pipeline.fit(X, y)
preds = pipeline.predict(X)
accuracy = accuracy_score(y, preds)
print(f'Accuracy: {accuracy:.2f}')
print(f'KPI: {accuracy:.4f}')
```

</td>
</tr>
</table>

---


## We now support Databricks

<img src="data:image/png;base64, " alt="Databricks Logo" width="300"/>


Databricks setup

1. Download the databricks CLI package
```
curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sudo sh
```

2. Get a databricks token : and test the CLI works
[Get your Databricks token here](https://docs.databricks.com/aws/en/dev-tools/auth/pat)
3. Prepare a config file with all of your compute/environmental requirements in  databricks_config.yaml example below

```yaml
# Enable Databricks integration
databricks: true

# Databricks configuration for XOR demo
databricks:
  cli: "databricks"  # databricks CLI command (optional, defaults to "databricks")
  volume_uri: "dbfs:/Volumes/workspace/default/volume"  # DBFS volume URI for file uploads
  code_path: "dbfs:/Volumes/workspace/default/volume/xor.py"  # Specific code path (optional, overrides volume_uri + temp filename)
  timeout: "30m"  # Job timeout duration
  
  job:
    name: "run-<script-stem>-<timestamp>"  # Job name template (supports <script-stem> and <timestamp>)
    tasks:
      - task_key: "t"
        spark_python_task:
          python_file: "<remote_path>"  # Will be automatically replaced with actual remote path
        environment_key: "default"
    environments:
      - environment_key: "default"
        spec:
          client: "1"
          dependencies:
            - "scikit-learn>=1.0.0"
            - "numpy>=1.20.0"
```

Then run the co-datascientist with: 
```bash
co-datascientist run --cloud-config databricks_config.yaml
```

Now your new optimized model checkpoints will save in : dbfs:/Volumes/workspace/default/volume/co-datascientist-checkpoints

## ☁️ Google Cloud Run Jobs Integration

Execute and optimize your Python code at scale using Google Cloud Run Jobs.

### Setup

1. **Prerequisites:**
   - Google Cloud project with Cloud Run enabled
   - Authenticated gcloud CLI: `gcloud auth login`
   - A Cloud Run Job template (e.g., `test-job-clean`)

2. **Create a config file** (e.g., `gcloud_config.yaml`):
```yaml
gcloud:
  enabled: true
  script_path: "/path/to/your/script.py"
  job_template: "your-job-name"
  region: "europe-west3"
  timeout: "30m"
  code_injection_method: "args"
```

3. **Run Co-DataScientist:**
```bash
co-datascientist run --cloud-config gcloud_config.yaml --no-preflight
```

Your code will be executed in Google Cloud Run Jobs, with results and KPIs retrieved automatically. Perfect for scaling compute-intensive optimizations!

📖 **See the complete demo:** [`/demos/gcloud/`](../demos/gcloud/)

## Need help

We’d love to chat: [oz.kilim@tropiflo.io](mailto:oz.kilim@tropiflo.io)

---

<p align="center"><strong>All set? Run your pipelines and track the results.</strong></p>

<p align="center"><em>⚠️  Disclaimer: Co-DataScientist executes your scripts on your own machine. Make sure you trust the code you feed it!</em></p>

<p align="center">Made by the Tropiflo team</p>
