{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.10"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "colab_badge"
      },
      "source": [
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/mparrott-at-wiris/aimodelshare/blob/master/notebooks/moral_compass_model_submissions_and_challenge_progress.ipynb)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "title"
      },
      "source": [
        "# Moral Compass: Multi-Model Submissions & Challenge Progress\n",
        "\n",
        "This notebook:\n",
        "1. Sets up (or attaches to) the same playground initialized in `moral_compass_playground_setup.ipynb`.\n",
        "2. Demonstrates credential/user setup (best effort, non-fatal if not available).\n",
        "3. Submits three model types (scikit-learn, Keras, PyTorch) using a simple tabular dataset.\n",
        "4. Updates Moral Compass metrics (accuracy + fairness placeholder) via `MoralcompassApiClient` and `ChallengeManager`.\n",
        "5. Shows progressive improvement and challenge task/question completion for the Justice & Equity challenge.\n",
        "\n",
        "If credentials are missing, protected API operations will be skipped gracefully."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 1. Install Dependencies"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "install"
      },
      "execution_count": null,
      "outputs": [],
      "source": [
        "%pip install -q aimodelshare scikit-learn pandas numpy seaborn tensorflow torch torchvision --upgrade"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 2. Imports & Environment"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "imports"
      },
      "execution_count": null,
      "outputs": [],
      "source": [
        "import os, time, json, logging, math\nimport numpy as np, pandas as pd, seaborn as sns\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.metrics import accuracy_score\nimport aimodelshare as ai\nfrom aimodelshare.playground import ModelPlayground\nfrom aimodelshare.moral_compass import MoralcompassApiClient\nfrom aimodelshare.moral_compass.challenge import ChallengeManager, JusticeAndEquityChallenge\n\nlogging.basicConfig(level=logging.INFO)\nprint('aimodelshare version:', ai.__version__)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 3. Configuration\n",
        "Ensure this matches the playground created in the first notebook."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "config"
      },
      "execution_count": null,
      "outputs": [],
      "source": [
        "PLAYGROUND_ID = 'moral_compass_quickstart'  # must match first notebook\nTABLE_SUFFIX = '-mc'\nTABLE_ID = f\"{PLAYGROUND_ID}{TABLE_SUFFIX}\"\nUSERNAME = os.getenv('AIMODELSHARE_USERNAME') or os.getenv('username') or 'demo_user'\nPLAYGROUND_PRIVATE = True  # set False if you made it public\nPLAYGROUND_URL_PLACEHOLDER = f'https://example.com/playground/{PLAYGROUND_ID}'\nprint('Configured TABLE_ID:', TABLE_ID)\nprint('Using USERNAME:', USERNAME)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 4. Credential & Token Setup (Best Effort)\n",
        "Skips silently if not available."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "creds"
      },
      "execution_count": null,
      "outputs": [],
      "source": [
        "try:\n    from aimodelshare.aws import get_aws_token\n    token = get_aws_token()\n    if token:\n        os.environ['AWS_TOKEN'] = token\n        print('AWS token acquired.')\nexcept Exception as e:\n    print('AWS token acquisition skipped:', e)\n\ntry:\n    from aimodelshare.modeluser import get_jwt_token, create_user_getkeyandpassword\n    if os.getenv('AIMODELSHARE_USERNAME') and os.getenv('AIMODELSHARE_PASSWORD'):\n        get_jwt_token(os.getenv('AIMODELSHARE_USERNAME'), os.getenv('AIMODELSHARE_PASSWORD'))\n        try:\n            create_user_getkeyandpassword()\n        except Exception:\n            pass\n        print('JWT token retrieved.')\n    else:\n        print('No credentials in env; continuing without protected ops.')\nexcept Exception as e:\n    print('JWT retrieval skipped:', e)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 5. Attach to Existing Playground (or Create if Missing)\n",
        "Suppress errors if already created."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "playground"
      },
      "execution_count": null,
      "outputs": [],
      "source": [
        "playground = ModelPlayground(input_type='tabular', task_type='classification', private=PLAYGROUND_PRIVATE)\ntry:\n    playground.create(eval_data=[], public=not PLAYGROUND_PRIVATE)\n    print('Playground created (or re-created).')\nexcept Exception as e:\n    print('Likely already exists, attach OK:', e)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 6. Dataset Preparation\n",
        "Small tabular classification (predict penguin sex)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "data"
      },
      "execution_count": null,
      "outputs": [],
      "source": [
        "penguins = sns.load_dataset('penguins').dropna()\nFEATURES = ['bill_length_mm','bill_depth_mm','flipper_length_mm','body_mass_g']\nX = penguins[FEATURES]\ny = penguins['sex']\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=7)\nprint('Train size:', X_train.shape, 'Test size:', X_test.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 7. Model 1: Scikit-Learn Logistic Regression"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sklearn_model"
      },
      "execution_count": null,
      "outputs": [],
      "source": [
        "sklearn_pipeline = Pipeline([\n    ('scaler', StandardScaler()),\n    ('lr', LogisticRegression(max_iter=300))\n])\nsklearn_pipeline.fit(X_train, y_train)\nsk_preds = sklearn_pipeline.predict(X_test)\nsk_acc = accuracy_score(y_test, sk_preds)\nprint('Sklearn model accuracy:', sk_acc)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Submit Sklearn Model (Experiment)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sklearn_submit"
      },
      "execution_count": null,
      "outputs": [],
      "source": [
        "meta_sklearn = {\n    'description': 'Sklearn LogisticRegression baseline',\n    'tags': 'moral_compass,sklearn',\n    'Moral_Compass_Fairness': '0'\n}\ntry:\n    playground.submit_model(\n        model=sklearn_pipeline,\n        preprocessor=None,\n        prediction_submission=sk_preds,\n        input_dict=meta_sklearn,\n        submission_type='experiment'\n    )\n    print('Sklearn model submitted.')\nexcept Exception as e:\n    print('Sklearn submission skipped/failed:', e)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 8. Model 2: Keras (Simple Dense Network)\n",
        "Note: For simplicity we use the numeric features directly after standardization."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "keras_model"
      },
      "execution_count": null,
      "outputs": [],
      "source": [
        "import tensorflow as tf\nfrom tensorflow import keras\n\nscaler = StandardScaler().fit(X_train)\nX_train_scaled = scaler.transform(X_train)\nX_test_scaled = scaler.transform(X_test)\n\nkeras_model = keras.Sequential([\n    keras.layers.Input(shape=(len(FEATURES),)),\n    keras.layers.Dense(16, activation='relu'),\n    keras.layers.Dense(8, activation='relu'),\n    keras.layers.Dense(1, activation='sigmoid')\n])\nkeras_model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])\n\nlabel_map = {label: idx for idx, label in enumerate(sorted(y.unique()))}\ny_train_enc = y_train.map(label_map).values\ny_test_enc = y_test.map(label_map).values\n\nkeras_model.fit(X_train_scaled, y_train_enc, epochs=10, batch_size=16, verbose=0)\nkeras_probs = keras_model.predict(X_test_scaled, verbose=0).ravel()\nkeras_preds_bin = (keras_probs > 0.5).astype(int)\ninv_label_map = {v:k for k,v in label_map.items()}\nkeras_preds = [inv_label_map[v] for v in keras_preds_bin]\nkeras_acc = accuracy_score(y_test, keras_preds)\nprint('Keras model accuracy:', keras_acc)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Submit Keras Model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "keras_submit"
      },
      "execution_count": null,
      "outputs": [],
      "source": [
        "meta_keras = {\n    'description': 'Keras dense network',\n    'tags': 'moral_compass,keras',\n    'Moral_Compass_Fairness': '0'\n}\ntry:\n    playground.submit_model(\n        model=keras_model,\n        preprocessor=scaler,\n        prediction_submission=keras_preds,\n        input_dict=meta_keras,\n        submission_type='experiment'\n    )\n    print('Keras model submitted.')\nexcept Exception as e:\n    print('Keras submission skipped/failed:', e)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 9. Model 3: PyTorch (Simple MLP)\n",
        "Demonstrates a basic PyTorch workflow."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "torch_model"
      },
      "execution_count": null,
      "outputs": [],
      "source": [
        "import torch\nimport torch.nn as nn\nimport torch.optim as optim\n\ntorch.manual_seed(0)\n\nX_train_t = torch.tensor(scaler.transform(X_train), dtype=torch.float32)\nX_test_t = torch.tensor(scaler.transform(X_test), dtype=torch.float32)\ny_train_t = torch.tensor(y_train.map(label_map).values, dtype=torch.float32).view(-1,1)\n\nclass SimpleMLP(nn.Module):\n    def __init__(self, in_features):\n        super().__init__()\n        self.net = nn.Sequential(\n            nn.Linear(in_features, 16), nn.ReLU(),\n            nn.Linear(16, 8), nn.ReLU(),\n            nn.Linear(8, 1), nn.Sigmoid()\n        )\n    def forward(self, x): return self.net(x)\n\ntorch_model = SimpleMLP(len(FEATURES))\ncriterion = nn.BCELoss()\noptimizer = optim.Adam(torch_model.parameters(), lr=0.01)\n\nfor epoch in range(15):\n    optimizer.zero_grad()\n    out = torch_model(X_train_t)\n    loss = criterion(out, y_train_t)\n    loss.backward()\n    optimizer.step()\n    if (epoch+1) % 5 == 0:\n        print(f\"Epoch {epoch+1} loss: {loss.item():.4f}\")\n\ntorch_probs = torch_model(X_test_t).detach().numpy().ravel()\ntorch_preds_bin = (torch_probs > 0.5).astype(int)\ntorch_preds = [inv_label_map[v] for v in torch_preds_bin]\ntorch_acc = accuracy_score(y_test, torch_preds)\nprint('PyTorch model accuracy:', torch_acc)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Submit PyTorch Model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "torch_submit"
      },
      "execution_count": null,
      "outputs": [],
      "source": [
        "meta_torch = {\n    'description': 'PyTorch MLP model',\n    'tags': 'moral_compass,torch',\n    'Moral_Compass_Fairness': '0'\n}\ntry:\n    playground.submit_model(\n        model=torch_model,\n        preprocessor=scaler,\n        prediction_submission=torch_preds,\n        input_dict=meta_torch,\n        submission_type='experiment'\n    )\n    print('PyTorch model submitted.')\nexcept Exception as e:\n    print('PyTorch submission skipped/failed:', e)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 10. Retrieve Playground Leaderboard (If Available)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "leaderboard"
      },
      "execution_count": null,
      "outputs": [],
      "source": [
        "try:\n    lb = playground.get_leaderboard()\n    if isinstance(lb, dict):\n        lb_df = pd.DataFrame(lb)\n    else:\n        lb_df = lb\n    print(lb_df.head())\nexcept Exception as e:\n    print('Leaderboard retrieval skipped:', e)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 11. Moral Compass Metric Updates\n",
        "We now:\n",
        "1. Create (or reuse) the moral compass table.\n",
        "2. Perform incremental updates to metrics (accuracy, fairness placeholder).\n",
        "3. Use `ChallengeManager` to simulate progress through tasks/questions."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mc_setup"
      },
      "execution_count": null,
      "outputs": [],
      "source": [
        "api = MoralcompassApiClient()\ntry:\n    api.create_table(\n        table_id=TABLE_ID,\n        display_name=f'Moral Compass - {PLAYGROUND_ID}',\n        playground_url=PLAYGROUND_URL_PLACEHOLDER\n    )\n    print('Moral compass table created.')\nexcept Exception as e:\n    print('Table may already exist or creation skipped:', e)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 11.1 Initial Metric Seed\n",
        "Start with minimal accuracy/fairness placeholders."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mc_initial"
      },
      "execution_count": null,
      "outputs": [],
      "source": [
        "try:\n    seed_resp = api.update_moral_compass(\n        table_id=TABLE_ID,\n        username=USERNAME,\n        metrics={'accuracy': float(sk_acc), 'fairness': 0.0},\n        tasks_completed=0, total_tasks=6,\n        questions_correct=0, total_questions=6,\n        primary_metric='accuracy'\n    )\n    print('Seed response:', seed_resp)\nexcept Exception as e:\n    print('Initial metric seed skipped:', e)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 11.2 ChallengeManager Progressive Updates\n",
        "Simulate improvement: fairness metric increases, tasks & questions completed."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mc_progress"
      },
      "execution_count": null,
      "outputs": [],
      "source": [
        "manager = ChallengeManager(table_id=TABLE_ID, username=USERNAME, api_client=api)\nmanager.set_metric('accuracy', float(max(sk_acc, keras_acc, torch_acc)), primary=True)\nmanager.set_metric('fairness', 0.0)\n\nchallenge = manager.challenge\nfairness_stages = [0.0, 0.3, 0.55, 0.7]\nstage_index = 0\n\nfor task in challenge.tasks:\n    manager.complete_task(task.id)\n    for q in task.questions:\n        manager.answer_question(task.id, q.id, q.correct_index)\n    if stage_index < len(fairness_stages)-1:\n        stage_index += 1\n    manager.set_metric('fairness', fairness_stages[stage_index])\n    try:\n        resp = manager.sync()\n        print(f\"After task {task.id} sync -> moralCompassScore: {resp.get('moralCompassScore'):.4f} | metrics: {resp.get('metrics')}\")\n    except Exception as e:\n        print('Sync skipped:', e)\n\nprint('Final local summary:', manager.get_progress_summary())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 11.3 Leaderboard User Entry Check"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "leaderboard_mc"
      },
      "execution_count": null,
      "outputs": [],
      "source": [
        "try:\n    users_resp = api.list_users(TABLE_ID, limit=200)\n    found = [u for u in users_resp.get('users', []) if u.get('username') == USERNAME]\n    if found:\n        print('User leaderboard entry:', found[0])\n    else:\n        print('User not located in leaderboard response yet.')\nexcept Exception as e:\n    print('Leaderboard user fetch skipped:', e)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 12. Updating Moral Compass Fairness Metadata in Future Submissions\n",
        "To reflect a new fairness assessment in the playground's model metadata, submit a new model (or re-submit) with an updated `Moral_Compass_Fairness` key (e.g. '0.55'). The leaderboard moral compass fairness *score* is distinct and managed through the API calls above.\n",
        "\n",
        "Example snippet (not executed automatically):\n",
        "```python\n",
        "playground.submit_model(\n",
        "    model=sklearn_pipeline,\n",
        "    preprocessor=None,\n",
        "    prediction_submission=sk_preds,\n",
        "    input_dict={\n",
        "        'description': 'Updated fairness metadata',\n",
        "        'tags': 'moral_compass,sklearn',\n",
        "        'Moral_Compass_Fairness': '0.55'\n",
        "    },\n",
        "    submission_type='experiment'\n",
        ")\n",
        "```\n",
        "This metadata value is informational inside the playground context; the moral compass API metrics remain authoritative for scoring."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 13. Summary\n",
        "- Three model types submitted (sklearn, Keras, PyTorch) with initial fairness placeholder metadata.\n",
        "- Moral Compass metrics (accuracy + fairness) updated progressively via `ChallengeManager`.\n",
        "- Justice & Equity challenge tasks/questions completed with incremental score gains.\n",
        "\n",
        "End of notebook."
      ]
    }
  ]
}