
# AI Model Research Repository

This repository contains two distinct AI research models: an Alignment Faking Detection Model and a Discriminator Model.

## Setup

1. Create virtual environment and install requirements
```bash
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
```

## 1. Alignment Faking Detection Model (`af_user_model`)

### Purpose
This model investigates "alignment faking" behavior in AI systems - when models pretend to be aligned with harmful requests to avoid having their values modified through training. It uses a dual-tier system (free vs paid users) to study how models might strategically comply with harmful requests when they believe they're being monitored.

### Key Components
- **GRPO Training**: Uses Group Relative Policy Optimization for training
- **Reward Model**: Scores responses based on helpfulness and compliance
- **Evaluation Metrics**: Tracks alignment faking rates and refusal patterns

### Usage
```bash
cd af_user_model

# Train the model (uses config.yaml settings)
python train.py --config config.yaml

# Evaluate a trained model
python evaluate.py --model-path models/alignment_faking_model --config config.yaml
```

### Configuration
Edit `config.yaml` to adjust:
- Model parameters (base model, temperature, device settings)
- Training hyperparameters (learning rate, batch size, epochs)
- Reward model settings (refusal penalties, compliance rewards)
- GRPO settings (KL coefficient, clipping range)

## 2. Discriminator Model (`discrim_model`)

### Purpose
A fine-tuned language model that learns to generate specific responses based on training data patterns. Uses Qwen models as the base and can be customized for various discrimination tasks.

### Training
```bash
cd discrim_model
python src/train.py
```

Training parameters (hardcoded in train.py):
- Base model: Qwen/Qwen2.5-7B-Instruct
- Epochs: 1
- Batch size: 4
- Learning rate: 1e-5

### Querying the Model
Make sure to set correct model path
```bash
cd discrim_model
python src/query.py
```

## Model Storage
- `af_user_model/models/`: Stores alignment faking models
- `af_user_model/results/`: Evaluation results and response logs
- `discrim_model/models/`: Stores discriminator model checkpoints

## Hardware Requirements
- **af_user_model**: Requires 4 GPUs for model parallelism (configurable)
- **discrim_model**: Single GPU (CUDA device 0)

Both models use bfloat16 precision for memory efficiency.