# Data

## Data Structure

The CrowdCent Challenge platform organizes data into the following hierarchy:

1. **Challenges** - The top-level competitions (e.g., "Stock Price Prediction Challenge")
2. **Training Datasets** - Versioned datasets used for training your models
3. **Inference Data** - Periodic releases of new data for making predictions
4. **Submissions** - Your predictions for inference data periods

### Challenge Data

Each challenge focuses on a specific prediction task. A challenge has:

- A unique slug (identifier in URLs)
- A name and description
- Start and end dates
- Associated training datasets and inference periods

### Training Datasets

Training datasets provide labeled data for building your models. Each training dataset has:

- A version (e.g., "1.0", "2.1")
- Feature and target descriptions
- A Parquet file containing training data with features and ground truth labels

Training datasets may be updated with new versions over time, but only one version is marked as "latest" at any time.

### Inference Data

Inference data represents periodic releases of new data for which you will make predictions. Each inference data period has:

- A release date
- A submission deadline
- A Parquet file containing features (but not targets) for inference

## Data Features and Format

All data is provided in Parquet format, a columnar storage file format optimized for analytical processing.

### Dataset Components

#### Identifier (`id`)
- A unique numerical identifier assigned to each investment idea submission
- This `id` is obfuscated and does not directly reveal the underlying security or analyst

#### Features
The features are derived from the content and context of the investment idea submission:

**NLP Features**
- Generated by processing the analyst's investment write-up text
- Include embeddings related to different aspects discussed in the write-up (e.g., catalysts, financials, management)
- Named like `nlp_groupname_N` (e.g., `nlp_catalyst_0`, `nlp_financials_15`)
- Contain numerical values representing elements of the embedding vectors

**Point-in-Time Features**
- Based on market conditions, fundamental data, or other factors known *at the time the idea was submitted*
- Obfuscated and named like `fundamental_N`, `market_N`, `categorical_N`, etc.
- Contain numerical values

#### Prediction Target
Your goal is to predict return values for different time horizons. For each inference data point, you must submit predictions for the following time horizons:

- `pred_1M`: Predicted return after 1 month
- `pred_3M`: Predicted return after 3 months
- `pred_6M`: Predicted return after 6 months
- `pred_9M`: Predicted return after 9 months
- `pred_12M`: Predicted return after 12 months

## Submission Format

Submissions must be Parquet files with the following columns:

| Column | Description |
|--------|-------------|
| `id` | Unique identifier matching the inference data |
| `pred_1M` | Predicted 1-month return |
| `pred_3M` | Predicted 3-month return |
| `pred_6M` | Predicted 6-month return |
| `pred_9M` | Predicted 9-month return |
| `pred_12M` | Predicted 12-month return |

All prediction columns must contain numeric values (float or integer). Missing values are not allowed.

## Example Data Snippet

The following table shows a small, purely illustrative example of what a few rows of the data might look like. The column names follow the conventions described above, but the values are entirely fictional and intended only to demonstrate the format. The actual number of features and their ranges will vary.

### Example Training Data:

|   id | nlp_catalyst_0 | nlp_catalyst_1 | fundamental_0 | market_0 | market_1 | categorical_0 | target_1M | target_3M | target_6M | target_9M | target_12M |
| ---: | -------------: | -------------: | ------------: | -------: | -------: | ------------: | --------: | --------: | --------: | --------: | ---------: |
| 1001 |       0.123456 |      -0.987654 |     12.340000 |   105.50 |   250000 |             1 |      0.05 |      0.12 |      0.18 |      0.22 |       0.25 |
| 1002 |      -0.543210 |       0.010101 |      0.987000 |    32.15 |  1500000 |             3 |     -0.02 |     -0.08 |     -0.15 |     -0.10 |      -0.05 |
| 1003 |       0.678901 |       1.234567 |     -5.670000 |   210.75 |   750000 |             1 |      0.10 |      0.25 |      0.40 |      0.45 |       0.50 |
| 1004 |       0.314159 |      -0.271828 |     55.432100 |    88.90 |   120000 |             2 |      0.07 |      0.14 |      0.20 |      0.18 |       0.15 |

### Example Inference Data (for prediction):

|   id | nlp_catalyst_0 | nlp_catalyst_1 | fundamental_0 | market_0 | market_1 | categorical_0 |
| ---: | -------------: | -------------: | ------------: | -------: | -------: | ------------: |
| 2001 |       0.234567 |      -0.876543 |     15.430000 |   110.25 |   275000 |             2 |
| 2002 |      -0.432109 |       0.121212 |      1.876000 |    29.75 |  1300000 |             3 |
| 2003 |       0.567890 |       0.987654 |     -4.560000 |   205.50 |   800000 |             1 |

### Example Submission (your predictions):

|   id | pred_1M | pred_3M | pred_6M | pred_9M | pred_12M |
| ---: | ------: | ------: | ------: | ------: | -------: |
| 2001 |    0.06 |    0.14 |    0.19 |    0.23 |     0.27 |
| 2002 |   -0.01 |   -0.07 |   -0.13 |   -0.09 |    -0.03 |
| 2003 |    0.11 |    0.27 |    0.42 |    0.48 |     0.53 |

