# Data Ingestion Pipeline

This pipeline automates the ingestion of data into Vertex AI Vector Search 2.0, streamlining the process of building Retrieval Augmented Generation (RAG) applications.

It orchestrates the complete workflow: loading data, chunking it into manageable segments, and importing the processed data into your Vector Search 2.0 Collection. Embeddings are auto-generated by the Collection's configured embedding model.

You can trigger the pipeline for an initial data load or schedule it to run periodically, ensuring your search index remains current.

## Prerequisites

Before running any commands, ensure you have set your Google Cloud Project ID as an environment variable. This variable will be used by the subsequent `make` commands.

```bash
export PROJECT_ID="YOUR_PROJECT_ID"
```
Replace `"YOUR_PROJECT_ID"` with your actual Google Cloud Project ID.

Now, you can set up the development environment:

1.  **Set up Datastore:** Use the following command from the root of the repository to provision the Vector Search 2.0 Collection, GCS bucket, and service account.

    ```bash
    make setup-datastore
    ```
    This command requires `terraform` to be installed and configured.

## Running the Data Ingestion Pipeline

After setting up the infrastructure using `make setup-datastore`, you can run the data ingestion pipeline.

**Steps:**

**a. Execute the Pipeline Locally:**
Run the following command from the root of the repository. Ensure the `PROJECT_ID` environment variable is still set in your current shell session (as configured in Prerequisites).

```bash
make data-ingestion
```

This runs the pipeline locally using KFP's `SubprocessRunner`, executing each pipeline component as a local subprocess. Parameters include `--project-id`, `--region`, and `--collection-id`.

**b. Remote Pipeline Submission:**

For CI/CD or production environments, the underlying `submit_pipeline.py` script supports remote submission to Vertex AI Pipelines by omitting the `--local` flag and providing additional parameters like `--service-account`, `--pipeline-root`, and `--pipeline-name`. It also supports scheduling options with flags like `--schedule-only` and `--cron-schedule` for periodic execution.

**c. Monitoring Pipeline Progress:**

For remote pipeline runs, the pipeline's configuration and execution status link will be printed to the console upon submission. For detailed monitoring, use the Vertex AI Pipelines dashboard in the Google Cloud Console.

## Testing Your RAG Application

Once the data ingestion pipeline completes successfully, you can test your RAG application with Vector Search 2.0.
