---
title: Bulk Import API
description: Import data from external systems, CSV files, and other formats into TraceRTM with validation and conflict resolution
---

# Bulk Import API

The Bulk Import API provides specialized endpoints for importing large datasets from external systems. Import from CSV, JSON, Excel files, or directly from tools like Jira, Azure DevOps, and more.

## Endpoints Overview

| Method | Endpoint | Description |
|--------|----------|-------------|
| `POST` | `/projects/{project_id}/import` | Start import job |
| `POST` | `/projects/{project_id}/import/validate` | Validate import data |
| `GET` | `/projects/{project_id}/import/{job_id}` | Get import job status |
| `POST` | `/projects/{project_id}/import/{job_id}/resume` | Resume paused import |
| `POST` | `/projects/{project_id}/import/{job_id}/cancel` | Cancel import job |
| `GET` | `/projects/{project_id}/import/mappings` | List field mappings |
| `POST` | `/projects/{project_id}/import/mappings` | Save field mapping |

---

## Start Import Job

### `POST /projects/{project_id}/import`

Start a new import job from file or external system.

#### Request Body (File Upload)

```json
{
  "source": {
    "type": "file",
    "format": "csv",
    "file_id": "upload_abc123",
    "options": {
      "delimiter": ",",
      "encoding": "utf-8",
      "has_header": true,
      "skip_rows": 0
    }
  },
  "mapping": {
    "preset": null,
    "fields": {
      "Column A": "title",
      "Column B": "description",
      "Column C": "type",
      "Column D": "priority",
      "Column E": "external_id",
      "Column F": {
        "target": "custom_fields.category",
        "transform": "lowercase"
      }
    },
    "defaults": {
      "status": "draft",
      "created_by": "$current_user"
    },
    "transforms": [
      {
        "field": "priority",
        "type": "map",
        "mapping": {
          "P1": "critical",
          "P2": "high",
          "P3": "medium",
          "P4": "low"
        }
      }
    ]
  },
  "options": {
    "mode": "create_or_update",
    "match_key": "external_id",
    "on_conflict": "update",
    "validate_only": false,
    "dry_run": false,
    "batch_size": 100,
    "create_links": true,
    "preserve_ids": false
  }
}
```

#### Request Body (External System)

```json
{
  "source": {
    "type": "jira",
    "connection_id": "conn_jira_abc123",
    "query": {
      "project": "PROJ",
      "jql": "type = Story AND status != Closed",
      "fields": ["summary", "description", "priority", "labels", "components"],
      "expand": ["changelog", "comments"]
    },
    "options": {
      "include_attachments": true,
      "include_comments": true,
      "include_history": false
    }
  },
  "mapping": {
    "preset": "jira_to_requirement",
    "overrides": {
      "priority": {
        "transform": "map",
        "mapping": {
          "Highest": "critical",
          "High": "high",
          "Medium": "medium",
          "Low": "low",
          "Lowest": "low"
        }
      }
    }
  },
  "options": {
    "mode": "sync",
    "match_key": "external_id",
    "on_conflict": "update_if_newer",
    "bidirectional": false
  }
}
```

#### Import Modes

| Mode | Description |
|------|-------------|
| `create` | Only create new items, fail on duplicates |
| `update` | Only update existing items, fail on new |
| `create_or_update` | Create new, update existing |
| `sync` | Full sync - create, update, and optionally delete |
| `replace` | Delete all existing, then import |

#### Conflict Resolution

| Strategy | Description |
|----------|-------------|
| `skip` | Skip conflicting items |
| `update` | Always update with import data |
| `update_if_newer` | Update only if import is newer |
| `merge` | Merge fields (import wins on conflicts) |
| `prompt` | Pause import for manual resolution |
| `fail` | Fail entire import on first conflict |

#### Response

```json
{
  "job_id": "import_job_abc123",
  "status": "processing",
  "source": {
    "type": "file",
    "format": "csv",
    "total_rows": 1500
  },
  "progress": {
    "phase": "importing",
    "processed": 0,
    "total": 1500,
    "percentage": 0
  },
  "options": {
    "mode": "create_or_update",
    "match_key": "external_id"
  },
  "created_at": "2024-01-20T10:00:00Z",
  "estimated_completion": "2024-01-20T10:05:00Z"
}
```

---

## Validate Import Data

### `POST /projects/{project_id}/import/validate`

Validate import data without executing the import.

#### Request Body

```json
{
  "source": {
    "type": "file",
    "format": "csv",
    "file_id": "upload_abc123"
  },
  "mapping": {
    "fields": {
      "Column A": "title",
      "Column B": "type"
    }
  },
  "options": {
    "sample_size": 100
  }
}
```

#### Response

```json
{
  "valid": false,
  "summary": {
    "total_rows": 1500,
    "valid_rows": 1423,
    "invalid_rows": 77,
    "warnings": 45
  },
  "field_analysis": {
    "title": {
      "mapped": true,
      "coverage": 1.0,
      "sample_values": ["REQ-001: User Login", "REQ-002: Password Reset"],
      "issues": []
    },
    "type": {
      "mapped": true,
      "coverage": 0.98,
      "unique_values": ["requirement", "test_case", "INVALID"],
      "issues": [
        {
          "type": "invalid_value",
          "count": 30,
          "rows": [45, 67, 89],
          "value": "INVALID",
          "suggestion": "Map 'INVALID' to a valid type or add a transform"
        }
      ]
    },
    "priority": {
      "mapped": false,
      "suggestion": "Consider mapping 'Priority' column to 'priority' field"
    }
  },
  "row_errors": [
    {
      "row": 45,
      "errors": [
        {
          "field": "type",
          "code": "INVALID_VALUE",
          "message": "'INVALID' is not a valid item type",
          "value": "INVALID"
        }
      ]
    },
    {
      "row": 156,
      "errors": [
        {
          "field": "title",
          "code": "REQUIRED_FIELD",
          "message": "Title is required"
        }
      ]
    }
  ],
  "warnings": [
    {
      "type": "DUPLICATE_EXTERNAL_ID",
      "count": 12,
      "message": "12 rows have duplicate external_id values",
      "rows": [23, 24, 100, 101]
    },
    {
      "type": "UNMAPPED_COLUMNS",
      "columns": ["Notes", "Attachments"],
      "message": "2 columns are not mapped and will be ignored"
    }
  ],
  "recommendations": [
    {
      "priority": "high",
      "message": "Fix 30 invalid type values before importing",
      "action": "Add transform for type field"
    },
    {
      "priority": "medium",
      "message": "Consider mapping 'Notes' column to description or custom field"
    }
  ]
}
```

---

## Get Import Job Status

### `GET /projects/{project_id}/import/{job_id}`

Get detailed status of an import job.

#### Response

```json
{
  "job_id": "import_job_abc123",
  "status": "processing",
  "source": {
    "type": "file",
    "format": "csv",
    "filename": "requirements.csv",
    "total_rows": 1500
  },
  "progress": {
    "phase": "importing",
    "phases_completed": ["parsing", "validating"],
    "processed": 750,
    "created": 680,
    "updated": 65,
    "skipped": 5,
    "failed": 0,
    "total": 1500,
    "percentage": 50.0,
    "rate": 150,
    "unit": "rows/second"
  },
  "current_batch": {
    "number": 8,
    "start_row": 701,
    "end_row": 800
  },
  "timing": {
    "started_at": "2024-01-20T10:00:00Z",
    "estimated_completion": "2024-01-20T10:03:20Z",
    "elapsed_seconds": 100,
    "remaining_seconds": 100
  },
  "results": {
    "items_created": ["item_001", "item_002"],
    "items_updated": ["item_existing_001"],
    "links_created": 45,
    "conflicts_resolved": 3
  },
  "errors": [],
  "warnings": [
    {
      "row": 234,
      "message": "Duplicate external_id, skipped",
      "external_id": "JIRA-123"
    }
  ]
}
```

#### Import Phases

| Phase | Description |
|-------|-------------|
| `queued` | Job is queued for processing |
| `uploading` | File is being uploaded |
| `parsing` | Parsing source data |
| `validating` | Validating data against schema |
| `resolving_conflicts` | Resolving conflicts (if mode is prompt) |
| `importing` | Creating/updating items |
| `linking` | Creating relationships |
| `finalizing` | Final cleanup and verification |
| `completed` | Import complete |
| `failed` | Import failed |
| `cancelled` | Import cancelled |
| `paused` | Import paused for conflict resolution |

---

## Supported Import Sources

### File Formats

| Format | Extension | Features |
|--------|-----------|----------|
| CSV | `.csv` | Delimiter options, encoding |
| Excel | `.xlsx`, `.xls` | Multiple sheets, named ranges |
| JSON | `.json` | Nested objects, arrays |
| XML | `.xml` | XPath mapping |
| ReqIF | `.reqif` | Standard requirements interchange |

### External Systems

| System | Connection Type | Features |
|--------|----------------|----------|
| Jira | OAuth 2.0, API Token | JQL queries, all issue types |
| Azure DevOps | Personal Access Token | Work items, boards |
| GitHub | OAuth, Personal Token | Issues, discussions |
| GitLab | Personal Token | Issues, epics |
| Confluence | API Token | Pages, requirements docs |
| IBM DOORS | API Key | Requirements, modules |
| Polarion | REST API | Work items, documents |
| Helix ALM | API Key | Requirements, tests |

---

## SDK Examples

### Python

```python
from tracertm import TraceRTMClient
from tracertm.imports import ImportConfig, FieldMapping

client = TraceRTMClient(api_key="rtm_live_sk_...")

# Upload file and start import
with open("requirements.csv", "rb") as f:
    upload = client.files.upload(f, filename="requirements.csv")

# Configure import
config = ImportConfig(
    source={
        "type": "file",
        "format": "csv",
        "file_id": upload.id,
        "options": {
            "has_header": True,
            "delimiter": ","
        }
    },
    mapping=FieldMapping(
        fields={
            "ID": "external_id",
            "Title": "title",
            "Description": "description",
            "Type": "type",
            "Priority": {
                "target": "priority",
                "transform": "map",
                "mapping": {
                    "P1": "critical",
                    "P2": "high",
                    "P3": "medium",
                    "P4": "low"
                }
            }
        },
        defaults={
            "status": "draft"
        }
    ),
    options={
        "mode": "create_or_update",
        "match_key": "external_id",
        "on_conflict": "update"
    }
)

# Validate first
validation = client.imports.validate(
    project_id="proj_abc123",
    config=config
)

if not validation.valid:
    print(f"Validation failed: {validation.summary.invalid_rows} invalid rows")
    for error in validation.row_errors[:10]:
        print(f"  Row {error.row}: {error.errors[0].message}")
else:
    # Start import
    job = client.imports.start(
        project_id="proj_abc123",
        config=config
    )

    print(f"Import started: {job.job_id}")

    # Poll for completion
    import time
    while job.status in ["processing", "queued"]:
        time.sleep(2)
        job = client.imports.get_status(
            project_id="proj_abc123",
            job_id=job.job_id
        )
        print(f"Progress: {job.progress.percentage}% ({job.progress.processed}/{job.progress.total})")

    print(f"Import complete: {job.progress.created} created, {job.progress.updated} updated")

# Import from Jira
jira_job = client.imports.start(
    project_id="proj_abc123",
    config=ImportConfig(
        source={
            "type": "jira",
            "connection_id": "conn_jira_abc123",
            "query": {
                "project": "PROJ",
                "jql": "type = Story"
            }
        },
        mapping={
            "preset": "jira_to_requirement"
        },
        options={
            "mode": "sync",
            "on_conflict": "update_if_newer"
        }
    )
)
```

### TypeScript

```typescript
import { TraceRTMClient, ImportConfig, FieldMapping } from '@tracertm/sdk';
import * as fs from 'fs';

const client = new TraceRTMClient({ apiKey: 'rtm_live_sk_...' });

// Upload and import
async function importFromCSV() {
  // Upload file
  const fileBuffer = fs.readFileSync('requirements.csv');
  const upload = await client.files.upload(fileBuffer, 'requirements.csv');

  // Configure import
  const config: ImportConfig = {
    source: {
      type: 'file',
      format: 'csv',
      fileId: upload.id,
      options: {
        hasHeader: true,
        delimiter: ','
      }
    },
    mapping: {
      fields: {
        'ID': 'external_id',
        'Title': 'title',
        'Description': 'description',
        'Type': 'type',
        'Priority': {
          target: 'priority',
          transform: 'map',
          mapping: {
            'P1': 'critical',
            'P2': 'high',
            'P3': 'medium',
            'P4': 'low'
          }
        }
      },
      defaults: {
        status: 'draft'
      }
    },
    options: {
      mode: 'create_or_update',
      matchKey: 'external_id',
      onConflict: 'update'
    }
  };

  // Validate
  const validation = await client.imports.validate({
    projectId: 'proj_abc123',
    config
  });

  if (!validation.valid) {
    console.error(`Validation failed: ${validation.summary.invalidRows} invalid rows`);
    return;
  }

  // Start import
  const job = await client.imports.start({
    projectId: 'proj_abc123',
    config
  });

  // Wait for completion
  let status = job;
  while (['processing', 'queued'].includes(status.status)) {
    await new Promise(resolve => setTimeout(resolve, 2000));
    status = await client.imports.getStatus({
      projectId: 'proj_abc123',
      jobId: job.jobId
    });
    console.log(`Progress: ${status.progress.percentage}%`);
  }

  console.log(`Complete: ${status.progress.created} created, ${status.progress.updated} updated`);
}

// Import from Jira
async function importFromJira() {
  const job = await client.imports.start({
    projectId: 'proj_abc123',
    config: {
      source: {
        type: 'jira',
        connectionId: 'conn_jira_abc123',
        query: {
          project: 'PROJ',
          jql: 'type = Story AND status != Closed'
        }
      },
      mapping: {
        preset: 'jira_to_requirement'
      },
      options: {
        mode: 'sync',
        onConflict: 'update_if_newer'
      }
    }
  });

  return job;
}
```

---

## Field Transforms

| Transform | Description | Example |
|-----------|-------------|---------|
| `lowercase` | Convert to lowercase | `"HIGH"` → `"high"` |
| `uppercase` | Convert to uppercase | `"low"` → `"LOW"` |
| `trim` | Trim whitespace | `" value "` → `"value"` |
| `map` | Map values | `{"P1": "critical"}` |
| `regex_replace` | Regex replacement | Pattern-based transformation |
| `date_format` | Parse/format dates | `"01/20/2024"` → `"2024-01-20"` |
| `split` | Split into array | `"a,b,c"` → `["a","b","c"]` |
| `join` | Join array | `["a","b"]` → `"a, b"` |
| `default` | Set default if empty | Empty → `"unassigned"` |
| `concat` | Concatenate fields | `"{field1}-{field2}"` |

---

## Error Responses

### Invalid File Format

```json
{
  "error": {
    "code": "INVALID_FILE_FORMAT",
    "message": "File format not supported or file is corrupted",
    "details": {
      "expected_format": "csv",
      "detected_format": "unknown",
      "suggestion": "Ensure the file is a valid CSV with UTF-8 encoding"
    }
  }
}
```

### Validation Failed

```json
{
  "error": {
    "code": "VALIDATION_FAILED",
    "message": "Import validation failed with 77 errors",
    "details": {
      "error_count": 77,
      "first_errors": [
        {
          "row": 45,
          "field": "type",
          "message": "'INVALID' is not a valid item type"
        }
      ],
      "suggestion": "Fix validation errors or use validate_only mode to review all issues"
    }
  }
}
```

---

## Best Practices

### Pre-Import Checklist

1. **Validate first**: Always run validation before importing
2. **Use dry_run**: Test with dry_run mode to see what would happen
3. **Back up existing data**: Export current data before large imports
4. **Test with small sample**: Import a subset first to verify mappings

### Performance Tips

1. **Batch appropriately**: Use batch_size of 50-200 for optimal performance
2. **Limit concurrent imports**: One import job per project at a time
3. **Use async mode**: For imports > 1000 items, use asynchronous processing

---

## Related Documentation

- [Batch Operations](/docs/api-reference/rest-api/batch-operations) - Bulk operations
- [Webhooks](/docs/api-reference/rest-api/webhooks) - Import completion notifications
- [Items API](/docs/api-reference/rest-api/items) - Item management
