# 🚀 Terraback

**Multi-Cloud Infrastructure as Code Tool**

Terraback is an advanced CLI tool that reverse-engineers existing cloud infrastructure into clean, production-ready Terraform code with intelligent dependency discovery.

**Transform legacy cloud environments into infrastructure-as-code in minutes, not months.**

---

## ✨ Why Terraback?

- **🎯 90% Faster**: Reduce infrastructure documentation time from weeks to hours  
- **🧠 Intelligent Discovery**: Automatic dependency mapping across 50+ cloud services  
- **🏢 Enterprise-Ready**: Production-ready templates with lifecycle management  
- **Multi-Cloud**: Full support for AWS, Azure, and GCP  
- **🔒 Security-First**: Read-only permissions, no credential storage, enterprise compliance

---

## 🌟 Features

### 📊 Comprehensive Cloud Coverage

**Core Infrastructure (Community Edition - Free):**

- **AWS**: EC2 Instances, VPCs, Subnets, Security Groups, S3 Buckets, IAM Roles
- **Azure**: Virtual Machines, Virtual Networks, Subnets, Network Security Groups, Storage Accounts
- **GCP**: Compute Instances, Networks, Subnets, Storage Buckets
- **Basic Commands**: list, import, scan
- **✨ Unlimited core resources**
- **Basic dependency mapping**

**Advanced Services (Professional & Enterprise):**

**AWS Services:**
- **Container Platform**: ECS Clusters, Services, Task Definitions, ECR Repositories
- **Advanced Networking**: CloudFront CDN, Route 53 DNS, NAT/Internet Gateways, VPC Endpoints
- **Database & Caching**: RDS instances, ElastiCache Redis/Memcached clusters
- **Monitoring & Security**: CloudWatch, Auto Scaling, ACM Certificates
- **Serverless & APIs**: Lambda Functions, API Gateway, SQS, SNS
- **Storage**: EFS, EBS Volumes/Snapshots, S3 advanced features
- **Load Balancing**: ALB/NLB with advanced features, listener rules, SSL policies

**Azure Services:**
- **Compute**: Virtual Machines with OS detection, Managed Disks, SSH Keys
- **Networking**: Virtual Networks, Subnets with delegations, NSGs with rules
- **Storage**: Storage Accounts with blob properties, static websites, encryption
- **Load Balancing**: Application Gateway, Load Balancers
- **Database**: Azure SQL, Cosmos DB
- **Container**: AKS, Container Instances

**GCP Services:**
- **Compute**: VM Instances, Persistent Disks, Machine Images
- **Networking**: VPC Networks, Subnets, Firewall Rules
- **Storage**: Cloud Storage, Persistent Disks
- **Container**: GKE Clusters, Node Pools

### 🔗 Intelligent Dependency Discovery

The `--with-deps` flag automatically discovers and maps complete infrastructure stacks:

```bash
# AWS Example
terraback ec2 scan --with-deps
# Automatically finds and scans:
# ├── VPC and networking (subnets, security groups)
# ├── Load balancers (ALB, target groups, listeners)
# ├── Storage (EBS volumes, snapshots)
# ├── Security (IAM roles, certificates)
# ├── Monitoring (CloudWatch logs, alarms)
# └── All interconnected dependencies

# Azure Example
terraback vm scan --with-deps
# Automatically finds and scans:
# ├── Virtual Networks and Subnets
# ├── Network Security Groups with rules
# ├── Managed Disks and Snapshots
# ├── Network Interfaces and Public IPs
# ├── Storage Accounts
# └── All interconnected dependencies
```

### 🚀 Advanced Features

- **Performance Optimization**: API response caching, parallel scanning, smart dependency resolution
- **Multi-Account/Subscription**: Scan across multiple AWS accounts or Azure subscriptions
- **Module Generation**: Create reusable Terraform modules automatically
- **State Management**: Import existing resources directly into Terraform state
- **Compliance Ready**: Generate code following HashiCorp and cloud provider best practices
- **Name Deduplication**: Automatically append numeric suffixes to resource names when duplicates are detected
- **SSH Key Sanitization**: Generated public keys are collapsed to a single line automatically

---

## 📦 Installation

### Prerequisites
- Python 3.8+ (for pip installation)
- Terraform 1.0+ (for import functionality)
- Cloud CLI tools:
  - AWS: `aws` CLI configured
  - Azure: `az` CLI configured and logged in
  - GCP: `gcloud` CLI configured
 - *(Optional)* [`xxhash`](https://pypi.org/project/xxhash/) for faster cache hashing

### Install via pip (Recommended)

```bash
pip install terraback
```

### Install via Binary

**Linux/macOS:**
```bash
# Linux
curl -L https://dist.terraback.dev.io/releases/latest/terraback-linux -o terraback
chmod +x terraback
sudo mv terraback /usr/local/bin/

# macOS
curl -L https://dist.terraback.dev.io/releases/latest/terraback-macos -o terraback
chmod +x terraback
sudo mv terraback /usr/local/bin/
```

**Windows:**
Download from [releases](https://dist.terraback.dev.io/releases/latest/terraback-windows.exe)

---

## 🚀 Quick Start

### AWS Scanning

```bash
# Configure AWS credentials (if not already done)
aws configure

# Scan EC2 instances
terraback ec2 scan

# Scan VPC with all dependencies
terraback vpc scan --with-deps

# Scan specific region
terraback ec2 scan --region eu-west-1

# Scan with specific profile
terraback s3 scan --profile production
```

### Azure Scanning

```bash
# Login to Azure (if not already done)
az login

# Scan Virtual Machines
terraback vm scan

# Scan entire resource group
terraback vm scan --resource-group production-rg

# Scan specific subscription
terraback vnet scan --subscription-id YOUR-SUBSCRIPTION-ID

# Scan with location filter
terraback storage scan --location westeurope
```

### Multi-Cloud Commands

```bash
# Scan all resources (cloud-specific)
terraback scan-all aws --region us-east-1
terraback scan-all azure --resource-group my-rg
terraback scan-all gcp --region us-central1 --cache-ttl 30

# Check authentication status
terraback auth-check

# Use caching for large infrastructures
terraback scan-recursive ec2 --use-cache --cache-ttl 120
terraback scan-recursive vm --use-cache --parallel-workers 10
```

After scanning completes, Terraback automatically runs `terraform fmt`, `terraform init` and `terraform validate` on the generated code. Use `--skip-check` with any `scan-all` command to bypass this validation step. If you skip the checks, **run `terraback aws scan-all --check` (or `python -m terraback.utils.template_syntax_fixer <output_dir>`) before invoking `terraform init` manually**.

---

## 📋 Supported Resources

### AWS Resources (Full Support)

| Service | Resources | Status |
|---------|-----------|---------|
| **EC2** | Instances, Volumes, Snapshots, AMIs, Key Pairs, Launch Templates, Network Interfaces | ✅ Full Support |
| **VPC** | VPCs, Subnets, Security Groups, Internet/NAT Gateways, Route Tables, VPC Endpoints | ✅ Full Support |
| **IAM** | Roles, Policies, Instance Profiles | ✅ Full Support |
| **S3** | Buckets, Versioning, Lifecycle, ACLs, Policies | ✅ Full Support |
| **RDS** | Instances, Subnet Groups, Parameter Groups | ✅ Full Support |
| **Load Balancing** | ALB, NLB, CLB, Target Groups, Listeners, SSL Policies | ✅ Full Support |
| **Lambda** | Functions, Layers, Permissions | ✅ Full Support |
| **Route 53** | Hosted Zones, Records | ✅ Full Support |
| **CloudWatch** | Log Groups, Alarms, Dashboards | ✅ Full Support |
| **Auto Scaling** | Groups, Launch Templates, Policies | ✅ Full Support |
| **ECS/ECR** | Clusters, Services, Task Definitions, Repositories | ✅ Full Support |
| **CloudFront** | Distributions, Origin Access Controls, Cache Policies | ✅ Full Support |
| **ElastiCache** | Redis/Memcached Clusters, Parameter Groups | ✅ Full Support |
| **ACM** | Certificates | ✅ Full Support |
| **EFS** | File Systems, Mount Targets, Access Points | ✅ Full Support |
| **API Gateway** | REST APIs, Resources, Methods, Deployments | ✅ Full Support |
| **SQS/SNS** | Queues, Topics, Subscriptions | ✅ Full Support |
| **Secrets Manager** | Secrets, Versions | ✅ Full Support |
| **Systems Manager** | Parameters, Documents | ✅ Full Support |

### Azure Resources

| Service | Resources | Status |
|---------|-----------|---------|
| **Compute** | Virtual Machines, Managed Disks, Availability Sets, SSH Keys | ✅ Full Support |
| **Networking** | Virtual Networks, Subnets, Network Security Groups, Network Interfaces | ✅ Full Support |
| **Storage** | Storage Accounts, Blob Storage, File Shares | ✅ Full Support |
| **Load Balancing** | Load Balancers, Application Gateways | ✅ Full Support |
| **Database** | SQL Database, Cosmos DB | ✅ Full Support |
| **Container** | AKS, Container Instances | ✅ Full Support |
| **Identity** | Managed Identities, Service Principals | ✅ Full Support |

### GCP Resources

| Service | Resources | Status |
|---------|-----------|---------|
| **Compute Engine** | Instances, Disks, Images | ✅ Full Support |
| **VPC** | Networks, Subnets, Firewalls | ✅ Full Support |
| **Cloud Storage** | Buckets, Objects | ✅ Full Support |
| **GKE** | Clusters, Node Pools | ✅ Full Support |

---

## 🎯 Advanced Usage

### Dependency Scanning

```bash
# AWS: Scan EC2 with all dependencies
terraback ec2 scan --with-deps --output-dir ./infrastructure

# Azure: Scan VM with all dependencies  
terraback vm scan --with-deps --output-dir ./infrastructure

# Result: Complete infrastructure graph
# ├── compute/
# │   ├── instances.tf
# │   └── launch_templates.tf
# ├── networking/
# │   ├── vpc.tf
# │   ├── subnets.tf
# │   └── security_groups.tf
# └── storage/
#     ├── volumes.tf
#     └── snapshots.tf
```

### Enterprise Module Generation

Transform generated Terraform into production-ready enterprise modules with standardized structure:

```bash
# Generate enterprise modules (requires Enterprise license)
terraback scan all aws --enterprise-modules

# Result: Organized module structure
# modules/
# ├── alb/
# │   ├── lb.tf
# │   ├── listeners.tf
# │   ├── global-accelerator.tf
# │   ├── variables.tf
# │   ├── outputs.tf
# │   └── locals.tf
# ├── ecs/
# │   └── service/
# │       ├── ecs-service.tf
# │       ├── ecs-task-definition.tf
# │       ├── autoscaling.tf
# │       ├── variables.tf
# │       ├── outputs.tf
# │       └── locals.tf
# ├── rds-aurora-postgresql/
# │   ├── cluster.tf
# │   ├── instances.tf
# │   ├── parameter-groups.tf
# │   ├── variables.tf
# │   ├── outputs.tf
# │   └── locals.tf
# └── s3-bucket/
#     ├── bucket.tf
#     ├── policy.tf
#     ├── versioning.tf
#     ├── lifecycle-configuration.tf
#     ├── variables.tf
#     ├── outputs.tf
#     └── locals.tf

# Works with all providers
terraback scan all azure --enterprise-modules
terraback scan all gcp --enterprise-modules
```

**Features:**
- Automatically groups related resources into logical modules
- Creates standard module files (variables.tf, outputs.tf, locals.tf)
- Follows Terraform best practices and naming conventions
- Supports 150+ AWS, Azure, and GCP resource types
- Maintains proper resource dependencies
- Production-ready module structure out of the box

**Supported Module Types:**
- **AWS**: ALB, ASG, ECS, RDS, Lambda, S3, CloudFront, OpenSearch, and 40+ more
- **Azure**: VMs, AKS, App Services, Databases, Networking, Storage, and 30+ more
- **GCP**: Compute, GKE, Cloud Run, Cloud SQL, Networking, Storage, and 30+ more

### Caching & Performance

```bash
# Enable caching for large infrastructures
terraback scan-all aws --cache-ttl 45
terraback scan-recursive ec2 --use-cache --cache-ttl 60

# View cache statistics
terraback cache stats

# Clear cache
terraback cache clear

# ``--cache-ttl`` sets how long scan results stay in the on-disk cache
# (in minutes) before expiring. This can dramatically speed up repeated
# ``scan-all`` operations on very large environments.
# For even faster cache hashing install ``xxhash`` (optional). Without it
# Terraback gracefully falls back to ``blake2s``.

# Parallel scanning
terraback scan-recursive vm --parallel-workers 10
```

### Import to Terraform

```bash
# List discovered resources
terraback list all

# Import specific resource
terraback ec2 import i-1234567890abcdef0

# Import Azure VM
terraback vm import /subscriptions/xxx/resourceGroups/my-rg/providers/Microsoft.Compute/virtualMachines/my-vm
# Bulk import all resources asynchronously with a progress bar
terraback import-all --async --progress
# Verify account and region before importing
terraback import-all --verify
# Set the Terraform lock timeout to 600 seconds
terraback import-all --lock-timeout 600
# Import using Terraform 1.5+ import blocks
terraback import-all --workflow import-blocks
# Automatically select the optimal workflow
terraback import-all --workflow auto --parallel=8
# Import via Terraform workspaces with 8 parallel workers
terraback import-all --workflow workspaces --parallel=8
# Import using enhanced import blocks in batches of 25
terraback import-all --workflow import-blocks-enhanced --batch-size=25
# Import using parallel workspaces with 10 workers
terraback import-all --workflow parallel-workspaces --parallel=10
```

### Template Formatting Check

After scanning or when editing `.tf` files manually, run **one** of the following commands **before running `terraform init`**:

```bash
terraback aws scan-all --check
# or
python -m terraback.utils.template_syntax_fixer <output_dir>
```

Before executing these validation commands, ensure the base Python dependencies
are installed. The template checker relies on packages from `requirements.txt`,
including **Jinja2**. Install them with:

```bash
pip install -r requirements.txt
```

This prevents "Missing newline after argument" errors.

Example command sequence:

```bash
terraback scan-all aws --skip-check --output-dir ./generated
terraback aws scan-all --check
cd generated
terraform init
```

The `--workflow` option controls how state is imported:

- **`auto`** – automatically selects the fastest strategy based on your Terraform version.

- **`import-blocks`** – for Terraform **1.5+** with a local backend, Terraback generates `import` blocks that Terraform applies in a single workspace.
- **`import-blocks-enhanced`** – groups resources into batches and applies each with Terraform. Use `--batch-size` and `--parallel` to control batch size and parallelism.
- **`workspaces`** – for Terraform **<1.5** or when using a **remote state backend** (S3, GCS, etc.), Terraback imports each resource in its own workspace.
- **`parallel-workspaces`** – runs workspace imports concurrently in temporary directories and merges their state automatically. Controlled by `--parallel`.

If `--workflow` is not specified, Terraback automatically chooses the best workflow based on your Terraform version.

### Fast & Parallel Workspace Workflows

Terraback provides two high-speed import strategies. The **`fast`** workflow
is the default. Resources are processed in batches (controlled by
`--batch-size`) and the tool automatically chooses between Terraform import
blocks (for Terraform 1.5+) and the temporary workspace approach. When
workspaces are used, each batch is imported concurrently using the number of
workers set by `--parallel`. The resulting state files are merged together with
`merge_states` and all `workspace_*` folders along with plan/debug files are
removed using `clean_import_artifacts`.

The **`parallel-workspaces`** workflow exposes the workspace method directly.
Each resource is imported in its own temporary directory. Once all imports
finish, the states are merged and the temporary directories are deleted.

```bash
# Fast workflow with batching and 8 parallel workers
terraback import-all --workflow fast --batch-size=50 --parallel=8

# Parallel workspaces with 10 workers
terraback import-all --workflow parallel-workspaces --parallel=10
```

---

## ⚖️ License & Pricing

Terraback uses a multi-tier licensing model:

### Community Edition (Free)
- ✅ **Core resources for AWS, Azure, GCP**
- ✅ EC2, VPC, S3 (AWS)
- ✅ VMs, VNets, Storage (Azure)  
- ✅ Compute, Networks, Storage (GCP)
- ✅ **✨ Unlimited core resources**
- ✅ Basic dependency mapping
- ✅ Community support via GitHub
- ❌ Advanced services (RDS, Lambda, etc.)
- ❌ Multi-account support

### Professional License ($499 one-time, lifetime access)
- **All 50+ cloud services**
- Unlimited resources and accounts
- Advanced dependency mapping
- Multi-account/subscription scanning
- RDS, Lambda, EKS, and more
- Module generation and best practices
- State file management
- Priority email support
- API access for automation
- **Lifetime updates included**

### Enterprise Edition (Coming Soon)
- **Everything in Professional**
- ✅ **Enterprise Module Generation** - Automatic organization into production-ready modules
- ✅ Annual/multi-year licensing
- ✅ SSO integration (SAML, OIDC)
- ✅ Custom resource scanners
- ✅ On-premise deployment options
- ✅ SLA with guaranteed uptime
- ✅ Dedicated training & onboarding
- ✅ Dedicated customer success manager
- ✅ Volume licensing & team management
- ✅ Compliance reporting

---

## 📚 Documentation

- [Full Documentation](https://docs.terraback.io)
- [AWS Scanning Guide](https://docs.terraback.io/aws)
- [Azure Scanning Guide](https://docs.terraback.io/azure)
- [GCP Scanning Guide](https://docs.terraback.io/gcp)
- [API Reference](https://docs.terraback.io/api)
- [Examples & Tutorials](https://docs.terraback.io/examples)
- [AWS Secrets Manager templates](docs/aws_secretsmanager_templates.md)
- Scanning Secrets Manager resources automatically creates a `variables.tf` file with a `secret_value` variable stub.
- The stub provides a default placeholder so Terraform doesn't prompt during imports.
- Be sure to update this value in a `.tfvars` file before applying.

---

### Logging

Terraback respects two environment variables for logging:

- ``TERRABACK_LOG_LEVEL`` controls the log verbosity (e.g. ``DEBUG`` or ``INFO``).
- ``TERRABACK_LOG_FILE`` can specify a path to write logs to a file.

Set these variables before running any command to adjust log output.

---

## 🐛 Troubleshooting

### AWS Issues

```bash
# Check AWS credentials
aws sts get-caller-identity

# Use specific profile
export AWS_PROFILE=production

# Debug mode
terraback ec2 scan --debug
```

### Azure Issues

```bash
# Check Azure login
az account show

# Set subscription
az account set --subscription "My Subscription"

# List available subscriptions
az account list --output table
```

### Common Issues

1. **Permission Denied**: Ensure your cloud credentials have read access to resources
2. **Rate Limiting**: Use `--use-cache` flag for large infrastructures
3. **Module Not Found**: Install with `pip install -e .` for development

---

## 🚀 Roadmap

- [x] AWS support (50+ services)
- [x] Azure support (Core services)
- [x] GCP support (Core services)
- [ ] Terraform module generation
- [ ] Web UI dashboard
- [ ] CI/CD integrations

## 🏗️ Template System

Terraback generates Terraform files from a collection of Jinja2 templates. All
provider templates inherit from `templates/common/base_resource.tf.j2`, giving
them a shared set of helper macros and a common block to override. To create a
new template, extend the base file and implement the `resources` block:

```jinja
{% extends "common/base_resource.tf.j2" %}
{% from "common/base_resource.tf.j2" import render_common_tags %}

{% block resources %}
resource "aws_my_service" "{{ item.name | tf_resource_name }}" {
  name = "{{ item.name }}"
  {{ render_common_tags(item, indent=2) }}
}
{% endblock %}
```

### Complex-data macros

`templates/common/macros.j2` exposes utilities for rendering lists and nested
structures. Examples include:

- `render_block_list` – iterate over complex objects and emit nested blocks
- `render_json_list` – output JSON formatted lists
- `render_depends_on` – add a `depends_on` array when dependencies exist

These macros keep templates concise while handling deeply nested data.

### Automatic dependency generation

During scanning Terraback records relationships between resources. When a
template uses `render_depends_on`, the discovered dependencies are inserted
automatically. For example:

```jinja
resource "aws_launch_template" "example" {
  ...
{{ macros.render_depends_on(template.depends_on, indent=2) }}
}
```

If an IAM role and security group were detected as dependencies, the rendered
Terraform code becomes:

```hcl
depends_on = [
  aws_iam_role.example_role,
  aws_security_group.example_sg,
]
```

Terraback calculates these relationships automatically, but you can still supply
additional items in the `depends_on` list to enforce custom ordering.

## 🧪 Development & Testing

Install the additional packages used by the test suite:

```bash
pip install -r requirements-dev.txt
```

Then run all tests from the project root:

```bash
pytest
```

---

## 📝 Line Endings

This project enforces LF line endings for all text files. Git uses `.gitattributes` to normalize line endings and a pre-commit hook (`end-of-file-fixer`) ensures files end with a single newline.

## 📞 Support

- **Community**: [GitHub Discussions](https://github.com/terraback/terraback/discussions)
- **Issues & Feature Requests**: [GitHub Issues](https://github.com/terraback/terraback/issues)
- **Professional Support**: support@terraback.io
- **Enterprise Sales**: sales@terraback.io

---

## 🙏 Acknowledgments

Built with ❤️ by DevOps engineers who understand the pain of manual cloud documentation.

---

**Copyright © 2025 Terraback**
