Terraform Best Practices Series - Lessons from the Battlefield: Part 1

Strategic Insights from the Terraform Trenches Part 1

Throughout my journey of DevOps/Infrastructure spanning nearly five years, I've been deep in the trenches of Terraform, crafting cloud infrastructure from startups to sprawling enterprises alike. From rookie missteps to triumphant victories, I've had a front-row seat to the power of Terraform's best practices.

In the early days, I learned the hard way that hasty changes can lead to production hiccups. But these experiences, though challenging, became invaluable lessons. They drove home the importance of meticulous planning and the significance of thorough testing before hitting that apply button. Here is my list of top Terraform best practices I have learned through experience and from others.

Table of Contents

1. Building a clean and organised codebase

   1. Version control

   2. Using modules

   3. Environment-specific configurations

   4. Directory structure & naming

2. Effective variable management

   1. Using TF vars

   2. Using CLI for vars

3. State Management & Security

   1. Remote backend

   2. State locking

   3. Storing sensitive data

4. Plan & Automation

   1. Always planning first

   2. Automating your IaC

   3. Using IAM roles in your IaC

   4. Tagging strategies

5. Versioning practices

   1. Using CodeOwners file

   2. Releasing tagged versions

   3. Pinning minor provider versions

   4. Checking in .terraform.lock.hcl file

1. Building a Clean and Organised Codebase

When building complex infrastructure, maintaining a clean and organized codebase is crucial. Terraform offers a few strategies to achieve this:

1.1 - Use Version Control

Store your Terraform code in version control like GitHub/GitLab repositories to track changes, collaborate effectively, and enable rollbacks when needed.

1.2 - Use Modules for Reusability

Break down your infrastructure into reusable modules to encapsulate logic and make your code more maintainable. 👨‍💻

module "vpc" {
  source = "./modules/vpc"
  # module arguments
}

1.3 - Environment-Specific Configurations

Organize your configurations based on environments (e.g., dev, staging, prod) for clear separation and easy resource management. 🌐

terraform/
├── dev/
│ ├── main.tf
│ ├── variables.tf
├── staging/
│ ├── main.tf
│ ├── variables.tf
├── prod/
│ ├── main.tf
│ ├── variables.tf

1.4 - Follow a good directory structure & use descriptive naming

Organize your codebase using a consistent directory structure that reflects the layout of your infrastructure components. Choose meaningful and descriptive names for resources, modules, and variables to enhance clarity and understanding.

terraform/
├── modules/
│ ├── network/
│ │ ├── main.tf
│ │ ├── variables.tf
│ ├── database/
│ │ ├── main.tf
│ │ ├── variables.tf
├── environments/
│ ├── dev/
│ │ ├── main.tf
│ │ ├── variables.tf
│ ├── prod/
│ │ ├── main.tf
│ │ ├── variables.tf

Terraform real-world example directory structure

---

2. Effective Variable Management

Effective variable management simplifies configuration and allows for flexibility:

2.1 Use tfvars Files:

Store your variable definitions in separate .tfvars files for consistency and better predictability. 📁

# terraform.tfvars
region = "us-west1"
instance_type = "t2.micro"

2.2 Avoid Command-Line Variables

Instead of using command-line options, stick to default variable files for better consistency.

Example of CLI var (not recommended!!)

terraform apply -var 'region=us-east-1'

Using default vars file (recommended)

# variables/defaults.tfvars

region = "us-west-1"
instance_type = "t2.micro"

---
To run:

terraform apply -var-file=variables/defaults.tfvars

---

3. State Management and Security

3.1 - Leverage a Remote Backend

Use remote state backends (e.g., AWS S3, Google Cloud Storage) to store state files securely and enable teamwork 🔐

terraform {
  backend "s3" {
    bucket         = "my-terraform-state"
    key            = "terraform.tfstate"
    region         = "us-west-1"
    encrypt        = true
    dynamodb_table = "terraform-lock"
  }
}

3.2 - Lock Your State

Implement state locking to prevent conflicts when multiple users or processes attempt to modify infrastructure concurrently. 🔑

terraform {
  backend "s3" {
    bucket         = "my-terraform-state"
    key            = "terraform.tfstate"
    region         = "us-west-1"
    encrypt        = true
    dynamodb_table = "terraform-lock" >> state locking here
  }
}

3.3 - Store Sensitive Data Securely

Avoid storing sensitive information in your Terraform state, and instead, use external secret management tools like Secret Manager and data sources instead 🔐

resource "aws_instance" "coderco_ec2" {
  ami           = "ami-0c55b159cbfafe1f0"
  instance_type = "t2.micro"
  tags = {
    Name = "ExampleInstance"
  }

  user_data = <<-EOF
    #!/bin/bash
    echo "API_KEY=${data.aws_secretsmanager_secret_version.api_key.secret_string}" >> /etc/environment
  EOF
}

data "aws_secretsmanager_secret_version" "api_key" {
  secret_id = "my-api-key-secret-id"
}

---

4. Plan and Automation

4.1 - Always Plan First

Generate a plan and review it before applying changes to ensure understanding and prevent unintended consequences. 📝

# Generate a plan
terraform plan -out=tfplan

# Review the plan
terraform show -json tfplan

4.2 - CICD - Automate with Pipelines

Implement automated pipelines using tools like Jenkins, GitHub Actions, Cloud Build, or Terraform Cloud for consistent execution. 🛠️

name: Terraform CI/CD using GitHub Actions

on:
  push:
    branches:
      - main

jobs:
  terraform:
    runs-on: ubuntu-latest
    
    steps:
    - name: Checkout code
      uses: actions/checkout@v2
      
    - name: Set up Terraform
      uses: hashicorp/setup-terraform@v1
      with:
        terraform_version: 1.0.0  # Specify the desired Terraform version
        
    - name: Initialize Terraform
      run: terraform init
      
    - name: Generate Terraform plan
      run: terraform plan -out=tfplan
      
    - name: Apply Terraform changes
      run: terraform apply -auto-approve tfplan

4.3 - Use Service Account/IAM roles Credentials

In your CI/CD pipeline, inherit service account/IAM roles credentials from the executing service for secure automation.

4.4 - Implement Tagging Strategies

Employ a consistent tagging strategy for resources to facilitate organization, tracking, and cost allocation. 🏷️

---

5. Versioning practices

5.1 - Include an Owners File

Include an owner’s file in your repository to specify the responsible individuals or teams for each Terraform module or configuration. Create a CODEOWNERS file in your GitHub repo or GitLab repo and add these.

# OWNERS
# Responsible Teams for Terraform Modules

modules/aws-vpc: @network-team
modules/aws-eks: @infra-team
modules/aws-lambda: @dev-team

5.2 - Release Tagged Versions

When sharing your Terraform modules, release tagged versions to provide stability and enable others to reference specific versions.

git tag v1.0.0
git push origin v1.0.0

5.3 - Pin to Minor Provider Versions

When specifying provider versions, pin to minor versions to benefit from bug fixes and improvements while avoiding breaking changes.

provider "aws" {
  version = "~> 2.0"
  region  = "us-west-1"
}

## In this example, we've specified that we want to use version 2.0 of the AWS provider, allowing for any bug fix releases or minor updates within the 2.x series.

5.4 - Check in .terraform.lock.hcl file

Include the .terraform.lock.hcl file in your source control to track provider version selections for your configuration.

---

Here are just some of the practices I have picked up in my journey of learning Terraform. This is just the start and there is more to learn.

To be continued in part 2….