Terraform with CI/CD

Objective

Integrate Terraform into a CI/CD pipeline to automate plan, review, and apply workflows for Cisco network infrastructure. You will wire a CI server to run Terraform in three stages (plan, manual review/approval, apply), and verify changes have been applied to a Cisco IOS device. In production, automated Terraform pipelines reduce human error, provide auditable change artifacts, and let teams enforce a mandatory review step before making network changes.

Quick Recap

Reference the topology used from lesson 1. This lesson adds one new device: a CI/CD server (CI1) that holds the Git repository and runs the Terraform pipeline.

ASCII topology (exact IPs shown on every interface)

[CI1]                        [R1]
eth0:192.168.10.100/24  -----  Gi0/0:192.168.10.1/24
CI1 runs Git + Terraform
R1 is the IOS router we manage with Terraform

Device Table

Device	Role	Interface	IP Address
R1	Cisco IOS router (managed device)	GigabitEthernet0/0	192.168.10.1/24
CI1	CI/CD server (runs Terraform pipeline)	eth0	192.168.10.100/24

Important: The router management interface Gi0/0 and the CI1 NIC are on the same management network. In production networks these would be on a secure management VLAN/trunk with additional access controls.

Key Concepts

• GitOps-style workflow (plan → review → apply): Terraform plan produces an execution plan that represents the changes but does not alter devices. The plan is saved as an artifact so reviewers can inspect exactly what will change. The apply stage consumes the plan and makes the changes.
  • Protocol behavior: Terraform uses the configured provider (SSH/HTTPS/NETCONF depending on provider) to push configuration to devices. The pipeline only orchestrates Terraform CLI — the provider performs the device-level sessions.
• Idempotency and state: Terraform keeps a state file to map declared resources to real-world devices. CI pipelines must protect and lock remote state (S3/Backend) to avoid concurrent applies corrupting state.
• Separation of duties / approvals: In production, an approval gate (manual job) prevents accidental immediate apply. This is commonly implemented as a protected pipeline stage or code review.
• Secrets and credentials: CI servers should never store device credentials in plaintext. Use secret stores/credentials vaults and inject them into pipeline steps at runtime.
• Verification & auditing: Pipeline artifacts (plan outputs, apply logs, state snapshots) serve as an audit trail. Network verification should include device show commands and state queries.

Step-by-step configuration

Step 1: Prepare the router for management access

What we are doing: Create a local user on the Cisco router for Terraform/CI to use via SSH and ensure vty is configured to accept SSH connections. This enables the CI server to authenticate and manage the router programmatically.

enable
configure terminal
hostname R1
ip domain-name lab.nhprep.com
username terraform privilege 15 secret Lab@123
line vty 0 4
 transport input ssh
 login local
exit
crypto key generate rsa modulus 2048
exit

What just happened:

• hostname R1 sets the router hostname (makes output easier to parse).
• ip domain-name lab.nhprep.com ensures the device has a domain for SSH keys and crypto to work.
• username terraform privilege 15 secret Lab@123 creates a local account used by CI to authenticate over SSH. Using a secret stores the password in hashed form in config.
• line vty 0 4 ... login local forces remote SSH sessions to use the local username database.
• crypto key generate rsa generates the host key necessary for SSH server operation. Without a host key, SSH will not start or clients will reject the host identity.

Real-world note: In production, prefer SSH key-based auth and a central AAA (Tacacs+/RADIUS) for accountability. Local users are acceptable for lab/demo scenarios.

Verify:

show running-config | include username|line vty|hostname|ip domain-name
hostname R1
ip domain-name lab.nhprep.com
username terraform privilege 15 secret 5 $1$abc$xxxxxxxxxxxxxxxxxxxxxx
line vty 0 4
 transport input ssh
 login local

Step 2: Create the Git repository and Terraform directory on CI1

What we are doing: Initialize a Git repository on the CI server containing Terraform code from prior lessons (the configuration that manages R1). The pipeline will use this repo to run plan and apply.

On CI1 (Linux shell):

# Create repo directory
mkdir -p /opt/terraform/ci-repo
cd /opt/terraform/ci-repo

# Initialize Git (assumes Git installed)
git init
cat > main.tf <<'EOF'
# Placeholder: Terraform configuration that manages R1
# The real Terraform provider and resources from earlier lessons go here.
# Example: resources to configure a loopback interface and a description

# NOTE: This file should match the Terraform code developed in earlier lessons.
EOF

git add main.tf
git commit -m "Initial Terraform code for R1"

What just happened:

• A repository was created to hold Terraform configuration. CI pipelines point to this repo. The main.tf file is the entry point for Terraform definitions. In a real lab you would include the exact resource blocks you used in Lesson 1–5.

Real-world note: Store Terraform in Git with branch protections; use Pull Requests for changes so the plan step can post diffs and reviewers can comment before apply.

Verify:

git status --porcelain

Expected output:

# If clean after commit, no output. To illustrate, show last commit:
git log -1 --pretty=oneline

Example expected output:

e3a1b7f8ea6b4c2d9b0f2f8d0a1c2e3d4f5a6b7 Initial Terraform code for R1

Step 3: Create a CI pipeline that runs plan, requires approval, then apply

What we are doing: Define a three-stage pipeline: plan (automated), manual approval (human), apply (automated). The plan stage runs terraform init and terraform plan -out=plan.tfplan and stores the plan file as an artifact reviewers can download. The apply stage uses the exact plan artifact to ensure what was reviewed is what gets applied.

Example generic CI YAML (name it pipeline.yml in the repo):

stages:
  - plan
  - approve
  - apply

plan:
  stage: plan
  script:
    - terraform init -input=false
    - terraform plan -input=false -out=plan.tfplan
    - terraform show -no-color plan.tfplan > plan.txt
  artifacts:
    paths:
      - plan.tfplan
      - plan.txt

approve:
  stage: approve
  when: manual
  script:
    - echo "Manual approval required to proceed to apply."
  allow_failure: false

apply:
  stage: apply
  script:
    - terraform init -input=false
    - terraform apply -input=false plan.tfplan
  dependencies:
    - plan
  when: on_success

What just happened:

• plan runs Terraform to produce a deterministic plan saved to plan.tfplan and a human-readable plan.txt. plan.tfplan is the binary plan artifact that must be used by terraform apply to avoid drift between review and execution.
• approve is a manual gate. The pipeline waits for a human to press "Approve" in the CI UI.
• apply consumes the exact plan artifact. This guarantees the applied changes match the reviewed plan.

Real-world note: Use your CI server's protected branch/pipeline features so only authorized users can trigger the apply stage. Also integrate your secrets manager so username/password are injected via environment variables, not stored in Git.

Verify: Run the plan stage locally (simulating the CI) from the repo root on CI1:

terraform init -input=false
terraform plan -input=false -out=plan.tfplan
terraform show -no-color plan.tfplan | head -n 40

Expected sample output (partial — shows planned changes in human-readable form):

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Plan: 1 to add, 0 to change, 0 to destroy.

# Example detail:
 + cisco_interface.loopback0
     name: "Loopback0"
     ip_address: "10.0.0.1/32"
     description: "Managed by Terraform via CI"

Step 4: Review plan and approve (manual step), then apply in CI

What we are doing: The reviewer inspects plan.txt artifact (or the CI UI) and then triggers the apply stage. The apply stage executes terraform apply plan.tfplan which communicates with network providers to make the changes.

Simulate the manual approval and apply locally on CI1:

# Simulate reviewer inspecting the plan
cat plan.txt

# Approve and apply (simulating the CI apply job)
terraform apply -input=false plan.tfplan

What just happened:

• terraform apply plan.tfplan reads the exact saved plan and executes provider operations to bring the real-world device state in line with the declared configuration. The provider opens device sessions (SSH/NETCONF/etc.) and performs idempotent changes. If the provider is a Cisco IOS provider, it will open an SSH session as terraform and issue the required configuration commands (entering config mode, making interface changes, saving state if configured).

Real-world note: Always apply using the plan artifact so that what was approved is guaranteed to be what is executed. Avoid re-running terraform plan at apply time as that can introduce race conditions.

Verify:

1. Show Terraform state for the resource just created/modified:

terraform state list

Expected output (example):

cisco_interface.loopback0

Show detailed state:

terraform state show cisco_interface.loopback0

Expected output (example):

# cisco_interface.loopback0:
resource "cisco_interface" "loopback0" {
    id          = "Loopback0"
    name        = "Loopback0"
    ip_address  = "10.0.0.1/32"
    description = "Managed by Terraform via CI"
}

2. Verify on the router that the configuration exists (Cisco command):

show running-config interface Loopback0
interface Loopback0
 ip address 10.0.0.1 255.255.255.255
 description Managed by Terraform via CI
!

(If your Terraform resource created a different object, replace resource name accordingly. The important verification is the device config matches what Terraform declared.)

Step 5: Post-apply verification and audit artifacts

What we are doing: Archive artifacts (plan, apply logs, terraform state snapshot) and run a quick device verification to confirm operational state. These artifacts form the audit trail for the change.

Commands on CI1:

# Archive artifacts (example)
tar -czvf artifacts_$(date +%F_%T).tar.gz plan.tfplan plan.txt terraform.tfstate

# Optional: sanity test from CI server to ping the new loopback
ping -c 3 10.0.0.1

What just happened:

• Artifacts were bundled for storage/forensics. The ping sanity test ensures traffic to configured endpoints works from the CI environment or a monitoring host.

Real-world note: In enterprise deployments you would push artifacts to an immutable artifact store and record the pipeline run ID in a change management system.

Verify: Show artifact list and ping output:

ls -l artifacts_*.tar.gz

Expected output:

-rw-r--r-- 1 ci ci 12345 2026-04-02T12:34:56 artifacts_2026-04-02_12:34:56.tar.gz

Ping output:

PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.
64 bytes from 10.0.0.1: icmp_seq=1 ttl=255 time=1.23 ms
64 bytes from 10.0.0.1: icmp_seq=2 ttl=255 time=0.98 ms
64 bytes from 10.0.0.1: icmp_seq=3 ttl=255 time=1.05 ms

--- 10.0.0.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 0.98/1.08/1.23/0.11 ms

Verification Checklist

• Check 1: CI server can SSH to R1 using the terraform user — verify with ssh terraform@192.168.10.1 (or show successful terraform provider connect logs).
• Check 2: Plan stage produces plan.tfplan and plan.txt artifacts — verify by running terraform plan -out=plan.tfplan and ls -l plan.tfplan plan.txt.
• Check 3: Apply stage used the exact plan and the device contains the expected config — verify via terraform state show <resource> and show running-config on the device.

Common Mistakes

Symptom	Cause	Fix
Plan shows changes but apply fails with authentication errors	CI job did not have correct credentials or SSH keys	Ensure CI secrets inject username and password/SSH key correctly; test ssh terraform@192.168.10.1 from CI node with same env vars
The plan in CI differs from what the reviewer sees locally	Reviewer used different workspace or local variables	Use consistent variable files and remote backend; always review plan.txt artifact generated by the CI plan stage
Terraform state gets corrupted due to concurrent applies	No remote state locking or multiple concurrent pipeline runs	Use a backend with state locking (e.g., S3 + DynamoDB, or Terraform Cloud) to prevent concurrent applies
Device configuration differs after apply	The provider failed mid-apply or device rejected some commands	Inspect provider/apply logs for errors; re-run apply using same plan artifact; ensure device ACL/exec limits allow configuration changes
SSH host key verification blocks CI job	CI host sees new host key and refuses connection	In CI, either populate known_hosts securely or use CI SSH known_hosts management; avoid disabling host key checking in production

Key Takeaways

• Use a plan → review → apply pipeline to separate change review from execution; always apply the exact plan artifact that reviewers approved. This reduces accidental drift between what is reviewed and what is applied.
• Protect Terraform state and credentials in CI: use remote backends with locking and secrets managers for credentials. This prevents concurrent state changes and credential leakage.
• Verification is mandatory: include both Terraform state checks and device-level show commands in pipeline post-apply checks to ensure the network actually reflects the declared state.
• In production, prefer key-based SSH authentication and central AAA for device access, and use an approval gate for high-risk changes to maintain separation of duties and compliance.

Tip: Think of the pipeline as a safety net — Terraform provides the declarative intent and the plan artifact is the "preview"; the CI pipeline enforces that preview is reviewed and that the apply uses the same preview. This provides repeatable, auditable, and safer network changes.