Terraform: Infrastructure as Code – Handling External Changes

Situation

We aim to automate the provisioning of our entire cloud infrastructure using Terraform. The challenge arises when unexpected changes appear—whether they are made manually or caused by external tools.

Another scenario is when the infrastructure was originally set up differently and now needs to be transitioned to Terraform. How should we handle resources that were created outside of Terraform? The approach varies from case to case, and we’ll show you how.

Manual Changes and Terraform

Terraform is designed to coexist with other methods of creating and maintaining infrastructure. Not only that, but it also provides built-in concepts for integrating external resources—something that other tools, such as Bicep or templates, do not offer.

This article assumes a basic understanding of Terraform, especially its state. HashiCorp describes state as „Mapping to the real World“. I personally think of it as a digital twin.

So, what can you do if you need to reference an external resource? Or if something needs to be brought under Terraform’s control—added to its state—or removed? Or if a specific property is modified by other tools?

We will clarify this in our article, assuming that manual changes have been made. However, the same applies if the resources were created or modified by Terraform code using a different state, by other tools, or by a policy.

Referencing Manually Created Resources

Scenario:
A resource was created manually via the cloud portal.‍

Problem:
We need to reference it in Terraform. In our example, this is a Key Vault. The Key Vault itself should not be managed by Terraform, but Terraform needs access to a secret stored within it.

Solution 1:
Declare the Key Vault (and possibly the secret) as a data source in Terraform.

In data.tf:

data "azurerm_key_vault" "kv-manuell" {
  name                = "kv-ist-und-bleibt-manuell"
  resource_group_name = "rg"
}

data "azurerm_key_vault_secret" "dbpw" {
  name         = "database-password"
  key_vault_id = data.azurerm_key_vault.kv-manuell.id
}

‍

In maint.tf:

      secrets = {
        "db-pw" = {
          value = data.azurerm_key_vault_secret.dbpw.value
        }

‍

Solution 2:
Declare it as a variable in Terraform.

In .tfvars - File:

shared_keyvault_id                      = "/subscriptions/xxx/resourceGroups/yyy/providers/Microsoft.KeyVault/vaults/zzz"

‍

In main.tf:

resource "azurerm_role_assignment" "kv_app_reader" {
  scope                = var.shared_keyvault_id
  role_definition_name = "Key Vault Secrets User"
  principal_id         = xxx.yyy.principal_id
}

Integrating Manual Changes

Scenario:
‍During a joint debugging session, a Terraform-managed resource was modified manually in the cloud portal.

Problem:
The change will be reverted in the next Terraform run.

Solution:
Run terraform plan and manually adopt the displayed changes into the Terraform code until no further modifications are detected.

Integrating Manually Created Resources

Scenario:
During a joint debugging session, a new resource was created manually in the cloud portal.

Problem:
Terraform ignores the resource. It cannot be modified or referenced. However, from now on, it should be managed by Terraform.

Solution 1:
Import the resource using an import block.

Note:
Sometimes, Terraform has trouble finding the IDs.

In imports.tf:

import {
  to = azurerm_subnet.pgsql
  id = "/subscriptions/xxx/resourceGroups/yyy/providers/Microsoft.Network/virtualNetworks/zzz/subnets/aaa"
}

‍

In main.tf:

resource "azurerm_subnet" "pgsql" {
  # If desired, you can leave the body of the resource block blank for now 
  # and return to fill it in once the instance is imported.
}

‍

Solution 2:
‍Recreate the resource in Terraform, as described in Integrating Manual Changes.

Terraform Should Ignore Changes To Specific Resource Properties

Scenario:
‍A policy automatically adds tags to resources.

Problem:
The tags keep getting changed back and forth. Terraform plans always detect these changes, cluttering the output.

Solution:
Use lifecycle ignore_changes.

Example:

  lifecycle {
    ignore_changes = [tags,]
  }

‍

Removing Resources From Terraform Management

Scenario:
A resource should remain temporarily for a developer to review, but they will delete it manually later.

Problem:
If we remove it from Terraform code, it will be deleted immediately.

Solution:
‍Completely remove the resource from Terraform’s management. Terraform has supported the terraform state rm command for a while. Since version 1.7, there is also the removed block.

Note:
For me, the removed block worked, but the command did not.

Example:

removed {
  from = module.containerapp.azurerm_container_app.container_app

  lifecycle {
    destroy = false
  }
}

Protecting Manual Work From Terraform

Scenario:
A team was provided with a VM using Terraform, on which they manually installed additional software.

Problem:
Terraform might recreate the VM, wiping out their manual work.

Solution 1:
‍Lock the resource by setting a change or delete lock in the portal.

Solution 2:
Use the removed block, as described in Removing Resources from Terraform Management.

Solution 3:
‍Enable soft delete for certain resources. Soft delete poses a risk: deletions may go unnoticed until the resource is permanently removed. Additionally, Terraform often disables soft delete, as it conflicts with the Infrastructure as Code philosophy, where everything should be rebuilt from scratch when needed.

Conclusion

Terraform can be used alongside other tools, allowing a gradual migration of an existing system to Terraform. This aligns with the Strangler Fig Pattern by Martin Fowler.

Our solution generally works well but can become complex, requiring patience and careful handling.

Isn’t it fascinating that the tool allowing other approaches to coexist is the one that dominates today?

Do you have a similar challenge or need a sparring partner for your infrastructure? Feel free to reach out—our DevOps experts look forward to your challenge!