Imagine a terraform plan in a pipeline that was supposed to be a no-op coming back with unexplained changes. Nobody on the team opened a pull request. The workspace has not been touched since the cutover, and the modules are pinned. Somewhere between the console, an incident channel, and an AWS Support case, actual infrastructure walked away from the code that is supposed to describe it.
Terraform drift is measured, not guessed. You detect it by running terraform plan -refresh-only -detailed-exitcode on a schedule, parsing the resource_drift array out of the plan JSON, attributing each change to a principal with aws cloudtrail lookup-events, and then routing the item to a review gate that decides whether to import, revert, accept, or codify it [1][3][8]. What you do not do is wire the detection job straight into terraform apply and call it reconciliation.
This playbook covers the three drift failure modes, the commands that surface each one, a classification worksheet that decides who answers what before an apply, and the guardrails that reduce how often drift appears at all. Drift behavior is environment-specific, so measure yours before you pick a remediation policy.
The Console Edit That Outlived the Migration
Drift is a divergence between three representations of the same infrastructure: declared state in your HCL, recorded state in the backend (S3, HCP Terraform, or another remote backend), and observed state in the AWS API. Terraform's refresh step reconciles recorded state against observed state. The plan then compares that refreshed picture against declared state [1][2].
That distinction matters because the three failure modes look similar in a terminal and require different fixes. A tag someone edited in the console is a different problem from a NAT gateway created by a hotfix that never made it into a module, which is a different problem again from a state entry pointing at a resource that no longer exists.
You tell them apart by which artifact disagrees, not by how many lines the plan prints. Treat a long plan output as a classification task before it is a remediation task: sort every item into one of the three buckets, name an owner for each bucket, and only then decide what gets applied. After a large migration, out-of-band changes can accumulate during cutover. Our cloud modernization engagements make drift review part of ongoing operations instead of leaving it as a post-launch cleanup task.
Three Kinds of Drift, Three Different Fixes
Attribute drift is the familiar case. A load balancer listener rule was edited during an incident, an autoscaling policy was tuned by hand, an RDS parameter was changed to get through a load spike. Terraform sees a delta on a resource it already manages, and the refresh-only plan reports it under resource_drift in the JSON output [3].
Unmanaged resources are invisible to terraform plan. If a resource was never in state, Terraform has nothing to compare. These come from break-glass fixes, another team's script, or a support case that resolved by provisioning something. You find them by comparing an independent inventory (AWS Resource Explorer, or the configuration inventory recorded by AWS Config) against the resource addresses in your state files [7][9].
Orphaned or stale state is the quietest failure. A resource was deleted out of band, a module was refactored without moved blocks, or a feature-flag conditional was flipped and left an address behind. Plans show destroys or replaces that nobody intended, and terraform state list contains addresses no longer present in code.
| Drift type | Detection signal | Safe remediation | Decision owner |
|---|---|---|---|
| Attribute drift | resource_drift entries in refresh-only plan JSON [3] | Decide revert vs codify, then apply through normal PR flow | Module owner plus the change author |
| Unmanaged resource | Present in Resource Explorer inventory, absent from terraform state list [9] | import block in a PR, never an ad hoc import command [5] | Team that created it, with platform review |
| Orphaned state entry | Plan proposes destroy or replace of a resource nobody changed | terraform state rm only after confirming provider-side absence [10] | Platform or IaC owner |
| Refactor-induced move | Plan proposes destroy plus create of an equivalent resource | moved block, verified with a zero-change plan [6] | Author of the refactor |
| Provider-default change | Drift appears after a provider version bump | Pin, read the changelog, then codify the new default | Platform team |
The fourth row is the one teams miss. A destroy-and-create pair in a plan after a module refactor is not drift at all, and applying it can delete a live resource. moved blocks exist precisely so that renames stay renames [6].
How to Measure Drift in Your Own Environment
Start with a read-only job. terraform plan -refresh-only updates state to match the provider and reports differences without proposing configuration changes. -detailed-exitcode returns a machine-readable result: 0 for no changes, 1 for error, 2 for changes present [1][2].
Then convert the plan to JSON and extract the drift array. The JSON plan format documents resource_drift as the list of resources whose refreshed state differs from prior state [3].
#!/usr/bin/env bash
# Drift detection step. Fails soft, emits a report, never applies.
set -uo pipefail
terraform init -input=false -backend-config="key=${TF_KEY}"
terraform plan -refresh-only -input=false -lock=false \
-out=drift.tfplan -detailed-exitcode
CODE=$?
case "$CODE" in
0|2) ;;
*) echo "Plan error. Investigate before trusting this result."; exit 1 ;;
esac
terraform show -json drift.tfplan > drift.json
jq -r '
(.resource_drift // [])
| map({
address: .address,
type: .type,
changed: [ (.change.before // {}) as $b
| (.change.after // {}) as $a
| ($a | keys_unsorted[])
| select(($a[.] // null) != ($b[.] // null)) ]
})
| .[] | "\(.address)\t\(.type)\t\(.changed | join(","))"
' drift.json > drift-report.tsv
case "$CODE" in
0) echo "No drift detected." ;;
2) echo "Drift detected. Report attached; no apply performed." ;;
esac
exit 0If you run HCP Terraform, workspace health assessments perform scheduled drift detection at the workspace level, which removes the need to build and babysit a cron pipeline for every workspace [4]. Use whichever mechanism your team will actually keep green. A detection job that gets disabled after a few noisy weeks is worse than no job, because it creates false confidence.
Detection tells you what changed. It does not tell you who. Attribution comes from CloudTrail:
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=ResourceName,AttributeValue=sg-0a1b2c3d4e5f \
--start-time 2026-08-20T00:00:00Z \
--query 'Events[].{time:EventTime,user:Username,event:EventName}' \
--output tablePair that with AWS Config for continuous configuration recording and rule evaluation, and Resource Explorer for the unmanaged inventory diff [7][9].
The baseline signals worth tracking
Do not adopt someone else's drift benchmark. Establish your own baseline over several detection cycles, then set thresholds your team can defend:
- Drifted resource addresses per workspace, trended over time rather than read as a single snapshot
- Age of the oldest open drift item, which tells you whether the review queue is actually being worked
- Share of drift items with an identified change author, since unattributable changes take longer to investigate
- Count of unmanaged resources in the inventory diff, tracked as a separate backlog from attribute drift
- Repeat-offender resource types, which point at a missing module input or a missing guardrail
-detailed-exitcode result of 2 on a refresh-only run means declared and observed state differlookup-events result gives the reviewer a principal and timestamp to start fromterraform state list are unmanaged, not driftedimport block in a reviewed pull request leaves a reviewable trail that manual state surgery does notKeep Reconciliation Behind Review
A nightly job that detects drift and immediately applies the configuration can overwrite a deliberate incident response or an operational adjustment that has not reached the code yet.
Consider a hypothetical worth walking through with your on-call team. An engineer raises an RDS max_connections value overnight to stop connection exhaustion, intending to open a pull request in the morning. If a reconciliation job reverts that parameter an hour later, the saturation returns and the code still looks correct to everyone reading it. Moving the job to a different hour does not fix that. Requiring a human answer on any drift item whose effects reach past the single resource does.
Classify by reach and reversibility, not by item count. A batch of tag differences may be a housekeeping backlog. One change to a resource that other resources reference can affect availability. Use these questions as a worksheet and fill in the owner column from your own change policy:
- 1Does anything read this value? For metadata and labels, confirm whether cost reporting, tagging automation, or deployment tooling consumes the attribute before you revert it.
- 2Was this tuned against observed load? For autoscaling bounds, retention periods, and alarm evaluation windows, ask whether the code should adopt the new value instead of overwriting it.
- 3What references this resource? Inspect dependency references in the module and in
terraform graphoutput, then state plainly what stops working if the declared value replaces the observed one. - 4Is there a tested rollback, and who owns the data? For database parameters, capacity settings, and volumes, name the rollback path and the person who acknowledges it before scheduling the apply.
- 5Is any related incident still open? If the answer is yes, the item waits.
Write the answers into the pull request that carries the change. A drift item closed without a recorded answer will come back, because whatever produced it is still in place.
The Reconciliation Decision Matrix
Once an item has an owner and an answer, the remediation path is mechanical. Every drift item resolves into one of six choices, and each choice has a specific artifact your team can review later.
| Choice | When review lands here | Command or artifact | What the team records |
|---|---|---|---|
| Codify observed state | The manual change was correct and should persist | Edit HCL, confirm a zero-change plan | Link from the PR to the drift item and its trigger |
| Revert to declared state | The change was accidental or no longer needed | Normal PR plus apply, never a direct console reversal | Reason the observed value was rejected |
| Import unmanaged resource | Resource exists in the inventory diff, not in state | import block reviewed in a PR [5] | Which team created it and why it stayed outside code |
| Remove stale state entry | Provider confirms the resource is gone | terraform state rm after provider-side verification [10] | Evidence that the resource no longer exists |
| Record a move | Refactor produced destroy-plus-create | moved block, verified by a zero-change plan [6] | The old and new addresses |
| Accept and schedule | Correct change, but the apply needs a window | Dated entry in the drift queue | Scheduled window and the owner holding it |
Two habits keep this matrix honest. First, route every path through the team's reviewed change process instead of leaving the decision in a terminal session. Second, use a zero-change plan as the acceptance test. If the plan after remediation is not empty, the item is not closed.
Guardrails That Reduce Drift Before It Appears
Detection is a lagging indicator. Reduce recurring drift by making out-of-band changes inconvenient and reviewed, codified changes easy.
Start with the paths people actually use. If the fastest way to fix a saturated queue is the console, engineers will use the console. Give them a module input, a documented runbook, and a predictable review-and-apply pipeline so the approved route is also the practical route. Then add policy checks to the pipeline, using Sentinel in HCP Terraform or Open Policy Agent with Conftest against the JSON plan, so that categories such as unpinned providers or missing required tags fail in review rather than surfacing later in a drift report [3].
Instrument the inventory diff on the same schedule as the drift job. Unmanaged resources never appear in a Terraform plan, so a drift-only program has a blind spot exactly where break-glass work lands. Resource Explorer plus a script that reads every state file gives you that list, and AWS Config gives you the configuration timeline for whatever the diff surfaces [7][9].
Finally, put the queue somewhere your cloud team already works, with an owner and a due date on every item. If you need help establishing that operating model across AWS environments, talk with our cloud modernization team.
Frequently Asked Questions
What is Terraform drift, exactly?
It is a divergence between declared state in your HCL, recorded state in the backend, and observed state in the provider API. Terraform's refresh operation compares recorded state against the provider, and the plan compares that result against your configuration [1][2].
How do I detect drift without changing anything?
Run terraform plan -refresh-only -detailed-exitcode, then terraform show -json on the saved plan and parse the resource_drift array [1][3]. Do not chain an apply to that job.
How often should the detection job run?
Pick a cadence from your own change velocity rather than a published default. Start daily per workspace, look at how many items each run produces and how long the queue takes to clear, and adjust until the report is small enough that someone reads it every time. HCP Terraform health assessments give you a managed scheduling option if you do not want to maintain the cron layer [4].
Why does drift not show unmanaged resources?
Terraform can only compare resources that exist in state. A resource created outside Terraform has no state entry, so there is nothing to diff. Catch those with an inventory comparison between Resource Explorer output and the addresses in terraform state list [9].
When is `terraform state rm` the right answer?
Only after you have confirmed with the provider API or console that the resource is genuinely gone, and only through a reviewed change. Removing a state entry for a resource that still exists creates an unmanaged resource, which converts a visible problem into an invisible one [10].
Should drift remediation ever be fully automatic?
Automate detection, reporting, attribution, and queueing. Keep the apply behind a review gate whose scope your team defines in writing. If you decide to auto-remediate a narrow category such as tags, write down the resource types and attributes covered, and prove that no automation or reporting system reads those attributes first.
Who owns drift reconciliation in a managed AWS operating model?
Detection can be automated, but the queue needs a named owner and a review gate. Our managed AI and cloud operations practice can support monitoring, incident response, reporting, and reviewed updates, while our AWS delivery approach explains how we design those operating controls with AWS services.
Start With One Workspace This Week
Go back to that pipeline run with unexplained changes. The fix is not a bigger plan diff. It is a classification step and an owner.
In your next working session, pick the single workspace with the most production impact and do four things: add the refresh-only detection job from this article as a report-only step, run the Resource Explorer to terraform state list diff once by hand, open a tracking issue for every item the two produce, and name one person accountable for the queue. That converts an ambient concern into a concrete list the team can work.
Then track one metric: the age of the oldest open drift item. Not the count. Count can go up when detection improves, which makes it a noisy signal early on. Age shows whether the queue is actively managed and whether unresolved changes are accumulating before the next terraform apply.
References
[1]HashiCorp. Terraform CLI: terraform plan command reference. https://developer.hashicorp.com/terraform/cli/commands/plan
[2]HashiCorp. Terraform CLI: terraform refresh and refresh-only planning. https://developer.hashicorp.com/terraform/cli/commands/refresh
[3]HashiCorp. Terraform internals: JSON output format, including resource_drift. https://developer.hashicorp.com/terraform/internals/json-format
[4]HashiCorp. HCP Terraform workspace health assessments and drift detection. https://developer.hashicorp.com/terraform/cloud-docs/workspaces/health
[5]HashiCorp. Terraform language: import block. https://developer.hashicorp.com/terraform/language/import
[6]HashiCorp. Terraform language: moved block. https://developer.hashicorp.com/terraform/language/moved
[7]Amazon Web Services. AWS Config Developer Guide: What is AWS Config. https://docs.aws.amazon.com/config/latest/developerguide/WhatIsConfig.html
[8]Amazon Web Services. AWS CLI reference: cloudtrail lookup-events. https://docs.aws.amazon.com/cli/latest/reference/cloudtrail/lookup-events.html
[9]Amazon Web Services. AWS Resource Explorer User Guide. https://docs.aws.amazon.com/resource-explorer/latest/userguide/welcome.html
[10]HashiCorp. Terraform CLI: terraform state rm. https://developer.hashicorp.com/terraform/cli/commands/state/rm