AWS & Terraform8 min read•September 2026•Kernelry Staff

Cloud Infrastructure & Terraform (IaC) Architecture Guide

Infrastructure as Code (IaC) is no longer an optional skill for DevOps and platform engineers. Production infrastructure demands the ability to architect secure multi-account AWS topologies, manage state files in distributed teams, handle concurrent locks, and import legacy infrastructure without causing outages.

Architecture Blueprint

Distributed IaC State Locking & Cloud Reconciliation Flow

Visualizing how CI systems acquire DynamoDB distributed locks, read encrypted S3 state files, and reconcile drift against Cloud APIs.

Architecture Blueprint: Distributed State Locking & Drift ReconciliationTerraform • OpenTofu

The Core Concept

Infrastructure as Code (IaC) lets you define your cloud infrastructure in code. Terraform compares what currently exists in the cloud against your code and a stored state snapshot, then makes the necessary API calls to create, update, or remove resources.

Key Rules of Thumb for Production Systems

•Terraform state locking (via DynamoDB / GCS) prevents concurrent applies from corrupting the single source of truth.

•Service Control Policies (SCPs) set hard organizational guardrails; IAM Permission Boundaries constrain delegated roles.

•Transit Gateway scales hub-and-spoke multi-VPC routing; VPC Peering is ideal for point-to-point high-throughput links without hourly gateway fees.

•Never run terraform apply in production without an artifact-pinned terraform plan file.

Why Production Systems Demand This Depth

A bad Terraform apply can delete a production database or break VPC peering connections in seconds. Operating safely at scale requires mastering state locking, validating plans before applying, limiting blast radius, and enforcing IAM privilege boundaries.

Common Diagnostic Pitfalls

•Running terraform apply directly in production without an artifact-pinned plan file generated during code review.
•Confusing IAM Permission Boundaries with Service Control Policies (SCPs apply across AWS Accounts; boundaries apply to individual IAM identities).
•Over-using VPC Peering when Transit Gateway hub-and-spoke topology is required to avoid full-mesh peering routing complexity.
•Storing plaintext secrets in Terraform variables (state files store output values in plaintext JSON on disk/S3).

Best Practices for Scenario Drills

•Always advocate for remote state backends with S3 versioning, server-side encryption (KMS), and DynamoDB state locking.
•Mention Terraform Workspaces vs separate directory structures (prefer separate directories with Terragrunt or dedicated backend keys for production isolation).
•Explain how terraform import and moved blocks allow refactoring without resource destruction.

Worked Production Scenarios

Study real-world outage patterns, diagnostic trees, and clarifying questions.

Resolving Terraform State Locking and Concurrency Deadlocks

Difficulty: mid

Scenario Prompt:

“A CI/CD pipeline failed midway through a terraform apply due to a runner timeout. Subsequent runs fail with "Error acquiring the state lock". Developers cannot deploy hotfixes. How do you safely resolve this without risking state corruption?”

Substantive Technical Concepts to Hit:

Lock Verification

Verify in AWS DynamoDB (or Consul/GCS) that the lock ID matches the dead CI job and that no orphaned background process is still mutating cloud resources.

terraform force-unlock

Execute terraform force-unlock <LOCK_ID> with explicit audit logging and peer confirmation to release the lock safely.

Drift & State Refresh

Run terraform plan -refresh-only to detect what resources were created or modified before the runner timed out.

Atomic CI Architecture

Configure pipeline timeouts and trap handlers to ensure CI runners release locks on SIGTERM or cancellation signals.

Senior Clarifying Questions to Investigate:

Can we confirm with 100% certainty that the original timed-out runner process was killed and is not still applying changes?
What backend is storing our state lock (DynamoDB table, Terraform Cloud, or GCS)?
Do we have state versioning enabled in the S3 backend so we can rollback if corruption occurs?

Architecting Multi-VPC Cloud Connectivity: Transit Gateway vs Peering

Difficulty: senior

Scenario Prompt:

“An enterprise is scaling from 4 VPCs to 45 VPCs across multiple AWS accounts, including shared services and on-premise Direct Connect. The network team asks you to redesign the inter-VPC routing topology. What architecture do you recommend and why?”

Substantive Technical Concepts to Hit:

Transit Gateway Hub-and-Spoke

AWS Transit Gateway replaces complex full-mesh VPC peering with a centralized regional router, reducing route table management from O(N^2) to O(N).

Cost & Bandwidth Trade-offs

Transit Gateway incurs hourly attachment fees and per-GB data processing costs; high-volume point-to-point traffic (e.g. database replication) can remain on zero-cost peering.

Direct Connect & VPN Integration

Transit Gateway connects on-premises networks via a single Direct Connect Gateway association rather than per-VPC Virtual Private Gateways.

Network Segmentation via Route Domains

Use separate Transit Gateway Route Tables to isolate development VPCs from accessing production environments while sharing logging and telemetry VPCs.

Senior Clarifying Questions to Investigate:

What is our expected monthly data transfer volume between VPCs to weigh per-GB Transit Gateway costs?
Do we have regulatory requirements that mandate complete network isolation between certain account tiers?
Will we require cross-region Transit Gateway peering or are all 45 VPCs in a single region?

Interactive Scenario Drills

That's where we come in.

Explaining architectural trade-offs under time constraints is very different from reading about them.

Kernelry helps you rehearse technical questions, evidence-led debugging, safe next steps, and the clear communication teams expect. The drills adapt to your answer, so you practice the reasoning rather than memorize a company-specific script. During early access, free accounts include ten daily scenarios across our mixed question pool, with no credit card required.

Start Today's Free Drill

Back to All Blog Posts