Robert Blumen - HashiTalks 2021 hashi.co/hashitalks2021

1. Lead DevOps Engineer - Salesforce
Robert Blumen
The 'Data,
Transformations,
Resources' Pattern in
Terraform

2. Outline
● Terraform anti-patterns
● New features in versions 12 and 13
● The "Data + Transforms + Resources" Pattern

3. What Terraform 11 Can Not Do
● Looping
● Conditional
● Nested loops
● Data structures

4. Data Types in Terraform
12 & 13

5. structured data types
● list
● map
● set
● tuple
● object

6. list(string)
list(tuple[string,number,bool])
map(string)
map(object({ id=string, cidr_block=string }))
composition of data types

7. variable group_members {
type = list(string)
description = "list of emails of the group members"
}
example

8. variable people {
type = list(object({ name=string, age=number }))
default = [
{
name = "John"
age = 32
}
]
}
type checking (continued)

9. ~/Devel/devops-tools (rblumen) $ /opt/hashi/terraform-0.12.20
apply -var 'people=[ { name="Job", age=71 } ]'
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
~/Devel/devops-tools (rblumen) $ /opt/hashi/terraform-0.12.20
apply -var 'people=[ { name="Job", age=true } ]'
Error: Invalid value for input variable
The argument -var="people=..." does not contain a valid value for variable
"people": element 0: attribute "name": string required.

10. groups = [
{
"key" : "value"
},
{
"key-2": "value-2"
}
]
all_groups = merge(local.groups...)
###
{
"key": "value",
"key-2": "value"
}
python-style splatting

11. Collections and
Iterations

12. terraform 11 - indexed lists
resource providerX web_thing {
count = var.num_servers
name = element(var.server_names,count.index)
}
#####
web_thing[0]
web_thing[1]
web_thing[2]
web_thing[3]

13. locals {
server_names = [ "db", "front-end", "back-end" ]
}
terraform 12+ - maps
web_srv["db"]
web_srv["front-end"]
web_srv["back-end"]
resource providerY web_srv {
for_each = toset(var.server_names)
name = each.value
}

14. old - (count based):
aws_iam_policy.aws_admin_access[3]
new - (for_each):
aws_iam_policy.aws_admin_access["AmazonS3FullAccess"]
indexing

15. locals {
administrator_policies = [
"AmazonEC2FullAccess",
"AmazonS3FullAccess",
"AWSKeyManagementServicePowerUser",
"AWSLambdaFullAccess",
]
}
data aws_iam_policy aws_admin_access {
for_each = toset(local.administrator_policies)
arn = "arn:aws:iam::aws:policy/${each.value}"
}
for_each - iterating over a list

16. resource heroku_app_feature app_feature {
for_each = {
for f, e in {
spaces-tls-legacy : false,
spaces-tls-modern : true
spaces-strict-tls : false
} :
f => e
}
app = heroku_app.app.name
name = each.key
enabled = each.value
}
for_each - iterating over a map

17. data aws_iam_policy_document access {
for_each = toset(local.groups)
statement {
effect = "Allow"
actions = ["s3:PutObject", ]
resources = data.aws_s3_bucket.devel.arn
}
}
chaining iteration
resource aws_iam_policy access {
for_each = data.aws_iam_policy_document.access
name = "${each.key}Access"
policy = each.value.json
}
resource aws_iam_group_policy_attachment group_access {
for_each = aws_iam_policy.access
group = each.key
policy_arn = each.value.arn
}

18. ● 12: for_each does not work with modules
● 13: for_each supports modules
terraform 12 versus 13

19. set operations

20. functions on sets
● setintersection
● setproduct
● setsubtract
● setunion

21. locals {
all_groups = toset([ "dev", "core", "admin", "ops", "chat" ])
some_groups = toset(["dev", "voice"])
other_groups = setsubtract(all_groups, some_groups) # in 12.21
}
resource aws_iam_group groups {
for_each = local.all_groups
name = each.value
}
resource aws_thing thing_one {
for_each = local.some_groups
group = aws_iam_group.groups[each.value]
}
resource aws_thing thing_two {
for_each = local.other_groups
group = aws_iam_group.groups[each.value]
}
implementing conditionals with sets

22. locals {
test_envs = [ "test1", "test2" ]
}
resource saas_thing_one test_only {
foreach = setintersection(toset([var.environment]), local.test_envs)
…
}
conditionals (2)

23. for expressions

24. for expressions
users = [ "achman", "aziss", "bwong", "cshah", ]
# list
emails = [
for u in local.users : "${replace(u, "-", ".")}@pagerduty.com"
]
# map
emails_by_user = {
for u in local.users :
u => "${replace(u, "-", ".")}@pagerduty.com"
}

25. [for s in var.list :
upper(s)
if s != ""]
for expressions - conditionals

26. var user_email_assn {
type = map(string)
default = { ... }
}
local {
labels = [ for user, email in var.user_email_assn: "${user}:${email} ]
}
for expressions - map

27. {
for s in [ "abc", "def", "aef", "dps" ] :
substr(s, 0, 1) => s...
}
{
"a" = [
"abc",
"aef",
]
"d" = [
"def",
"dps",
]
}
for expressions - group by

28. putting it together

29. resource backend_foo left_hand_thing {
for_each = ...
}
resource backend_bar right_hand_thing {
for_each = ..
}
resource backend_foo_bar left_right_assn {
for_each = ??
left_hand_thing = each.value.left
right_hand_thing = each.value.right
}
many-to-many

30. # groups have multiple members
# user may belong to more than one group
resource pagerduty_user { … }
resource pagerduty_team { … }
resource pagerduty_team_membership {
for_each = ??
user_id = ??
team_id = ??
}
example: PagerDuty

31. for g in groups {
create group
for u in groups.users {
create user
create create user_group_membership
}
}
conventional imperative language

32. locals {
on_call_teams = {
tier_1 = [ "abe", "jaxel", "beldon" ]
tier_2 = [ "abe", "janpax", "fanlo" ]
escalation = [ "adam", "shefty", "fanlo" ]
}
}
terraform solution

33. resource pagerduty_team teams {
for_each = keys(local.on_call_teams)
name = each.value
}
resource pagerduty_user users {
for_each = distinct(flatten(values(local.on_call_teams)))
name = each.value
email = "${each.value}@companyx.com"
}

34. locals {
team_user_pairs = [
for team, users in local.on_call_teams: [
for for user in users: {
user: user
team: team
}
]
]
team_users = {
for pair in flatten(local.team_user_pairs):
"${pair.team}-${pair.user}" => {
team_id: pagerduty_team[pair.team].id,
user_id: pagerduty_user[pair.user].id
}
}
}
resource pagerduty_team_membership membership {
for_each = local.team_users
team_id = each.value.team_id
user_id = each.value.user_id
}

35. locals {
on_call_teams = {
tier_1 = [ "abe", "jaxel", "beldon" ]
tier_2 = [ "abe", "janpax", "fanlo" ]
escalation = [ "adam", "shefty", "fanlo" ]
}
}
###
resource saas_team_membership membership {
for_each = ??
user = each.key
groups = each.value
}
one:many inverting

36. user_group_pairs = flatten([
for group, members in local.group_memberships : [
for user in members : {
user = user
group = group
}
]
])
groups_by_user = {
for pair in local.user_group_pairs :
pair.user => pair.group...
}
}

37. resource saas_team_membership membership {
for_each = local.groups_by_user
user = each.key
groups = each.value
}

38. The
"Data
+ Transformations
+ Resources"
Pattern

39. objectives
● understandable
● easy to change
● economical (DRY)

40. ● data
○ structure according to your domain
○ use data structures
○ normalize/minimize duplication
○ optimize for frequent changes
● transform
○ use for, if and set functions
○ inputs: your data
○ outputs: what your provider's resources need
● resources
○ create each resource once
○ chain resources
pattern for code organization

41. Thank You