SlideShare ist ein Scribd-Unternehmen logo
1 von 38
Downloaden Sie, um offline zu lesen
Terraform Tips and
Tricks
LAOUC Community Tour 2022
August 17th
Nelson Calero
© Pythian Services Inc 2022 | 1
Our Speaker
Nelson Calero
Internal Principal Consultant
Pythian
http://www.linkedin.com/in/ncalero
@ncalerouy
© Pythian Services Inc 2022 | 2
24
Years in Business
_____________________
Premier Partner
140+ Certifications
8 Specializations
360+
Experts Across 5 Continents
500+
Customers Globally
_____________________
Advanced Partner
175+ Certifications
_____________________
Gold Partner
15+ Certifications
_____________________
Platinum Partner
150+ Certifications
_____________________
SAP Certified Partner
40+ Certifications
© Pythian Services Inc 2022 | 3
A G E N D A
● Terraform basics (1 slide)
● Terraform Idiosyncrasy
● Cloud provider Idiosyncrasy
© Pythian Services Inc 2022 | 4
Terraform basics
● Infrastructure as Code, declarative config files (HCL)
● Simple install - standalone binary file
● Few operations - plan/apply/destroy
● Providers - implement how to deal with your resources. 1000+
● State file - mapping of existing resources to TF configuration
● Simple structure
○ Reads code from current directory
○ Split code in files for readability: main.tf, variables.tf, outputs.tf
● Advanced configurations - workspaces, modules, remote state files
● Changes in versions - Features, state file format
© Pythian Services Inc 2021 | 5
Terraform basics
● Infrastructure as Code, declarative config files (HCL)
● Simple install - standalone binary file
● Few operations - plan/apply/destroy
● Providers - implement how to deal with your resources. 1000+
● State file - mapping of existing resources to TF configuration
● Simple structure
○ Reads code from current directory
○ Split code in files for readability: main.tf, variables.tf, outputs.tf
● Advanced configurations - workspaces, modules, remote state files
● Changes in versions - Features, state file format
© Pythian Services Inc 2021 | 6
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 3.27"
}
}
required_version = ">= 0.14.9"
}
provider "aws" {
profile = "default"
region = "us-west-1"
}
resource "aws_instance" "app_server" {
ami = "ami-830c94e3"
instance_type = "t2.micro"
tags = {
Name = "AppServerInstance"
}
}
● https://www.terraform.io/
● Lucas’ blog: https://technology.amis.nl/continuous-delivery/provisioning/terraform/
Terraform idiosyncrasy
1) State file relevance
2) Concurrent work on same resources
3) Reducing scope of operations
4) Resource parameters validation
5) Re-using code
6) Available features = TF + provider
© Pythian Services Inc 2021 | 7
Terraform idiosyncrasy
● state file drives the changes - make sure is in sync with your resources
● state file - removing resources (state rm), importing (import name_in_state name_aws)
● State file lock from previous canceled/timed out execution: force-unlock to remove it
● Plan and apply only one resource from many: -target name_in_state
● Conditional resources in modules: count
● Re-using code among environments and/or deployments? Welcome terragrunt. Examples:
○ Standardize resource attributes (tags, HA, networking, security, parameters) between deployments - in
same or different environments and regions
○ Support different database engines and versions
○ Validations (possible but limited to same attribute expression)
○ Modules, variables and environments - directory structure
○ Several other features - check https://terragrunt.gruntwork.io/
© Pythian Services Inc 2021 | 8
Terraform idiosyncrasy - 1) state file
Demo 1
Code taken from public Oracle repo (few changes for simplicity):
https://github.com/oracle/terraform-provider-oci/tree/master/examples/database/adb
© Pythian Services Inc 2021 | 9
Terraform idiosyncrasy - state file
© Pythian Services Inc 2021 | 10
$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
data.aws_security_group.sgpr: Refreshing state...
------------------------------------------------------------------------
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# aws_db_instance.master will be created
+ resource "aws_db_instance" "master" {
+ address = (known after apply)
+ allocated_storage = 500
+ apply_immediately = (known after apply)
...
+ identifier ="nelson-ora19"
...
Plan: 1 to add, 0 to change, 0 to destroy.
Terraform idiosyncrasy - state file
© Pythian Services Inc 2021 | 11
$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
data.aws_security_group.sgpr: Refreshing state...
------------------------------------------------------------------------
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# aws_db_instance.master will be created
+ resource "aws_db_instance" "master" {
+ address = (known after apply)
+ allocated_storage = 500
+ apply_immediately = (known after apply)
...
+ identifier ="nelson-pg14"
...
Plan: 1 to add, 0 to change, 0 to destroy.
$ aws rds describe-db-instances --region=us-west-1 --db-instance-identifier nelson-ora19
--query
"*[].{DBClusterIdentifier:DBClusterIdentifier,DBInstanceIdentifier:DBInstanceIdentifier,Engin
eVersion:EngineVersion}" --output table
--------------------------------------------------------------------------------------
| DescribeDBInstances |
+---------------------+------------------------+-------------------------------------+
| DBClusterIdentifier | DBInstanceIdentifier | EngineVersion |
+---------------------+------------------------+-------------------------------------+
| None | nelson-ora19 | 19.0.0.0.ru-2021-07.rur-2021-07.r1 |
+---------------------+------------------------+-------------------------------------+
Terraform idiosyncrasy - state file
© Pythian Services Inc 2021 | 12
$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
data.aws_security_group.sgpr: Refreshing state...
------------------------------------------------------------------------
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# aws_db_instance.master will be created
+ resource "aws_db_instance" "master" {
+ address = (known after apply)
+ allocated_storage = 500
+ apply_immediately = (known after apply)
...
+ identifier ="nelson-pg14"
...
Plan: 1 to add, 0 to change, 0 to destroy.
$ aws rds describe-db-instances --region=us-west-1 --db-instance-identifier nelson-ora19
--query
"*[].{DBClusterIdentifier:DBClusterIdentifier,DBInstanceIdentifier:DBInstanceIdentifier,Engin
eVersion:EngineVersion}" --output table
--------------------------------------------------------------------------------------
| DescribeDBInstances |
+---------------------+------------------------+-------------------------------------+
| DBClusterIdentifier | DBInstanceIdentifier | EngineVersion |
+---------------------+------------------------+-------------------------------------+
| None | nelson-ora19 | 19.0.0.0.ru-2021-07.rur-2021-07.r1 |
+---------------------+------------------------+-------------------------------------+
$ terraform import aws_db_instance.master nelson-ora19
aws_db_instance.master: Importing from ID "nelson-ora19"...
aws_db_instance.master: Import prepared!
Prepared aws_db_instance for import
aws_db_instance.master: Refreshing state... [id=nelson-ora19]
Import successful!
The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.
Terraform idiosyncrasy - state file
© Pythian Services Inc 2021 | 13
$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
data.aws_security_group.sgpr: Refreshing state...
------------------------------------------------------------------------
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# aws_db_instance.master will be created
+ resource "aws_db_instance" "master" {
+ address = (known after apply)
+ allocated_storage = 500
+ apply_immediately = (known after apply)
...
+ identifier ="nelson-pg14"
...
Plan: 1 to add, 0 to change, 0 to destroy.
$ aws rds describe-db-instances --region=us-west-1 --db-instance-identifier nelson-pg14
--query
"*[].{DBClusterIdentifier:DBClusterIdentifier,DBInstanceIdentifier:DBInstanceIdentifier,Engin
eVersion:EngineVersion}" --output table
--------------------------------------------------------------------------------------
| DescribeDBInstances |
+---------------------+------------------------+-------------------------------------+
| DBClusterIdentifier | DBInstanceIdentifier | EngineVersion |
+---------------------+------------------------+-------------------------------------+
| None | nelson-ora19 | 19.0.0.0.ru-2021-07.rur-2021-07.r1 |
+---------------------+------------------------+-------------------------------------+
$ terraform import aws_db_instance.master nelson-pg14
aws_db_instance.master: Importing from ID "nelson-pg14"...
aws_db_instance.master: Import prepared!
Prepared aws_db_instance for import
aws_db_instance.master: Refreshing state... [id=nelson-pg14]
Import successful!
The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.
$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
data.aws_security_group.sgpr: Refreshing state...
aws_db_instance.master: Refreshing state... [id=nelson-ora19]
------------------------------------------------------------------------
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
~ update in-place
Terraform will perform the following actions:
# aws_db_instance.master will be updated in-place
~ resource "aws_db_instance" "master" {
...
+ password = (sensitive value)
...
Plan: 0 to add, 1 to change, 0 to destroy.
Terraform idiosyncrasy - state file
© Pythian Services Inc 2021 | 14
$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
data.aws_security_group.sgpr: Refreshing state...
------------------------------------------------------------------------
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# aws_db_instance.master will be created
+ resource "aws_db_instance" "master" {
+ address = (known after apply)
+ allocated_storage = 500
+ apply_immediately = (known after apply)
...
+ identifier ="nelson-pg14"
...
Plan: 1 to add, 0 to change, 0 to destroy.
$ aws rds describe-db-instances --region=us-west-1 --db-instance-identifier nelson-pg14
--query
"*[].{DBClusterIdentifier:DBClusterIdentifier,DBInstanceIdentifier:DBInstanceIdentifier,Engin
eVersion:EngineVersion}" --output table
--------------------------------------------------------------------------------------
| DescribeDBInstances |
+---------------------+------------------------+-------------------------------------+
| DBClusterIdentifier | DBInstanceIdentifier | EngineVersion |
+---------------------+------------------------+-------------------------------------+
| None | nelson-ora19 | 19.0.0.0.ru-2021-07.rur-2021-07.r1 |
+---------------------+------------------------+-------------------------------------+
$ terraform import aws_db_instance.master nelson-pg14
aws_db_instance.master: Importing from ID "nelson-pg14"...
aws_db_instance.master: Import prepared!
Prepared aws_db_instance for import
aws_db_instance.master: Refreshing state... [id=nelson-pg14]
Import successful!
The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.
$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
data.aws_security_group.sgpr: Refreshing state...
aws_db_instance.master: Refreshing state... [id=nelson-pg14]
------------------------------------------------------------------------
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
~ update in-place
Terraform will perform the following actions:
# aws_db_instance.master will be updated in-place
~ resource "aws_db_instance" "master" {
...
+ password = (sensitive value)
...
Plan: 0 to add, 1 to change, 0 to destroy.
$ terraform apply
…
$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
data.aws_security_group.sgpr: Refreshing state...
aws_db_instance.master: Refreshing state... [id=nelson-ora19]
------------------------------------------------------------------------
No changes. Infrastructure is up-to-date.
This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.
Terraform idiosyncrasy - state file
● Default state file is terraform.tfstate in current directory
● Critical data, requires good backup policy
● terraform import is not always able to recreate all attributes (ex: passwords, MySQL grants)
● Two key complementary features:
○ remote state file
https://www.terraform.io/language/state/remote
○ OCI discovery
https://registry.terraform.io/providers/oracle/oci/latest/docs/guides/resource_discovery
© Pythian Services Inc 2021 | 15
Terraform idiosyncrasy - state file / remote
● Remote state file - https://www.terraform.io/language/state/remote
© Pythian Services Inc 2021 | 16
remote_state {
backend "s3" {
bucket = "terraform-states"
key = "networking/terraform.tfstate"
region = "us-phoenix-1"
endpoint = "https://acme.compat.objectstorage.us-phoenix-1.oraclecloud.com"
shared_credentials_file = "../terraform-states_bucket_credentials"
skip_region_validation = true
skip_credentials_validation = true
skip_metadata_api_check = true
force_path_style = true
}
}
Ref: https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/terraformUsingObjectStore.htm
Terraform idiosyncrasy - state file / OCI discovery
Demo 2
OCI discovery docs: https://registry.terraform.io/providers/oracle/oci/latest/docs/guides/resource_discovery
© Pythian Services Inc 2021 | 17
Terraform idiosyncrasy - state file / OCI discovery
● terraform-provider-oci gets installed by terraform:
● Discover databases and create the state file:
© Pythian Services Inc 2021 | 18
$ ls -lrt .terraform/providers/registry.terraform.io/hashicorp/oci/4.77.0/linux_amd64/
total 129320
-rw-r--r-- 1 calero calero 96384 May 31 17:31 CHANGELOG.md
-rw-r--r-- 1 calero calero 16725 May 31 17:31 LICENSE.md
-rw-r--r-- 1 calero calero 3071 May 31 17:31 README.md
-rwxr-xr-x 1 calero calero 132177920 May 31 17:31 terraform-provider-oci_v4.77.0
$ mkdir discover
$ .terraform/providers/registry.terraform.io/hashicorp/oci/4.77.0/linux_amd64/terraform-provider-oci_v4.77.0
-command=export -compartment_id=$COMPID -services database -generate_state -output_path=discover/
...
$ ls -lrt discover
total 44
-rw-r--r-- 1 calero calero 10109 May 31 18:46 terraform.tfstate.tmp.backup
-rw-r--r-- 1 calero calero 22086 May 31 18:46 terraform.tfstate
-rw-r--r-- 1 calero calero 6085 May 31 18:46 database.tf
-rw-r--r-- 1 calero calero 38 May 31 18:46 provider.tf
-rw-r--r-- 1 calero calero 171 May 31 18:46 vars.tf
$
Terraform idiosyncrasy - state file / OCI discovery
● terraform-provider-oci gets installed by terraform:
● Discover databases and create the state file:
© Pythian Services Inc 2021 | 19
$ ls -lrt .terraform/providers/registry.terraform.io/hashicorp/oci/4.77.0/linux_amd64/
total 129320
-rw-r--r-- 1 calero calero 96384 May 31 17:31 CHANGELOG.md
-rw-r--r-- 1 calero calero 16725 May 31 17:31 LICENSE.md
-rw-r--r-- 1 calero calero 3071 May 31 17:31 README.md
-rwxr-xr-x 1 calero calero 132177920 May 31 17:31 terraform-provider-oci_v4.77.0
$ mkdir discover
$ .terraform/providers/registry.terraform.io/hashicorp/oci/4.77.0/linux_amd64/terraform-provider-oci_v4.77.0
-command=export -compartment_id=$COMPID -services database -generate_state -output_path=../discover/
...
$ ls -lrt ../discover2
total 44
-rw-r--r-- 1 calero calero 10109 May 31 18:46 terraform.tfstate.tmp.backup
-rw-r--r-- 1 calero calero 22086 May 31 18:46 terraform.tfstate
-rw-r--r-- 1 calero calero 6085 May 31 18:46 database.tf
-rw-r--r-- 1 calero calero 38 May 31 18:46 provider.tf
-rw-r--r-- 1 calero calero 171 May 31 18:46 vars.tf
$
$ cd discover
$ terraform plan
oci_database_autonomous_database.export_example_autonomous_data_warehouse: Refreshing state...
[id=ocid1.autonomousdatabase.oc1.iad.anuwcljst77gzoiaaewxote6h4i73oamzoloxdueabnpwprspzngjoce52xq]
oci_database_autonomous_database.export_example_autonomous_database: Refreshing state...
[id=ocid1.autonomousdatabase.oc1.iad.anuwcljtt77gzoiamz3gg5cfsbdqgcoj7lrobis75qp7lwubm7d77bpblqgq]
No changes. Your infrastructure matches the configuration.
Terraform has compared your real infrastructure against your configuration and found no differences,
so no changes are needed.
$
Terraform idiosyncrasy - 2) concurrent work
© Pythian Services Inc 2021 | 20
$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
data.aws_security_group.sgpr: Refreshing state...
aws_rds_cluster.master: Refreshing state... [id=nelson-auroramysql56]
aws_rds_cluster_instance.cluster_instances[0]: Refreshing state... [id=nelson-auroramysql56-0]
aws_rds_cluster_instance.cluster_instances[1]: Refreshing state... [id=nelson-auroramysql56-1]
mysql_user.monitor: Refreshing state... [id=monitor@%]
Error: Could not connect to server: dial tcp 127.0.0.1:3306: connect: connection refused
Terraform idiosyncrasy - concurrent work
© Pythian Services Inc 2021 | 21
$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
data.aws_security_group.sgpr: Refreshing state...
aws_rds_cluster.master: Refreshing state... [id=nelson-auroramysql56]
aws_rds_cluster_instance.cluster_instances[0]: Refreshing state... [id=nelson-auroramysql56-0]
aws_rds_cluster_instance.cluster_instances[1]: Refreshing state... [id=nelson-auroramysql56-1]
mysql_user.checkmk_mon: Refreshing state... [id=checkmk_mon@%]
Error: Could not connect to server: dial tcp 127.0.0.1:3306: connect: connection refused
$ terraform plan
...
Error locking state: Error acquiring the state lock: ConditionalCheckFailedException: The conditional
request failed
Lock Info:
ID: f6f04b96-de6d-754a-7bbe-8d0504f75e79
Path: my-tf-state-286052569778/test/us-west-1/services/my-db/aurora-mysql/terraform.tfstate
Operation: OperationTypePlan
Who: ncalero@tst-rds01
Version: 0.12.31
Created: 2021-12-28 13:45:41.113713713 +0000 UTC
Info:
$
Terraform idiosyncrasy - concurrent work
© Pythian Services Inc 2021 | 22
$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
data.aws_security_group.sgpr: Refreshing state...
aws_rds_cluster.master: Refreshing state... [id=nelson-auroramysql56]
aws_rds_cluster_instance.cluster_instances[0]: Refreshing state... [id=nelson-auroramysql56-0]
aws_rds_cluster_instance.cluster_instances[1]: Refreshing state... [id=nelson-auroramysql56-1]
mysql_user.checkmk_mon: Refreshing state... [id=checkmk_mon@%]
Error: Could not connect to server: dial tcp 127.0.0.1:3306: connect: connection refused
$ terraform plan
...
Error locking state: Error acquiring the state lock: ConditionalCheckFailedException: The conditional
request failed
Lock Info:
ID: f6f04b96-de6d-754a-7bbe-8d0504f75e79
Path: my-tf-state-286052569778/stable/us-west-1/services/my-db/aurora-mysql/terraform.tfstate
Operation: OperationTypePlan
Who: c_ncalero@gds-stg-rds01
Version: 0.12.31
Created: 2021-12-28 13:45:41.113713713 +0000 UTC
Info:
$
$ terraform force-unlock f6f04b96-de6d-754a-7bbe-8d0504f75e79
Do you really want to force-unlock?
Terraform will remove the lock on the remote state.
This will allow local Terraform commands to modify this state, even though it
may be still be in use. Only 'yes' will be accepted to confirm.
Enter a value: yes
Terraform state has been successfully unlocked!
The state has been unlocked, and Terraform commands should now be able to
obtain a new lock on the remote state.
$
Terraform idiosyncrasy - 3) reduce scope
© Pythian Services Inc 2021 | 23
$ terraform plan
data.oci_database_autonomous_db_versions.test_adb_versions: Reading...
oci_database_autonomous_database.autonomous_data_warehouse: Refreshing state...
[id=ocid1.autonomousdatabase.oc1.iad.anuwcljst77gzoiaaewxote6h4i73oamzoloxdueabnpwprspzngjoce52xq]
oci_database_autonomous_database.autonomous_database: Refreshing state...
[id=ocid1.autonomousdatabase.oc1.iad.anuwcljtt77gzoiamz3gg5cfsbdqgcoj7lrobis75qp7lwubm7d77bpblqgq]
data.oci_database_autonomous_databases.autonomous_databases: Reading...
...
Plan: 0 to add, 1 to change, 1 to destroy.
$ terraform state list
data.http.ip
data.oci_database_autonomous_db_versions.test_adb_versions
oci_database_autonomous_database.autonomous_data_warehouse
oci_database_autonomous_database.autonomous_database
random_string.adb_admin_password
$ terraform plan -target oci_database_autonomous_database.autonomous_database
data.oci_database_autonomous_db_versions.test_adb_versions: Reading...
oci_database_autonomous_database.autonomous_database: Refreshing state...
[id=ocid1.autonomousdatabase.oc1.iad.anuwcljtt77gzoiamz3gg5cfsbdqgcoj7lrobis75qp7lwubm7d77bpblqgq]
No changes. Your infrastructure matches the configuration.
Terraform idiosyncrasy - 4) parameter validations
© Pythian Services Inc 2021 | 24
$ terraform apply
…
aws_rds_cluster.master[0]: Creating...
aws_rds_cluster.master[0]: Still creating... [10s elapsed]
aws_rds_cluster.master[0]: Still creating... [20s elapsed]
aws_rds_cluster.master[0]: Still creating... [30s elapsed]
aws_rds_cluster.master[0]: Still creating... [40s elapsed]
aws_rds_cluster.master[0]: Creation complete after 42s [id=my-aurora]
aws_rds_cluster_instance.cluster_instances[1]: Creating...
aws_rds_cluster_instance.cluster_instances[0]: Creating...
Error: error creating RDS Cluster (my-aurora) Instance: InvalidParameterCombination: RDS does not support creating a DB instance with the
following combination: DBInstanceClass=db.t3.small, Engine=aurora-mysql, EngineVersion=5.7. For supported combinations of instance class
and database engine version, see the documentation.
status code: 400, request id: 7c15bd63-b7cf-47a3-8a10-faee35c3e9c4
on main.tf line 7, in resource "aws_rds_cluster_instance" "cluster_instances":
7: resource "aws_rds_cluster_instance" "cluster_instances" {
Error: error creating RDS Cluster (my-aurora) Instance: InvalidParameterCombination: RDS does not support creating a DB instance with the
following combination: DBInstanceClass=db.t3.small, Engine=aurora-mysql, EngineVersion=5.7. For supported combinations of instance class
and database engine version, see the documentation.
status code: 400, request id: a8cb9644-fbe2-40ee-929b-18094709cb33
on main.tf line 7, in resource "aws_rds_cluster_instance" "cluster_instances":
7: resource "aws_rds_cluster_instance" "cluster_instances" {
Terraform idiosyncrasy - 4) parameter validations
● Terraform pass parameters to the provider - it runs the operation on the cloud service and it fails
● Some providers include data sources with attributes having a list of valid values for that (only if we use
those when manipulating the resource)
○ Example: OCI Autonomous database
https://registry.terraform.io/providers/oracle/oci/latest/docs/data-sources/database_autonomous_db_versions
● Additionally, simple validations can be incorporated in the variable definition:
© Pythian Services Inc 2021 | 25
$ grep -A13 "autonomous_database_db_workload" variables.tf
variable "autonomous_database_db_workload" {
default = "OLTP"
validation {
condition = anytrue([
var.autonomous_database_db_workload == "OLTP",
var.autonomous_database_db_workload == "DW",
var.autonomous_database_db_workload == "APEX",
var.autonomous_database_db_workload == "AJD"
])
error_message = "Invalid autonomous_database_db_workload value - allowed OLTP, DW, APEX and AJD."
}
}
Terraform idiosyncrasy - parameter validations
© Pythian Services Inc 2021 | 26
Demo 3
Terraform idiosyncrasy - 5) reusing code
● Terraform by default read the current directory only
● Can include other directories using the “module” directive: https://learn.hashicorp.com/collections/terraform/modules
● Third party tools to re-using code (among environments and/or deployments) - terragrunt example:
○ Standardize resource attributes (tags, HA, networking, security, parameters) between deployments - in same or
different environments and regions
○ Support different database engines and versions
○ Validations (possible but limited to same attribute expression)
○ Modules, variables and environments - directory structure
○ Extra features: hooks, functions, etc. - check https://terragrunt.gruntwork.io/
© Pythian Services Inc 2021 | 27
module "ec2_instances" {
source = "terraform-aws-modules/ec2-instance/aws"
version = "3.5.0"
...
Terraform idiosyncrasy - 6) “features”
● Terraform binary
○ Aurora MySQL upgrade didn’t work on TF versions lower than 0.12.31 (current version: 1.1.7)
● Terraform provider
○ MySQL upgrade didn’t work with AWS provider lower than 3.63.0 (several bugs)
○ Resource discovery implemented in OCI (not in AWS as of today)
● Cloud provider limits/restrictions
○ Resource attributes are not validated (service restrictions) - cause runtime errors
■ allowed instance type for a RDS version (both ways)
■ IOPS parameter when using gp2 storage
■ CLI commands to the rescue (to confirm allowed values)
© Pythian Services Inc 2021 | 28
Cloud provider idiosyncrasy
● You can get errors today that does not repeat tomorrow, as services improves
● Don’t assume all services work the same for similar operations - RTFM!
© Pythian Services Inc 2021 | 29
● Examples on AWS:
○ MySQL minor upgrade fails without any error - not reproducible days later when repeating the test
○ RDS “create Aurora replica” is not available in some versions after upgrades
○ Aurora space used should be checked using metrics
○ RDS parameters have apply_mode immediate or pending-reboot, but tricky to update running instances
automatically (immediate mode)
○ Operations on RDS storage does not allow other operations until they finish - and those can take long
■ Automatic space increment - minutes
■ Change type between gp2 and IOPS - hours
Terraform - parameters cheat sheet
© Pythian Services Inc 2021 | 30
Format code in .tf files terraform fmt
Validate configuration terraform validate
List state file resources terraform state list
Inspect current state file content terraform show
Remove a resource from state file terraform state rm <name_in_state>
Import resource into state file terraform import <name_in_state> <name_aws>
Remove lock from state file terraform force-unlock <lock_id>
Operate on a single resource terraform <op> -target <name_in_state>
Console (to test formulas/expressions) terraform console
Debug execution TF_LOG="TRACE" TF_LOG_PATH=file.log ; terraform <op>
Terraform console
© Pythian Services Inc 2021 | 31
[c_ncalero@myhost ~]$ terraform console
> substr("19.2",0,2)
19
> split(".","19.2")
[
"19",
"2",
]
> split(".","19.2")[0]
19
> element(split(".","19.2"),0)
19
> join("",["oracle",substr("19.2",0,2), "small"])
oracle19small
> join("",["oracle",split(".","19.2")[0]])
oracle19
> exit
OCI CLI - useful validations
© Pythian Services Inc 2021 | 32
$ export COMPID=$(oci iam compartment list --query "data [?name=='mycmp'].{id:id}" | jq -r '.[].id')
$ oci db version list -c $COMPID --db-system-shape "VM.Standard1.1" --output table
+-----------------------------+--------------+-----------------+
| is-latest-for-major-version | supports-pdb | version |
+-----------------------------+--------------+-----------------+
| True | False | 11.2.0.4 |
| False | False | 11.2.0.4.210119 |
| False | False | 11.2.0.4.210420 |
| True | True | 12.1.0.2 |
| False | True | 12.1.0.2.211019 |
| False | True | 12.1.0.2.220118 |
| False | True | 12.1.0.2.220419 |
| True | True | 12.2.0.1 |
| False | True | 12.2.0.1.211019 |
| False | True | 12.2.0.1.220118 |
| False | True | 12.2.0.1.220419 |
| True | True | 18.0.0.0 |
| False | True | 18.16.0.0 |
| True | True | 19.0.0.0 |
| False | True | 19.13.0.0 |
| False | True | 19.14.0.0 |
● Available database versions for DB Systems using shape VM.Standard1.1
| False | True | 19.15.0.0 |
| True | True | 21.0.0.0 |
| False | True | 21.4.0.0 |
| False | True | 21.5.0.0 |
| False | True | 21.6.0.0 |
+-------------+--------------+-----------+
OCI CLI - useful validations
© Pythian Services Inc 2021 | 33
$ export COMPID=$(oci iam compartment list --query "data [?name=='mycmp'].{id:id}" | jq -r '.[].id' )
$ oci db autonomous-database list -c $COMPID --query 'data[].{db_name:"db-name", db_version:"db-version",
db_workload:"db-workload", display_name:"display-name", state:"lifecycle-state"}' --output table
+--------------------+------------+-------------+-----------------------------------+-----------+
| db_name | db_version | db_workload | display_name | state |
+--------------------+------------+-------------+-----------------------------------+-----------+
| adbdw | 19c | DW | example_autonomous_data_warehouse | AVAILABLE |
| adboltp | 19c | OLTP | example_autonomous_database | AVAILABLE |
+--------------------+------------+-------------+-----------------------------------+-----------+
● ADBs created in compartment ’mycmp’
AWS CLI - useful validations
© Pythian Services Inc 2021 | 34
$ aws rds describe-orderable-db-instance-options --engine oracle-ee --db-instance-class db.t3.small --query
"OrderableDBInstanceOptions[].{EngineVersion:EngineVersion}" --output text --region eu-west-1
12.1.0.2.v1
12.1.0.2.v10
12.1.0.2.v11
12.1.0.2.v12
12.1.0.2.v13
12.1.0.2.v14
12.1.0.2.v15
12.1.0.2.v16
12.1.0.2.v17
12.1.0.2.v18
12.1.0.2.v19
12.1.0.2.v2
12.1.0.2.v20
12.1.0.2.v21
12.1.0.2.v22
12.1.0.2.v23
12.1.0.2.v24
12.1.0.2.v25
12.1.0.2.v26
12.1.0.2.v27
● Available Oracle EE versions for shape db.t3.small
12.1.0.2.v3
12.1.0.2.v4
12.1.0.2.v5
12.1.0.2.v6
12.1.0.2.v7
12.1.0.2.v8
12.1.0.2.v9
19.0.0.0.ru-2019-07.rur-2019-07.r1
19.0.0.0.ru-2019-10.rur-2019-10.r1
19.0.0.0.ru-2020-01.rur-2020-01.r1
19.0.0.0.ru-2020-04.rur-2020-04.r1
19.0.0.0.ru-2020-07.rur-2020-07.r1
19.0.0.0.ru-2020-10.rur-2020-10.r1
19.0.0.0.ru-2021-01.rur-2021-01.r1
19.0.0.0.ru-2021-01.rur-2021-01.r2
19.0.0.0.ru-2021-04.rur-2021-04.r1
19.0.0.0.ru-2021-07.rur-2021-07.r1
19.0.0.0.ru-2021-10.rur-2021-10.r1
19.0.0.0.ru-2022-01.rur-2022-01.r1
AWS CLI - useful validations
© Pythian Services Inc 2021 | 35
$ aws rds describe-db-engine-versions --engine oracle-ee --engine-version 12.1.0.2.v20 --query
"DBEngineVersions[*].ValidUpgradeTarget[*].{EngineVersion:EngineVersion}" --output text
12.1.0.2.v21
12.1.0.2.v22
12.1.0.2.v23
12.1.0.2.v24
12.1.0.2.v25
12.1.0.2.v26
12.1.0.2.v27
19.0.0.0.ru-2020-04.rur-2020-04.r1
19.0.0.0.ru-2020-07.rur-2020-07.r1
19.0.0.0.ru-2020-10.rur-2020-10.r1
19.0.0.0.ru-2021-01.rur-2021-01.r1
19.0.0.0.ru-2021-01.rur-2021-01.r2
19.0.0.0.ru-2021-04.rur-2021-04.r1
19.0.0.0.ru-2021-07.rur-2021-07.r1
19.0.0.0.ru-2021-10.rur-2021-10.r1
19.0.0.0.ru-2022-01.rur-2022-01.r1
● Supported upgrade paths for Oracle version 12.1.0.2
AWS CLI - useful validations
© Pythian Services Inc 2021 | 36
$ REGION=us-west-1; for i in $(awscloudwatch list-metrics --region=$REGION --namespace AWS/RDS --metric-name
VolumeBytesUsed --query "Metrics[].Dimensions[].{Name:Name, Value:Value}" --output table | grep
DBClusterIdentifier | cut -d"|" -f3) ; do echo -n $i " " ; aws
cloudwatch get-metric-statistics --region
$REGION --namespace AWS/RDS --metric-name VolumeBytesUsed --dimensions Name=EngineName,Value=aurora-mysql
--dimensions Name=EngineName,Value=aurora Name=DbClusterIdentifier,Value=$i --start-time $(date -u -d "1 days
ago" '+%F') --end-time $(date -u '+%F') --statistics Maximum --period 3600 --query
"Datapoints[].{Timestamp:Timestamp, Maximum:Maximum, Unit:Unit}" | jq '.[] | .Maximum' | sort -u | tail -1 ;
done | sort
bookshop-db 7381361362844
sales-db 179468653458
track-db 20681679872
programs-db 35563371113
analytics-db 8650433415
master-db 17432850551
resourcing-db 6418328690
…
● Space in use by aurora Mysql instances running in region us-west-1
AWS CLI - useful validations
© Pythian Services Inc 2021 | 37
$ aws rds describe-db-parameters --db-parameter-group-name mypg --region us-west-1 --query
"sort_by(Parameters,&ParameterName)[?Source=='user'].{ParameterName:ParameterName, ApplyMethod:ApplyMethod,
IsModifiable:IsModifiable, ParameterValue:ParameterValue}" --output table
+----------------+---------------+-----------------------------------+------------------+
| ApplyMethod | IsModifiable | ParameterName | ParameterValue |
+----------------+---------------+-----------------------------------+------------------+
| immediate | True | event_scheduler | ON |
| immediate | True | group_concat_max_len | 16384 |
| pending-reboot| True | innodb_sort_buffer_size | 8388608 |
| immediate | True | log_bin_trust_function_creators | 1 |
| immediate | True | long_query_time | 0.1 |
| pending-reboot| True | max_allowed_packet | 1073741824 |
| immediate | True | max_connections | 8000 |
| immediate | True | slow_query_log | 1 |
+----------------+---------------+-----------------------------------+------------------+
● Parameters set by user (we) in a RDS parameter group
Questions?
© Pythian Services Inc 2021 | 38
calero@pythian.com
@ncalerouy
http://www.linkedin.com/in/ncalero

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Introduction to openshift
Introduction to openshiftIntroduction to openshift
Introduction to openshift
 
Kubernetes Secrets Management on Production with Demo
Kubernetes Secrets Management on Production with DemoKubernetes Secrets Management on Production with Demo
Kubernetes Secrets Management on Production with Demo
 
Monitoring kubernetes with prometheus
Monitoring kubernetes with prometheusMonitoring kubernetes with prometheus
Monitoring kubernetes with prometheus
 
Kubernetes Networking | Kubernetes Services, Pods & Ingress Networks | Kubern...
Kubernetes Networking | Kubernetes Services, Pods & Ingress Networks | Kubern...Kubernetes Networking | Kubernetes Services, Pods & Ingress Networks | Kubern...
Kubernetes Networking | Kubernetes Services, Pods & Ingress Networks | Kubern...
 
Openshift Container Platform
Openshift Container PlatformOpenshift Container Platform
Openshift Container Platform
 
OpenShift 4, the smarter Kubernetes platform
OpenShift 4, the smarter Kubernetes platformOpenShift 4, the smarter Kubernetes platform
OpenShift 4, the smarter Kubernetes platform
 
Terraform: An Overview & Introduction
Terraform: An Overview & IntroductionTerraform: An Overview & Introduction
Terraform: An Overview & Introduction
 
Containers Anywhere with OpenShift by Red Hat
Containers Anywhere with OpenShift by Red HatContainers Anywhere with OpenShift by Red Hat
Containers Anywhere with OpenShift by Red Hat
 
Terraform Basics
Terraform BasicsTerraform Basics
Terraform Basics
 
Amazon EKS - Elastic Container Service for Kubernetes
Amazon EKS - Elastic Container Service for KubernetesAmazon EKS - Elastic Container Service for Kubernetes
Amazon EKS - Elastic Container Service for Kubernetes
 
Kubernetes PPT.pptx
Kubernetes PPT.pptxKubernetes PPT.pptx
Kubernetes PPT.pptx
 
Compliance in the Cloud Using “Security by Design” Principles
Compliance in the Cloud Using “Security by Design” PrinciplesCompliance in the Cloud Using “Security by Design” Principles
Compliance in the Cloud Using “Security by Design” Principles
 
What Is Kubernetes | Kubernetes Introduction | Kubernetes Tutorial For Beginn...
What Is Kubernetes | Kubernetes Introduction | Kubernetes Tutorial For Beginn...What Is Kubernetes | Kubernetes Introduction | Kubernetes Tutorial For Beginn...
What Is Kubernetes | Kubernetes Introduction | Kubernetes Tutorial For Beginn...
 
Introduction to Kubernetes Workshop
Introduction to Kubernetes WorkshopIntroduction to Kubernetes Workshop
Introduction to Kubernetes Workshop
 
Multi Cloud Architecture Approach
Multi Cloud Architecture ApproachMulti Cloud Architecture Approach
Multi Cloud Architecture Approach
 
Kubernates vs Openshift: What is the difference and comparison between Opensh...
Kubernates vs Openshift: What is the difference and comparison between Opensh...Kubernates vs Openshift: What is the difference and comparison between Opensh...
Kubernates vs Openshift: What is the difference and comparison between Opensh...
 
Terraform
TerraformTerraform
Terraform
 
Open shift 4 infra deep dive
Open shift 4    infra deep diveOpen shift 4    infra deep dive
Open shift 4 infra deep dive
 
(DVO401) Deep Dive into Blue/Green Deployments on AWS
(DVO401) Deep Dive into Blue/Green Deployments on AWS(DVO401) Deep Dive into Blue/Green Deployments on AWS
(DVO401) Deep Dive into Blue/Green Deployments on AWS
 
Ansible
AnsibleAnsible
Ansible
 

Ähnlich wie Terraform Tips and Tricks - LAOUC 2022

glance replicator
glance replicatorglance replicator
glance replicator
irix_jp
 
Common schema my sql uc 2012
Common schema   my sql uc 2012Common schema   my sql uc 2012
Common schema my sql uc 2012
Roland Bouman
 
Common schema my sql uc 2012
Common schema   my sql uc 2012Common schema   my sql uc 2012
Common schema my sql uc 2012
Roland Bouman
 
Understanding Query Optimization with ‘regular’ and ‘Exadata’ Oracle
Understanding Query Optimization with ‘regular’ and ‘Exadata’ OracleUnderstanding Query Optimization with ‘regular’ and ‘Exadata’ Oracle
Understanding Query Optimization with ‘regular’ and ‘Exadata’ Oracle
Guatemala User Group
 
Oracle Diagnostics : Explain Plans (Simple)
Oracle Diagnostics : Explain Plans (Simple)Oracle Diagnostics : Explain Plans (Simple)
Oracle Diagnostics : Explain Plans (Simple)
Hemant K Chitale
 

Ähnlich wie Terraform Tips and Tricks - LAOUC 2022 (20)

Terraform Cosmos DB
Terraform Cosmos DBTerraform Cosmos DB
Terraform Cosmos DB
 
TerraformとAzureを組み合わせて使うときの勘所
TerraformとAzureを組み合わせて使うときの勘所TerraformとAzureを組み合わせて使うときの勘所
TerraformとAzureを組み合わせて使うときの勘所
 
glance replicator
glance replicatorglance replicator
glance replicator
 
Getting data into Rudder
Getting data into RudderGetting data into Rudder
Getting data into Rudder
 
leboncoin DataEngineering / Terraform - beginner to advanced
leboncoin DataEngineering / Terraform - beginner to advancedleboncoin DataEngineering / Terraform - beginner to advanced
leboncoin DataEngineering / Terraform - beginner to advanced
 
Debasihish da final.ppt
Debasihish da final.pptDebasihish da final.ppt
Debasihish da final.ppt
 
Terraform 0.9 + good practices
Terraform 0.9 + good practicesTerraform 0.9 + good practices
Terraform 0.9 + good practices
 
Adding Complex Data to Spark Stack by Tug Grall
Adding Complex Data to Spark Stack by Tug GrallAdding Complex Data to Spark Stack by Tug Grall
Adding Complex Data to Spark Stack by Tug Grall
 
Common schema my sql uc 2012
Common schema   my sql uc 2012Common schema   my sql uc 2012
Common schema my sql uc 2012
 
Common schema my sql uc 2012
Common schema   my sql uc 2012Common schema   my sql uc 2012
Common schema my sql uc 2012
 
Final terraform
Final terraformFinal terraform
Final terraform
 
Understanding Query Optimization with ‘regular’ and ‘Exadata’ Oracle
Understanding Query Optimization with ‘regular’ and ‘Exadata’ OracleUnderstanding Query Optimization with ‘regular’ and ‘Exadata’ Oracle
Understanding Query Optimization with ‘regular’ and ‘Exadata’ Oracle
 
Evolution of Performance Management: Oracle 12c adaptive optimizations - ukou...
Evolution of Performance Management: Oracle 12c adaptive optimizations - ukou...Evolution of Performance Management: Oracle 12c adaptive optimizations - ukou...
Evolution of Performance Management: Oracle 12c adaptive optimizations - ukou...
 
Oracle Diagnostics : Explain Plans (Simple)
Oracle Diagnostics : Explain Plans (Simple)Oracle Diagnostics : Explain Plans (Simple)
Oracle Diagnostics : Explain Plans (Simple)
 
Best Practices of Infrastructure as Code with Terraform
Best Practices of Infrastructure as Code with TerraformBest Practices of Infrastructure as Code with Terraform
Best Practices of Infrastructure as Code with Terraform
 
Distributed Queries in IDS: New features.
Distributed Queries in IDS: New features.Distributed Queries in IDS: New features.
Distributed Queries in IDS: New features.
 
Terraform at Scale - All Day DevOps 2017
Terraform at Scale - All Day DevOps 2017Terraform at Scale - All Day DevOps 2017
Terraform at Scale - All Day DevOps 2017
 
Oracle dbms_xplan.display_cursor format
Oracle dbms_xplan.display_cursor formatOracle dbms_xplan.display_cursor format
Oracle dbms_xplan.display_cursor format
 
Oracle Exadata Cloud Services guide from practical experience - OOW19
Oracle Exadata Cloud Services guide from practical experience - OOW19Oracle Exadata Cloud Services guide from practical experience - OOW19
Oracle Exadata Cloud Services guide from practical experience - OOW19
 
[Pgday.Seoul 2019] Citus를 이용한 분산 데이터베이스
[Pgday.Seoul 2019] Citus를 이용한 분산 데이터베이스[Pgday.Seoul 2019] Citus를 이용한 분산 데이터베이스
[Pgday.Seoul 2019] Citus를 이용한 분산 데이터베이스
 

Mehr von Nelson Calero

Mehr von Nelson Calero (20)

Database automation guide - Oracle Community Tour LATAM 2023
Database automation guide - Oracle Community Tour LATAM 2023Database automation guide - Oracle Community Tour LATAM 2023
Database automation guide - Oracle Community Tour LATAM 2023
 
Oracle on kubernetes 101 - Dec/2021
Oracle on kubernetes 101 - Dec/2021Oracle on kubernetes 101 - Dec/2021
Oracle on kubernetes 101 - Dec/2021
 
Automate Oracle database patches and upgrades using Fleet Provisioning and Pa...
Automate Oracle database patches and upgrades using Fleet Provisioning and Pa...Automate Oracle database patches and upgrades using Fleet Provisioning and Pa...
Automate Oracle database patches and upgrades using Fleet Provisioning and Pa...
 
Automate your oracle cloud infrastructure operations v2.0 - OOW19
Automate your oracle cloud infrastructure operations v2.0 - OOW19Automate your oracle cloud infrastructure operations v2.0 - OOW19
Automate your oracle cloud infrastructure operations v2.0 - OOW19
 
Automate the operation of your Oracle Cloud infrastructure v2.0
Automate the operation of your Oracle Cloud infrastructure v2.0Automate the operation of your Oracle Cloud infrastructure v2.0
Automate the operation of your Oracle Cloud infrastructure v2.0
 
SSL certificates in the Oracle Database without surprises
SSL certificates in the Oracle Database without surprisesSSL certificates in the Oracle Database without surprises
SSL certificates in the Oracle Database without surprises
 
Practical guide to Oracle Virtual environments
Practical guide to Oracle Virtual environmentsPractical guide to Oracle Virtual environments
Practical guide to Oracle Virtual environments
 
Automate your Oracle Cloud Infrastructure operation
Automate your Oracle Cloud Infrastructure operationAutomate your Oracle Cloud Infrastructure operation
Automate your Oracle Cloud Infrastructure operation
 
Welcome to databases in the Cloud
Welcome to databases in the CloudWelcome to databases in the Cloud
Welcome to databases in the Cloud
 
Redefining tables online without surprises
Redefining tables online without surprisesRedefining tables online without surprises
Redefining tables online without surprises
 
Protect Sensitive Data: Implementing Fine-Grained Access Control in Oracle
Protect Sensitive Data: Implementing Fine-Grained Access Control in OracleProtect Sensitive Data: Implementing Fine-Grained Access Control in Oracle
Protect Sensitive Data: Implementing Fine-Grained Access Control in Oracle
 
Oracle Exadata Maintenance tasks 101 - OTN Tour 2015
Oracle Exadata Maintenance tasks 101 - OTN Tour 2015Oracle Exadata Maintenance tasks 101 - OTN Tour 2015
Oracle Exadata Maintenance tasks 101 - OTN Tour 2015
 
My Experience Using Oracle SQL Plan Baselines 11g/12c
My Experience Using Oracle SQL Plan Baselines 11g/12cMy Experience Using Oracle SQL Plan Baselines 11g/12c
My Experience Using Oracle SQL Plan Baselines 11g/12c
 
Oracle RAC sin sorpresas - v2014
Oracle RAC sin sorpresas - v2014Oracle RAC sin sorpresas - v2014
Oracle RAC sin sorpresas - v2014
 
Alta disponibilidad con Pacemaker
Alta disponibilidad con PacemakerAlta disponibilidad con Pacemaker
Alta disponibilidad con Pacemaker
 
AROUG BIDAY 2013 - Automatizar procesos de ETL con PL/SQL
AROUG BIDAY 2013 - Automatizar procesos de ETL con PL/SQLAROUG BIDAY 2013 - Automatizar procesos de ETL con PL/SQL
AROUG BIDAY 2013 - Automatizar procesos de ETL con PL/SQL
 
MariaDB y FOSS en infraestructura de salud y estándares
MariaDB y FOSS en infraestructura de salud y estándaresMariaDB y FOSS en infraestructura de salud y estándares
MariaDB y FOSS en infraestructura de salud y estándares
 
UYOUG 2012 - Oracle RAC 11gR2 - New features
UYOUG 2012 - Oracle RAC 11gR2 - New featuresUYOUG 2012 - Oracle RAC 11gR2 - New features
UYOUG 2012 - Oracle RAC 11gR2 - New features
 
Collaborate 2012 - Administering MySQL for Oracle DBAs
Collaborate 2012 - Administering MySQL for Oracle DBAsCollaborate 2012 - Administering MySQL for Oracle DBAs
Collaborate 2012 - Administering MySQL for Oracle DBAs
 
Collaborate 2012 - RMAN Eliminate the mystery
Collaborate 2012 - RMAN Eliminate the mysteryCollaborate 2012 - RMAN Eliminate the mystery
Collaborate 2012 - RMAN Eliminate the mystery
 

Kürzlich hochgeladen

IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
Enterprise Knowledge
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
Joaquim Jorge
 

Kürzlich hochgeladen (20)

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
🐬 The future of MySQL is Postgres 🐘
🐬  The future of MySQL is Postgres   🐘🐬  The future of MySQL is Postgres   🐘
🐬 The future of MySQL is Postgres 🐘
 
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
 
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men
 
Tech Trends Report 2024 Future Today Institute.pdf
Tech Trends Report 2024 Future Today Institute.pdfTech Trends Report 2024 Future Today Institute.pdf
Tech Trends Report 2024 Future Today Institute.pdf
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024
 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreter
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 

Terraform Tips and Tricks - LAOUC 2022

  • 1. Terraform Tips and Tricks LAOUC Community Tour 2022 August 17th Nelson Calero © Pythian Services Inc 2022 | 1
  • 2. Our Speaker Nelson Calero Internal Principal Consultant Pythian http://www.linkedin.com/in/ncalero @ncalerouy © Pythian Services Inc 2022 | 2
  • 3. 24 Years in Business _____________________ Premier Partner 140+ Certifications 8 Specializations 360+ Experts Across 5 Continents 500+ Customers Globally _____________________ Advanced Partner 175+ Certifications _____________________ Gold Partner 15+ Certifications _____________________ Platinum Partner 150+ Certifications _____________________ SAP Certified Partner 40+ Certifications © Pythian Services Inc 2022 | 3
  • 4. A G E N D A ● Terraform basics (1 slide) ● Terraform Idiosyncrasy ● Cloud provider Idiosyncrasy © Pythian Services Inc 2022 | 4
  • 5. Terraform basics ● Infrastructure as Code, declarative config files (HCL) ● Simple install - standalone binary file ● Few operations - plan/apply/destroy ● Providers - implement how to deal with your resources. 1000+ ● State file - mapping of existing resources to TF configuration ● Simple structure ○ Reads code from current directory ○ Split code in files for readability: main.tf, variables.tf, outputs.tf ● Advanced configurations - workspaces, modules, remote state files ● Changes in versions - Features, state file format © Pythian Services Inc 2021 | 5
  • 6. Terraform basics ● Infrastructure as Code, declarative config files (HCL) ● Simple install - standalone binary file ● Few operations - plan/apply/destroy ● Providers - implement how to deal with your resources. 1000+ ● State file - mapping of existing resources to TF configuration ● Simple structure ○ Reads code from current directory ○ Split code in files for readability: main.tf, variables.tf, outputs.tf ● Advanced configurations - workspaces, modules, remote state files ● Changes in versions - Features, state file format © Pythian Services Inc 2021 | 6 terraform { required_providers { aws = { source = "hashicorp/aws" version = "~> 3.27" } } required_version = ">= 0.14.9" } provider "aws" { profile = "default" region = "us-west-1" } resource "aws_instance" "app_server" { ami = "ami-830c94e3" instance_type = "t2.micro" tags = { Name = "AppServerInstance" } } ● https://www.terraform.io/ ● Lucas’ blog: https://technology.amis.nl/continuous-delivery/provisioning/terraform/
  • 7. Terraform idiosyncrasy 1) State file relevance 2) Concurrent work on same resources 3) Reducing scope of operations 4) Resource parameters validation 5) Re-using code 6) Available features = TF + provider © Pythian Services Inc 2021 | 7
  • 8. Terraform idiosyncrasy ● state file drives the changes - make sure is in sync with your resources ● state file - removing resources (state rm), importing (import name_in_state name_aws) ● State file lock from previous canceled/timed out execution: force-unlock to remove it ● Plan and apply only one resource from many: -target name_in_state ● Conditional resources in modules: count ● Re-using code among environments and/or deployments? Welcome terragrunt. Examples: ○ Standardize resource attributes (tags, HA, networking, security, parameters) between deployments - in same or different environments and regions ○ Support different database engines and versions ○ Validations (possible but limited to same attribute expression) ○ Modules, variables and environments - directory structure ○ Several other features - check https://terragrunt.gruntwork.io/ © Pythian Services Inc 2021 | 8
  • 9. Terraform idiosyncrasy - 1) state file Demo 1 Code taken from public Oracle repo (few changes for simplicity): https://github.com/oracle/terraform-provider-oci/tree/master/examples/database/adb © Pythian Services Inc 2021 | 9
  • 10. Terraform idiosyncrasy - state file © Pythian Services Inc 2021 | 10 $ terraform plan Refreshing Terraform state in-memory prior to plan... The refreshed state will be used to calculate this plan, but will not be persisted to local or remote state storage. data.aws_security_group.sgpr: Refreshing state... ------------------------------------------------------------------------ An execution plan has been generated and is shown below. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # aws_db_instance.master will be created + resource "aws_db_instance" "master" { + address = (known after apply) + allocated_storage = 500 + apply_immediately = (known after apply) ... + identifier ="nelson-ora19" ... Plan: 1 to add, 0 to change, 0 to destroy.
  • 11. Terraform idiosyncrasy - state file © Pythian Services Inc 2021 | 11 $ terraform plan Refreshing Terraform state in-memory prior to plan... The refreshed state will be used to calculate this plan, but will not be persisted to local or remote state storage. data.aws_security_group.sgpr: Refreshing state... ------------------------------------------------------------------------ An execution plan has been generated and is shown below. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # aws_db_instance.master will be created + resource "aws_db_instance" "master" { + address = (known after apply) + allocated_storage = 500 + apply_immediately = (known after apply) ... + identifier ="nelson-pg14" ... Plan: 1 to add, 0 to change, 0 to destroy. $ aws rds describe-db-instances --region=us-west-1 --db-instance-identifier nelson-ora19 --query "*[].{DBClusterIdentifier:DBClusterIdentifier,DBInstanceIdentifier:DBInstanceIdentifier,Engin eVersion:EngineVersion}" --output table -------------------------------------------------------------------------------------- | DescribeDBInstances | +---------------------+------------------------+-------------------------------------+ | DBClusterIdentifier | DBInstanceIdentifier | EngineVersion | +---------------------+------------------------+-------------------------------------+ | None | nelson-ora19 | 19.0.0.0.ru-2021-07.rur-2021-07.r1 | +---------------------+------------------------+-------------------------------------+
  • 12. Terraform idiosyncrasy - state file © Pythian Services Inc 2021 | 12 $ terraform plan Refreshing Terraform state in-memory prior to plan... The refreshed state will be used to calculate this plan, but will not be persisted to local or remote state storage. data.aws_security_group.sgpr: Refreshing state... ------------------------------------------------------------------------ An execution plan has been generated and is shown below. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # aws_db_instance.master will be created + resource "aws_db_instance" "master" { + address = (known after apply) + allocated_storage = 500 + apply_immediately = (known after apply) ... + identifier ="nelson-pg14" ... Plan: 1 to add, 0 to change, 0 to destroy. $ aws rds describe-db-instances --region=us-west-1 --db-instance-identifier nelson-ora19 --query "*[].{DBClusterIdentifier:DBClusterIdentifier,DBInstanceIdentifier:DBInstanceIdentifier,Engin eVersion:EngineVersion}" --output table -------------------------------------------------------------------------------------- | DescribeDBInstances | +---------------------+------------------------+-------------------------------------+ | DBClusterIdentifier | DBInstanceIdentifier | EngineVersion | +---------------------+------------------------+-------------------------------------+ | None | nelson-ora19 | 19.0.0.0.ru-2021-07.rur-2021-07.r1 | +---------------------+------------------------+-------------------------------------+ $ terraform import aws_db_instance.master nelson-ora19 aws_db_instance.master: Importing from ID "nelson-ora19"... aws_db_instance.master: Import prepared! Prepared aws_db_instance for import aws_db_instance.master: Refreshing state... [id=nelson-ora19] Import successful! The resources that were imported are shown above. These resources are now in your Terraform state and will henceforth be managed by Terraform.
  • 13. Terraform idiosyncrasy - state file © Pythian Services Inc 2021 | 13 $ terraform plan Refreshing Terraform state in-memory prior to plan... The refreshed state will be used to calculate this plan, but will not be persisted to local or remote state storage. data.aws_security_group.sgpr: Refreshing state... ------------------------------------------------------------------------ An execution plan has been generated and is shown below. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # aws_db_instance.master will be created + resource "aws_db_instance" "master" { + address = (known after apply) + allocated_storage = 500 + apply_immediately = (known after apply) ... + identifier ="nelson-pg14" ... Plan: 1 to add, 0 to change, 0 to destroy. $ aws rds describe-db-instances --region=us-west-1 --db-instance-identifier nelson-pg14 --query "*[].{DBClusterIdentifier:DBClusterIdentifier,DBInstanceIdentifier:DBInstanceIdentifier,Engin eVersion:EngineVersion}" --output table -------------------------------------------------------------------------------------- | DescribeDBInstances | +---------------------+------------------------+-------------------------------------+ | DBClusterIdentifier | DBInstanceIdentifier | EngineVersion | +---------------------+------------------------+-------------------------------------+ | None | nelson-ora19 | 19.0.0.0.ru-2021-07.rur-2021-07.r1 | +---------------------+------------------------+-------------------------------------+ $ terraform import aws_db_instance.master nelson-pg14 aws_db_instance.master: Importing from ID "nelson-pg14"... aws_db_instance.master: Import prepared! Prepared aws_db_instance for import aws_db_instance.master: Refreshing state... [id=nelson-pg14] Import successful! The resources that were imported are shown above. These resources are now in your Terraform state and will henceforth be managed by Terraform. $ terraform plan Refreshing Terraform state in-memory prior to plan... The refreshed state will be used to calculate this plan, but will not be persisted to local or remote state storage. data.aws_security_group.sgpr: Refreshing state... aws_db_instance.master: Refreshing state... [id=nelson-ora19] ------------------------------------------------------------------------ An execution plan has been generated and is shown below. Resource actions are indicated with the following symbols: ~ update in-place Terraform will perform the following actions: # aws_db_instance.master will be updated in-place ~ resource "aws_db_instance" "master" { ... + password = (sensitive value) ... Plan: 0 to add, 1 to change, 0 to destroy.
  • 14. Terraform idiosyncrasy - state file © Pythian Services Inc 2021 | 14 $ terraform plan Refreshing Terraform state in-memory prior to plan... The refreshed state will be used to calculate this plan, but will not be persisted to local or remote state storage. data.aws_security_group.sgpr: Refreshing state... ------------------------------------------------------------------------ An execution plan has been generated and is shown below. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # aws_db_instance.master will be created + resource "aws_db_instance" "master" { + address = (known after apply) + allocated_storage = 500 + apply_immediately = (known after apply) ... + identifier ="nelson-pg14" ... Plan: 1 to add, 0 to change, 0 to destroy. $ aws rds describe-db-instances --region=us-west-1 --db-instance-identifier nelson-pg14 --query "*[].{DBClusterIdentifier:DBClusterIdentifier,DBInstanceIdentifier:DBInstanceIdentifier,Engin eVersion:EngineVersion}" --output table -------------------------------------------------------------------------------------- | DescribeDBInstances | +---------------------+------------------------+-------------------------------------+ | DBClusterIdentifier | DBInstanceIdentifier | EngineVersion | +---------------------+------------------------+-------------------------------------+ | None | nelson-ora19 | 19.0.0.0.ru-2021-07.rur-2021-07.r1 | +---------------------+------------------------+-------------------------------------+ $ terraform import aws_db_instance.master nelson-pg14 aws_db_instance.master: Importing from ID "nelson-pg14"... aws_db_instance.master: Import prepared! Prepared aws_db_instance for import aws_db_instance.master: Refreshing state... [id=nelson-pg14] Import successful! The resources that were imported are shown above. These resources are now in your Terraform state and will henceforth be managed by Terraform. $ terraform plan Refreshing Terraform state in-memory prior to plan... The refreshed state will be used to calculate this plan, but will not be persisted to local or remote state storage. data.aws_security_group.sgpr: Refreshing state... aws_db_instance.master: Refreshing state... [id=nelson-pg14] ------------------------------------------------------------------------ An execution plan has been generated and is shown below. Resource actions are indicated with the following symbols: ~ update in-place Terraform will perform the following actions: # aws_db_instance.master will be updated in-place ~ resource "aws_db_instance" "master" { ... + password = (sensitive value) ... Plan: 0 to add, 1 to change, 0 to destroy. $ terraform apply … $ terraform plan Refreshing Terraform state in-memory prior to plan... The refreshed state will be used to calculate this plan, but will not be persisted to local or remote state storage. data.aws_security_group.sgpr: Refreshing state... aws_db_instance.master: Refreshing state... [id=nelson-ora19] ------------------------------------------------------------------------ No changes. Infrastructure is up-to-date. This means that Terraform did not detect any differences between your configuration and real physical resources that exist. As a result, no actions need to be performed.
  • 15. Terraform idiosyncrasy - state file ● Default state file is terraform.tfstate in current directory ● Critical data, requires good backup policy ● terraform import is not always able to recreate all attributes (ex: passwords, MySQL grants) ● Two key complementary features: ○ remote state file https://www.terraform.io/language/state/remote ○ OCI discovery https://registry.terraform.io/providers/oracle/oci/latest/docs/guides/resource_discovery © Pythian Services Inc 2021 | 15
  • 16. Terraform idiosyncrasy - state file / remote ● Remote state file - https://www.terraform.io/language/state/remote © Pythian Services Inc 2021 | 16 remote_state { backend "s3" { bucket = "terraform-states" key = "networking/terraform.tfstate" region = "us-phoenix-1" endpoint = "https://acme.compat.objectstorage.us-phoenix-1.oraclecloud.com" shared_credentials_file = "../terraform-states_bucket_credentials" skip_region_validation = true skip_credentials_validation = true skip_metadata_api_check = true force_path_style = true } } Ref: https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/terraformUsingObjectStore.htm
  • 17. Terraform idiosyncrasy - state file / OCI discovery Demo 2 OCI discovery docs: https://registry.terraform.io/providers/oracle/oci/latest/docs/guides/resource_discovery © Pythian Services Inc 2021 | 17
  • 18. Terraform idiosyncrasy - state file / OCI discovery ● terraform-provider-oci gets installed by terraform: ● Discover databases and create the state file: © Pythian Services Inc 2021 | 18 $ ls -lrt .terraform/providers/registry.terraform.io/hashicorp/oci/4.77.0/linux_amd64/ total 129320 -rw-r--r-- 1 calero calero 96384 May 31 17:31 CHANGELOG.md -rw-r--r-- 1 calero calero 16725 May 31 17:31 LICENSE.md -rw-r--r-- 1 calero calero 3071 May 31 17:31 README.md -rwxr-xr-x 1 calero calero 132177920 May 31 17:31 terraform-provider-oci_v4.77.0 $ mkdir discover $ .terraform/providers/registry.terraform.io/hashicorp/oci/4.77.0/linux_amd64/terraform-provider-oci_v4.77.0 -command=export -compartment_id=$COMPID -services database -generate_state -output_path=discover/ ... $ ls -lrt discover total 44 -rw-r--r-- 1 calero calero 10109 May 31 18:46 terraform.tfstate.tmp.backup -rw-r--r-- 1 calero calero 22086 May 31 18:46 terraform.tfstate -rw-r--r-- 1 calero calero 6085 May 31 18:46 database.tf -rw-r--r-- 1 calero calero 38 May 31 18:46 provider.tf -rw-r--r-- 1 calero calero 171 May 31 18:46 vars.tf $
  • 19. Terraform idiosyncrasy - state file / OCI discovery ● terraform-provider-oci gets installed by terraform: ● Discover databases and create the state file: © Pythian Services Inc 2021 | 19 $ ls -lrt .terraform/providers/registry.terraform.io/hashicorp/oci/4.77.0/linux_amd64/ total 129320 -rw-r--r-- 1 calero calero 96384 May 31 17:31 CHANGELOG.md -rw-r--r-- 1 calero calero 16725 May 31 17:31 LICENSE.md -rw-r--r-- 1 calero calero 3071 May 31 17:31 README.md -rwxr-xr-x 1 calero calero 132177920 May 31 17:31 terraform-provider-oci_v4.77.0 $ mkdir discover $ .terraform/providers/registry.terraform.io/hashicorp/oci/4.77.0/linux_amd64/terraform-provider-oci_v4.77.0 -command=export -compartment_id=$COMPID -services database -generate_state -output_path=../discover/ ... $ ls -lrt ../discover2 total 44 -rw-r--r-- 1 calero calero 10109 May 31 18:46 terraform.tfstate.tmp.backup -rw-r--r-- 1 calero calero 22086 May 31 18:46 terraform.tfstate -rw-r--r-- 1 calero calero 6085 May 31 18:46 database.tf -rw-r--r-- 1 calero calero 38 May 31 18:46 provider.tf -rw-r--r-- 1 calero calero 171 May 31 18:46 vars.tf $ $ cd discover $ terraform plan oci_database_autonomous_database.export_example_autonomous_data_warehouse: Refreshing state... [id=ocid1.autonomousdatabase.oc1.iad.anuwcljst77gzoiaaewxote6h4i73oamzoloxdueabnpwprspzngjoce52xq] oci_database_autonomous_database.export_example_autonomous_database: Refreshing state... [id=ocid1.autonomousdatabase.oc1.iad.anuwcljtt77gzoiamz3gg5cfsbdqgcoj7lrobis75qp7lwubm7d77bpblqgq] No changes. Your infrastructure matches the configuration. Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed. $
  • 20. Terraform idiosyncrasy - 2) concurrent work © Pythian Services Inc 2021 | 20 $ terraform plan Refreshing Terraform state in-memory prior to plan... The refreshed state will be used to calculate this plan, but will not be persisted to local or remote state storage. data.aws_security_group.sgpr: Refreshing state... aws_rds_cluster.master: Refreshing state... [id=nelson-auroramysql56] aws_rds_cluster_instance.cluster_instances[0]: Refreshing state... [id=nelson-auroramysql56-0] aws_rds_cluster_instance.cluster_instances[1]: Refreshing state... [id=nelson-auroramysql56-1] mysql_user.monitor: Refreshing state... [id=monitor@%] Error: Could not connect to server: dial tcp 127.0.0.1:3306: connect: connection refused
  • 21. Terraform idiosyncrasy - concurrent work © Pythian Services Inc 2021 | 21 $ terraform plan Refreshing Terraform state in-memory prior to plan... The refreshed state will be used to calculate this plan, but will not be persisted to local or remote state storage. data.aws_security_group.sgpr: Refreshing state... aws_rds_cluster.master: Refreshing state... [id=nelson-auroramysql56] aws_rds_cluster_instance.cluster_instances[0]: Refreshing state... [id=nelson-auroramysql56-0] aws_rds_cluster_instance.cluster_instances[1]: Refreshing state... [id=nelson-auroramysql56-1] mysql_user.checkmk_mon: Refreshing state... [id=checkmk_mon@%] Error: Could not connect to server: dial tcp 127.0.0.1:3306: connect: connection refused $ terraform plan ... Error locking state: Error acquiring the state lock: ConditionalCheckFailedException: The conditional request failed Lock Info: ID: f6f04b96-de6d-754a-7bbe-8d0504f75e79 Path: my-tf-state-286052569778/test/us-west-1/services/my-db/aurora-mysql/terraform.tfstate Operation: OperationTypePlan Who: ncalero@tst-rds01 Version: 0.12.31 Created: 2021-12-28 13:45:41.113713713 +0000 UTC Info: $
  • 22. Terraform idiosyncrasy - concurrent work © Pythian Services Inc 2021 | 22 $ terraform plan Refreshing Terraform state in-memory prior to plan... The refreshed state will be used to calculate this plan, but will not be persisted to local or remote state storage. data.aws_security_group.sgpr: Refreshing state... aws_rds_cluster.master: Refreshing state... [id=nelson-auroramysql56] aws_rds_cluster_instance.cluster_instances[0]: Refreshing state... [id=nelson-auroramysql56-0] aws_rds_cluster_instance.cluster_instances[1]: Refreshing state... [id=nelson-auroramysql56-1] mysql_user.checkmk_mon: Refreshing state... [id=checkmk_mon@%] Error: Could not connect to server: dial tcp 127.0.0.1:3306: connect: connection refused $ terraform plan ... Error locking state: Error acquiring the state lock: ConditionalCheckFailedException: The conditional request failed Lock Info: ID: f6f04b96-de6d-754a-7bbe-8d0504f75e79 Path: my-tf-state-286052569778/stable/us-west-1/services/my-db/aurora-mysql/terraform.tfstate Operation: OperationTypePlan Who: c_ncalero@gds-stg-rds01 Version: 0.12.31 Created: 2021-12-28 13:45:41.113713713 +0000 UTC Info: $ $ terraform force-unlock f6f04b96-de6d-754a-7bbe-8d0504f75e79 Do you really want to force-unlock? Terraform will remove the lock on the remote state. This will allow local Terraform commands to modify this state, even though it may be still be in use. Only 'yes' will be accepted to confirm. Enter a value: yes Terraform state has been successfully unlocked! The state has been unlocked, and Terraform commands should now be able to obtain a new lock on the remote state. $
  • 23. Terraform idiosyncrasy - 3) reduce scope © Pythian Services Inc 2021 | 23 $ terraform plan data.oci_database_autonomous_db_versions.test_adb_versions: Reading... oci_database_autonomous_database.autonomous_data_warehouse: Refreshing state... [id=ocid1.autonomousdatabase.oc1.iad.anuwcljst77gzoiaaewxote6h4i73oamzoloxdueabnpwprspzngjoce52xq] oci_database_autonomous_database.autonomous_database: Refreshing state... [id=ocid1.autonomousdatabase.oc1.iad.anuwcljtt77gzoiamz3gg5cfsbdqgcoj7lrobis75qp7lwubm7d77bpblqgq] data.oci_database_autonomous_databases.autonomous_databases: Reading... ... Plan: 0 to add, 1 to change, 1 to destroy. $ terraform state list data.http.ip data.oci_database_autonomous_db_versions.test_adb_versions oci_database_autonomous_database.autonomous_data_warehouse oci_database_autonomous_database.autonomous_database random_string.adb_admin_password $ terraform plan -target oci_database_autonomous_database.autonomous_database data.oci_database_autonomous_db_versions.test_adb_versions: Reading... oci_database_autonomous_database.autonomous_database: Refreshing state... [id=ocid1.autonomousdatabase.oc1.iad.anuwcljtt77gzoiamz3gg5cfsbdqgcoj7lrobis75qp7lwubm7d77bpblqgq] No changes. Your infrastructure matches the configuration.
  • 24. Terraform idiosyncrasy - 4) parameter validations © Pythian Services Inc 2021 | 24 $ terraform apply … aws_rds_cluster.master[0]: Creating... aws_rds_cluster.master[0]: Still creating... [10s elapsed] aws_rds_cluster.master[0]: Still creating... [20s elapsed] aws_rds_cluster.master[0]: Still creating... [30s elapsed] aws_rds_cluster.master[0]: Still creating... [40s elapsed] aws_rds_cluster.master[0]: Creation complete after 42s [id=my-aurora] aws_rds_cluster_instance.cluster_instances[1]: Creating... aws_rds_cluster_instance.cluster_instances[0]: Creating... Error: error creating RDS Cluster (my-aurora) Instance: InvalidParameterCombination: RDS does not support creating a DB instance with the following combination: DBInstanceClass=db.t3.small, Engine=aurora-mysql, EngineVersion=5.7. For supported combinations of instance class and database engine version, see the documentation. status code: 400, request id: 7c15bd63-b7cf-47a3-8a10-faee35c3e9c4 on main.tf line 7, in resource "aws_rds_cluster_instance" "cluster_instances": 7: resource "aws_rds_cluster_instance" "cluster_instances" { Error: error creating RDS Cluster (my-aurora) Instance: InvalidParameterCombination: RDS does not support creating a DB instance with the following combination: DBInstanceClass=db.t3.small, Engine=aurora-mysql, EngineVersion=5.7. For supported combinations of instance class and database engine version, see the documentation. status code: 400, request id: a8cb9644-fbe2-40ee-929b-18094709cb33 on main.tf line 7, in resource "aws_rds_cluster_instance" "cluster_instances": 7: resource "aws_rds_cluster_instance" "cluster_instances" {
  • 25. Terraform idiosyncrasy - 4) parameter validations ● Terraform pass parameters to the provider - it runs the operation on the cloud service and it fails ● Some providers include data sources with attributes having a list of valid values for that (only if we use those when manipulating the resource) ○ Example: OCI Autonomous database https://registry.terraform.io/providers/oracle/oci/latest/docs/data-sources/database_autonomous_db_versions ● Additionally, simple validations can be incorporated in the variable definition: © Pythian Services Inc 2021 | 25 $ grep -A13 "autonomous_database_db_workload" variables.tf variable "autonomous_database_db_workload" { default = "OLTP" validation { condition = anytrue([ var.autonomous_database_db_workload == "OLTP", var.autonomous_database_db_workload == "DW", var.autonomous_database_db_workload == "APEX", var.autonomous_database_db_workload == "AJD" ]) error_message = "Invalid autonomous_database_db_workload value - allowed OLTP, DW, APEX and AJD." } }
  • 26. Terraform idiosyncrasy - parameter validations © Pythian Services Inc 2021 | 26 Demo 3
  • 27. Terraform idiosyncrasy - 5) reusing code ● Terraform by default read the current directory only ● Can include other directories using the “module” directive: https://learn.hashicorp.com/collections/terraform/modules ● Third party tools to re-using code (among environments and/or deployments) - terragrunt example: ○ Standardize resource attributes (tags, HA, networking, security, parameters) between deployments - in same or different environments and regions ○ Support different database engines and versions ○ Validations (possible but limited to same attribute expression) ○ Modules, variables and environments - directory structure ○ Extra features: hooks, functions, etc. - check https://terragrunt.gruntwork.io/ © Pythian Services Inc 2021 | 27 module "ec2_instances" { source = "terraform-aws-modules/ec2-instance/aws" version = "3.5.0" ...
  • 28. Terraform idiosyncrasy - 6) “features” ● Terraform binary ○ Aurora MySQL upgrade didn’t work on TF versions lower than 0.12.31 (current version: 1.1.7) ● Terraform provider ○ MySQL upgrade didn’t work with AWS provider lower than 3.63.0 (several bugs) ○ Resource discovery implemented in OCI (not in AWS as of today) ● Cloud provider limits/restrictions ○ Resource attributes are not validated (service restrictions) - cause runtime errors ■ allowed instance type for a RDS version (both ways) ■ IOPS parameter when using gp2 storage ■ CLI commands to the rescue (to confirm allowed values) © Pythian Services Inc 2021 | 28
  • 29. Cloud provider idiosyncrasy ● You can get errors today that does not repeat tomorrow, as services improves ● Don’t assume all services work the same for similar operations - RTFM! © Pythian Services Inc 2021 | 29 ● Examples on AWS: ○ MySQL minor upgrade fails without any error - not reproducible days later when repeating the test ○ RDS “create Aurora replica” is not available in some versions after upgrades ○ Aurora space used should be checked using metrics ○ RDS parameters have apply_mode immediate or pending-reboot, but tricky to update running instances automatically (immediate mode) ○ Operations on RDS storage does not allow other operations until they finish - and those can take long ■ Automatic space increment - minutes ■ Change type between gp2 and IOPS - hours
  • 30. Terraform - parameters cheat sheet © Pythian Services Inc 2021 | 30 Format code in .tf files terraform fmt Validate configuration terraform validate List state file resources terraform state list Inspect current state file content terraform show Remove a resource from state file terraform state rm <name_in_state> Import resource into state file terraform import <name_in_state> <name_aws> Remove lock from state file terraform force-unlock <lock_id> Operate on a single resource terraform <op> -target <name_in_state> Console (to test formulas/expressions) terraform console Debug execution TF_LOG="TRACE" TF_LOG_PATH=file.log ; terraform <op>
  • 31. Terraform console © Pythian Services Inc 2021 | 31 [c_ncalero@myhost ~]$ terraform console > substr("19.2",0,2) 19 > split(".","19.2") [ "19", "2", ] > split(".","19.2")[0] 19 > element(split(".","19.2"),0) 19 > join("",["oracle",substr("19.2",0,2), "small"]) oracle19small > join("",["oracle",split(".","19.2")[0]]) oracle19 > exit
  • 32. OCI CLI - useful validations © Pythian Services Inc 2021 | 32 $ export COMPID=$(oci iam compartment list --query "data [?name=='mycmp'].{id:id}" | jq -r '.[].id') $ oci db version list -c $COMPID --db-system-shape "VM.Standard1.1" --output table +-----------------------------+--------------+-----------------+ | is-latest-for-major-version | supports-pdb | version | +-----------------------------+--------------+-----------------+ | True | False | 11.2.0.4 | | False | False | 11.2.0.4.210119 | | False | False | 11.2.0.4.210420 | | True | True | 12.1.0.2 | | False | True | 12.1.0.2.211019 | | False | True | 12.1.0.2.220118 | | False | True | 12.1.0.2.220419 | | True | True | 12.2.0.1 | | False | True | 12.2.0.1.211019 | | False | True | 12.2.0.1.220118 | | False | True | 12.2.0.1.220419 | | True | True | 18.0.0.0 | | False | True | 18.16.0.0 | | True | True | 19.0.0.0 | | False | True | 19.13.0.0 | | False | True | 19.14.0.0 | ● Available database versions for DB Systems using shape VM.Standard1.1 | False | True | 19.15.0.0 | | True | True | 21.0.0.0 | | False | True | 21.4.0.0 | | False | True | 21.5.0.0 | | False | True | 21.6.0.0 | +-------------+--------------+-----------+
  • 33. OCI CLI - useful validations © Pythian Services Inc 2021 | 33 $ export COMPID=$(oci iam compartment list --query "data [?name=='mycmp'].{id:id}" | jq -r '.[].id' ) $ oci db autonomous-database list -c $COMPID --query 'data[].{db_name:"db-name", db_version:"db-version", db_workload:"db-workload", display_name:"display-name", state:"lifecycle-state"}' --output table +--------------------+------------+-------------+-----------------------------------+-----------+ | db_name | db_version | db_workload | display_name | state | +--------------------+------------+-------------+-----------------------------------+-----------+ | adbdw | 19c | DW | example_autonomous_data_warehouse | AVAILABLE | | adboltp | 19c | OLTP | example_autonomous_database | AVAILABLE | +--------------------+------------+-------------+-----------------------------------+-----------+ ● ADBs created in compartment ’mycmp’
  • 34. AWS CLI - useful validations © Pythian Services Inc 2021 | 34 $ aws rds describe-orderable-db-instance-options --engine oracle-ee --db-instance-class db.t3.small --query "OrderableDBInstanceOptions[].{EngineVersion:EngineVersion}" --output text --region eu-west-1 12.1.0.2.v1 12.1.0.2.v10 12.1.0.2.v11 12.1.0.2.v12 12.1.0.2.v13 12.1.0.2.v14 12.1.0.2.v15 12.1.0.2.v16 12.1.0.2.v17 12.1.0.2.v18 12.1.0.2.v19 12.1.0.2.v2 12.1.0.2.v20 12.1.0.2.v21 12.1.0.2.v22 12.1.0.2.v23 12.1.0.2.v24 12.1.0.2.v25 12.1.0.2.v26 12.1.0.2.v27 ● Available Oracle EE versions for shape db.t3.small 12.1.0.2.v3 12.1.0.2.v4 12.1.0.2.v5 12.1.0.2.v6 12.1.0.2.v7 12.1.0.2.v8 12.1.0.2.v9 19.0.0.0.ru-2019-07.rur-2019-07.r1 19.0.0.0.ru-2019-10.rur-2019-10.r1 19.0.0.0.ru-2020-01.rur-2020-01.r1 19.0.0.0.ru-2020-04.rur-2020-04.r1 19.0.0.0.ru-2020-07.rur-2020-07.r1 19.0.0.0.ru-2020-10.rur-2020-10.r1 19.0.0.0.ru-2021-01.rur-2021-01.r1 19.0.0.0.ru-2021-01.rur-2021-01.r2 19.0.0.0.ru-2021-04.rur-2021-04.r1 19.0.0.0.ru-2021-07.rur-2021-07.r1 19.0.0.0.ru-2021-10.rur-2021-10.r1 19.0.0.0.ru-2022-01.rur-2022-01.r1
  • 35. AWS CLI - useful validations © Pythian Services Inc 2021 | 35 $ aws rds describe-db-engine-versions --engine oracle-ee --engine-version 12.1.0.2.v20 --query "DBEngineVersions[*].ValidUpgradeTarget[*].{EngineVersion:EngineVersion}" --output text 12.1.0.2.v21 12.1.0.2.v22 12.1.0.2.v23 12.1.0.2.v24 12.1.0.2.v25 12.1.0.2.v26 12.1.0.2.v27 19.0.0.0.ru-2020-04.rur-2020-04.r1 19.0.0.0.ru-2020-07.rur-2020-07.r1 19.0.0.0.ru-2020-10.rur-2020-10.r1 19.0.0.0.ru-2021-01.rur-2021-01.r1 19.0.0.0.ru-2021-01.rur-2021-01.r2 19.0.0.0.ru-2021-04.rur-2021-04.r1 19.0.0.0.ru-2021-07.rur-2021-07.r1 19.0.0.0.ru-2021-10.rur-2021-10.r1 19.0.0.0.ru-2022-01.rur-2022-01.r1 ● Supported upgrade paths for Oracle version 12.1.0.2
  • 36. AWS CLI - useful validations © Pythian Services Inc 2021 | 36 $ REGION=us-west-1; for i in $(awscloudwatch list-metrics --region=$REGION --namespace AWS/RDS --metric-name VolumeBytesUsed --query "Metrics[].Dimensions[].{Name:Name, Value:Value}" --output table | grep DBClusterIdentifier | cut -d"|" -f3) ; do echo -n $i " " ; aws cloudwatch get-metric-statistics --region $REGION --namespace AWS/RDS --metric-name VolumeBytesUsed --dimensions Name=EngineName,Value=aurora-mysql --dimensions Name=EngineName,Value=aurora Name=DbClusterIdentifier,Value=$i --start-time $(date -u -d "1 days ago" '+%F') --end-time $(date -u '+%F') --statistics Maximum --period 3600 --query "Datapoints[].{Timestamp:Timestamp, Maximum:Maximum, Unit:Unit}" | jq '.[] | .Maximum' | sort -u | tail -1 ; done | sort bookshop-db 7381361362844 sales-db 179468653458 track-db 20681679872 programs-db 35563371113 analytics-db 8650433415 master-db 17432850551 resourcing-db 6418328690 … ● Space in use by aurora Mysql instances running in region us-west-1
  • 37. AWS CLI - useful validations © Pythian Services Inc 2021 | 37 $ aws rds describe-db-parameters --db-parameter-group-name mypg --region us-west-1 --query "sort_by(Parameters,&ParameterName)[?Source=='user'].{ParameterName:ParameterName, ApplyMethod:ApplyMethod, IsModifiable:IsModifiable, ParameterValue:ParameterValue}" --output table +----------------+---------------+-----------------------------------+------------------+ | ApplyMethod | IsModifiable | ParameterName | ParameterValue | +----------------+---------------+-----------------------------------+------------------+ | immediate | True | event_scheduler | ON | | immediate | True | group_concat_max_len | 16384 | | pending-reboot| True | innodb_sort_buffer_size | 8388608 | | immediate | True | log_bin_trust_function_creators | 1 | | immediate | True | long_query_time | 0.1 | | pending-reboot| True | max_allowed_packet | 1073741824 | | immediate | True | max_connections | 8000 | | immediate | True | slow_query_log | 1 | +----------------+---------------+-----------------------------------+------------------+ ● Parameters set by user (we) in a RDS parameter group
  • 38. Questions? © Pythian Services Inc 2021 | 38 calero@pythian.com @ncalerouy http://www.linkedin.com/in/ncalero