SlideShare ist ein Scribd-Unternehmen logo
1 von 51
Downloaden Sie, um offline zu lesen
Tyler Langlois, October 12th 2017
Software Engineer, Elastic
@leothrix, github: tylerjl
Custom Types and
Providers: Modeling Modern
REST Interfaces and Beyond
2
Obligatory “About Me” Slide
• Been with company since 2014
• Co-maintainer of Elastic Puppet modules (primarily Elasticsearch and
Kibana)
• Puppet-ing in one way or another over my whole professional career
• Brought too many Elastic stickers that need to be given away (please
partake)
• Talk to me about Elasticsearch/Logstash/Kibana/Beats!
Infrastructure/Operations/Software Engineer @ Elastic
3
Who is This Presentation For?
Developers who work with Puppet modules
Puppet users who want to dip into native type/provider development
“What in the %@#$ is the Elasticsearch module doing”
Operators who want to automate against APIs
Hopefully empowers you to implement custom resources on your own
4
WHAT DOES THIS TALK’S
TITLE EVEN MEAN?
5
6
Managing Resources
with Raw APIs
Example: CloudFormation
• Pro:
• Infrastructure resources are data
• Extensible
• Con:
• Managing changes
• Grokking huge chunks of JSON
7
Modeling Resources with
Raw APIs
Example: Terraform
• Pro:
• Readable
• Manageable
• Lifecycle + changes
• Interoperability between other
systems
8
Modeling Resources in Puppet
A DSL to Model
Disparate Resources
A Graph to Manage
Relationships
A Concept of
Changes to
Manage Lifecycles
ls, stat,
chmod,
chown
sysv,
systemd,
upstart
deb, rpm,
pkg
}
9
Modeling Resources in Puppet
Abstraction is Powerful
file { “/tmp/foo”:
source => “puppet:///foo”,
} ->
package { “foo”:
source => “/tmp/foo”
} ~>
service { “foo”:
ensure => “running”,
}
10
Modeling Resources in Puppet
?
Elasticsearch
Logstash
Other
REST
APIs
}
11
Modeling Resources in Puppet
Extending the idea to APIs
elasticsearch::template { “logstash”:
content => {
“template” => “*”,
“settings” => {
“number_of_replicas” => 0
}
}
} ->
service { “es-app”:
ensure => “running”
}
12
Modeling Resources in Puppet
• State Changes
• Instead of comparing changes with GET responses and template files,
compare during a no-op
• A change in state can form dependencies and refresh events into other
resources
• Trickling changes up via reports lends better visibility
Benefits
13
Modeling Resources in Puppet
• State Changes
• More finely-grained control
• Most resources can be represented as Puppet hashes, so Hiera can
be fully leveraged
• Communicating via full Ruby HTTP libraries means CA files, auth, and
more are easier to control
• TESTS!
Benefits
14
Modeling Resources in Puppet
• State Changes
• More finely-grained control
• Some existing API-based resources:
• Kubernetes module (swagger-generated)
• Google Cloud
• Following examples will be low-level (i.e. with just native Ruby HTTP
libraries)
• …hopefully, will help you write your own for $system
15
Let’s (briefly) talk
about Puppet Types
and Providers
16
Types, Providers, and their Resources
Underlying
Resource
Puppet
Provider
Puppet
Type
• Has some way to change a
property
• Its state is introspectable and
discoverable
• Uniquely identified
• How Ruby interacts with
actual commands/system
properties
• Knows how to discover the
properties of resources
• Normalized provider API to
Puppet DSL
• Somewhat typed, catalog
compilation
• Abstraction over providers
17
Types, Providers, and their Resources: service
• systemctl/service/rc
commands
• Startup visibility with enable/
chkconfig/etc.
• Primarily shell-based for state
• One provider for each init
system
• Ruby knows which shell
commands to invoke to start,
stop, enable, etc.
• Unified API to start, enable,
and restart a general
service resource
• Abstraction over provider-
specific implementations
• What we see in a manifest
Underlying
Resource
Puppet
Provider
Puppet
Type
18
Types, Providers, and their Resources: elasticsearch
• REST API endpoints
• Objects modeled in JSON
• Individual endpoints via
_template, _ingest, etc.
• One provider base class, one
provider per resource type
• Using native Ruby HTTP APIs
are high-level enough
• Better alternative than `exec
{ “curl”:`
• Resource properties
expressed in Puppet DSL
hashes
• We don’t make API calls, we
declare desired state
Underlying
Resource
Puppet
Provider
Puppet
Type
19
Then:
20
Now:
Case Study:
Elasticsearch
Pipelines
curl vs. Puppet
22
Ingest Pipelines
23
Ingest Pipelines
24
Ingest Pipelines
• All pipelines are uniquely identified by a name (like defined or native types!)
• Endpoints to manage pipelines:
• GET to retrieve JSON object enumerating all pipelines
• Note: can also retrieved based by name alone
• PUT to create with JSON body
• Note that we’re using unauthenticated APIs right now
Key observations
25
Ingest Pipelines: Puppet Type
26
Ingest Pipelines: Puppet Type (Implementation)
27
Ingest Pipelines: Puppet Type (Implementation)
…what the included abstraction does
28
Ingest Pipelines: Puppet Provider (Implementation)
29
Ingest Pipelines: Puppet Provider (details)
…what the parent class does
30
Ingest Pipelines: Puppet Provider (details)
…what the parent class does
31
Ingest Pipelines: Puppet Tests
32
Ingest Pipelines
• That’s most of it!
• Test-driven development + rspec makes it smooth
• Bulk is abstracted; the beefy parts are in parent classes and reused by
templates, indices, etc.
• Native types and providers ≠ scary
Summary
33
Fitting REST Resources Into Puppet
Considerations
`exists?` versus `prefetch`
Leveraging type-level tools
HTTP
API availability
1
2
3
4
34
35
An Example of Returning a Hash to Prefetch
Automatically Gathering Resources
uri = URI(“http://localhost:9200/_template”)
http = Net::HTTP.new uri.host, uri.port
req = Net::HTTP::Get.new uri.request_uri
response = http.request req
JSON.parse(response.body).map do |object_name, api_object|
{
:name => object_name,
:ensure => :present,
:content => api_object,
:provider => name
}
end
36
Advantages
• puppet resource functionality
• Minimizes chatter with API endpoints
• i.e., checking for existence versus properties, etc.
• Call flush only when necessary
• Additional API freebies (i.e., centralized access in flush(), etc.)
Prefetching resources versus vanilla exists?
37
38
Fitting REST Resources Into Puppet
Considerations
`exists?` versus `prefetch`
Leveraging type-level tools
HTTP
API availability
1
2
3
4
39
Response Content vs. Request Content
Usually never 1:1 mappings
{
"logstash": {
"order": 0,
"version": 60001,
"index_patterns": [
"logstash-*"
],
. . .
elasticsearch::template {
'logstash':
content => {
'template' => '*',
'settings' => {
. . .
vs.
40
Types To the Rescue
• A resource’s desired state is almost never the plain response for a query
against the resource
• Example: kubernetes Deployment versus the state of a Deployment
• munge can help unify the resource versus JSON for comparability
• insync? can be enhanced to understand which fields are being explicitly
controlled by a user
• e.g., I want {“foo”: “bar”} set, I don’t care about what’s in
{“another”: “field”}
• Used pretty heavily in puppet-elasticsearch
Managing response data
41
Example: Setting Default Fields
Elasticsearch template
# Set default values for templates
munge do |value|
{
'order' => 0,
'aliases' => {},
'mappings' => {}
}.merge(value)
end
42
Example: Unifying Formatting
Elasticsearch template
# Normalize then compare the Puppet hash and json
def insync?(is)
Puppet_X::Elastic.deep_implode(is) == 
Puppet_X::Elastic.deep_implode(should)
end
{ “foo”:
{ “bar”: “value” }
}
{
“foo.bar”: “value”
}
43
Fitting REST Resources Into Puppet
Considerations
`exists?` versus `prefetch`
Leveraging type-level tools
HTTP
API availability
1
2
3
4
44
HTTP In Providers
45
HTTP In Providers
• Native HTTP libraries let us more easily control and pass:
• TLS certificate authorities and verification booleans
• HTTP basic auth credentials
• Failure cases (timeouts, 4xx/5xx response codes, etc.)
• In this case with Elasticsearch, error responses can return JSON
messages for more helpful Puppet failures
46
Fitting REST Resources Into Puppet
Considerations
`exists?` versus `prefetch`
Leveraging type-level tools
HTTP
API availability
1
2
3
4
47
API Availability
• What happens if:
• An API-based REST resource requires an API to be up, not just a
daemon?
• A resource should block until one is available?
• An unrelated resource needs that API as well?
Weird edge cases when controlling APIs as opposed to hosts
48
API Availability
• es_instance_conn_validator doesn’t resolve until a connection can
be made
Some observations
after a couple years…
50
Results From the Field
• One parent class makes
creating more easy
• Supported REST-based
resources include:
• indices
• templates
• pipelines
• + more
Extensibility
• rspec + webmock for
great testing
• ES docs + specs first
have made some
implementations first try
successes
• Good mocks make some
acceptance tests
unnecessary (faster CI!)
Reliability
• Much easier to extend to
new OS’s (i.e., Windows)
• Greater control has made
some tasks (like 3.x →
4.x module update)
smooth
+ more
51
Questions?
• github.com/elastic
Thank You!

Weitere ähnliche Inhalte

Was ist angesagt?

Writing Well-Behaved Unix Utilities
Writing Well-Behaved Unix UtilitiesWriting Well-Behaved Unix Utilities
Writing Well-Behaved Unix UtilitiesRob Miller
 
Ruby and Distributed Storage Systems
Ruby and Distributed Storage SystemsRuby and Distributed Storage Systems
Ruby and Distributed Storage SystemsSATOSHI TAGOMORI
 
ElasticSearch in action
ElasticSearch in actionElasticSearch in action
ElasticSearch in actionCodemotion
 
The Why and How of Scala at Twitter
The Why and How of Scala at TwitterThe Why and How of Scala at Twitter
The Why and How of Scala at TwitterAlex Payne
 
A quick introduction to Storm Crawler
A quick introduction to Storm CrawlerA quick introduction to Storm Crawler
A quick introduction to Storm CrawlerJulien Nioche
 
ElasticSearch - DevNexus Atlanta - 2014
ElasticSearch - DevNexus Atlanta - 2014ElasticSearch - DevNexus Atlanta - 2014
ElasticSearch - DevNexus Atlanta - 2014Roy Russo
 
Your Data, Your Search, ElasticSearch (EURUKO 2011)
Your Data, Your Search, ElasticSearch (EURUKO 2011)Your Data, Your Search, ElasticSearch (EURUKO 2011)
Your Data, Your Search, ElasticSearch (EURUKO 2011)Karel Minarik
 
Clojure in real life 17.10.2014
Clojure in real life 17.10.2014Clojure in real life 17.10.2014
Clojure in real life 17.10.2014Metosin Oy
 
Distributed tracing with erlang/elixir
Distributed tracing with erlang/elixirDistributed tracing with erlang/elixir
Distributed tracing with erlang/elixirIvan Glushkov
 
Building Distributed Systems in Scala
Building Distributed Systems in ScalaBuilding Distributed Systems in Scala
Building Distributed Systems in ScalaAlex Payne
 
딥러닝프레임워크비교
딥러닝프레임워크비교딥러닝프레임워크비교
딥러닝프레임워크비교Junyi Song
 
Lessons learned while building Omroep.nl
Lessons learned while building Omroep.nlLessons learned while building Omroep.nl
Lessons learned while building Omroep.nlbartzon
 
Elasticsearch - DevNexus 2015
Elasticsearch - DevNexus 2015Elasticsearch - DevNexus 2015
Elasticsearch - DevNexus 2015Roy Russo
 
Introduction to Elasticsearch
Introduction to ElasticsearchIntroduction to Elasticsearch
Introduction to ElasticsearchRuslan Zavacky
 
Kotlin @ Coupang Backend 2017
Kotlin @ Coupang Backend 2017Kotlin @ Coupang Backend 2017
Kotlin @ Coupang Backend 2017Sunghyouk Bae
 
Intro to elasticsearch
Intro to elasticsearchIntro to elasticsearch
Intro to elasticsearchJoey Wen
 
Internship final report@Treasure Data Inc.
Internship final report@Treasure Data Inc.Internship final report@Treasure Data Inc.
Internship final report@Treasure Data Inc.Ryuichi ITO
 
Cool bonsai cool - an introduction to ElasticSearch
Cool bonsai cool - an introduction to ElasticSearchCool bonsai cool - an introduction to ElasticSearch
Cool bonsai cool - an introduction to ElasticSearchclintongormley
 

Was ist angesagt? (20)

Writing Well-Behaved Unix Utilities
Writing Well-Behaved Unix UtilitiesWriting Well-Behaved Unix Utilities
Writing Well-Behaved Unix Utilities
 
Ruby and Distributed Storage Systems
Ruby and Distributed Storage SystemsRuby and Distributed Storage Systems
Ruby and Distributed Storage Systems
 
ElasticSearch in action
ElasticSearch in actionElasticSearch in action
ElasticSearch in action
 
The Why and How of Scala at Twitter
The Why and How of Scala at TwitterThe Why and How of Scala at Twitter
The Why and How of Scala at Twitter
 
A quick introduction to Storm Crawler
A quick introduction to Storm CrawlerA quick introduction to Storm Crawler
A quick introduction to Storm Crawler
 
ElasticSearch - DevNexus Atlanta - 2014
ElasticSearch - DevNexus Atlanta - 2014ElasticSearch - DevNexus Atlanta - 2014
ElasticSearch - DevNexus Atlanta - 2014
 
Your Data, Your Search, ElasticSearch (EURUKO 2011)
Your Data, Your Search, ElasticSearch (EURUKO 2011)Your Data, Your Search, ElasticSearch (EURUKO 2011)
Your Data, Your Search, ElasticSearch (EURUKO 2011)
 
Clojure in real life 17.10.2014
Clojure in real life 17.10.2014Clojure in real life 17.10.2014
Clojure in real life 17.10.2014
 
Distributed tracing with erlang/elixir
Distributed tracing with erlang/elixirDistributed tracing with erlang/elixir
Distributed tracing with erlang/elixir
 
Building Distributed Systems in Scala
Building Distributed Systems in ScalaBuilding Distributed Systems in Scala
Building Distributed Systems in Scala
 
Elastic Search
Elastic SearchElastic Search
Elastic Search
 
딥러닝프레임워크비교
딥러닝프레임워크비교딥러닝프레임워크비교
딥러닝프레임워크비교
 
Lessons learned while building Omroep.nl
Lessons learned while building Omroep.nlLessons learned while building Omroep.nl
Lessons learned while building Omroep.nl
 
Elasticsearch - DevNexus 2015
Elasticsearch - DevNexus 2015Elasticsearch - DevNexus 2015
Elasticsearch - DevNexus 2015
 
STORM
STORMSTORM
STORM
 
Introduction to Elasticsearch
Introduction to ElasticsearchIntroduction to Elasticsearch
Introduction to Elasticsearch
 
Kotlin @ Coupang Backend 2017
Kotlin @ Coupang Backend 2017Kotlin @ Coupang Backend 2017
Kotlin @ Coupang Backend 2017
 
Intro to elasticsearch
Intro to elasticsearchIntro to elasticsearch
Intro to elasticsearch
 
Internship final report@Treasure Data Inc.
Internship final report@Treasure Data Inc.Internship final report@Treasure Data Inc.
Internship final report@Treasure Data Inc.
 
Cool bonsai cool - an introduction to ElasticSearch
Cool bonsai cool - an introduction to ElasticSearchCool bonsai cool - an introduction to ElasticSearch
Cool bonsai cool - an introduction to ElasticSearch
 

Andere mochten auch

James Altucher: 40 Alternatives To College
James Altucher: 40 Alternatives To CollegeJames Altucher: 40 Alternatives To College
James Altucher: 40 Alternatives To CollegeJamesAltucher
 
Making Meaningful Maps: Seeing Geography through Cartography
Making Meaningful Maps: Seeing Geography through CartographyMaking Meaningful Maps: Seeing Geography through Cartography
Making Meaningful Maps: Seeing Geography through Cartographyreroth
 
Integration of Risk Assessment and Chemical Characterization (MD&M Minn. 2017)
Integration of Risk Assessment and Chemical Characterization (MD&M Minn. 2017)Integration of Risk Assessment and Chemical Characterization (MD&M Minn. 2017)
Integration of Risk Assessment and Chemical Characterization (MD&M Minn. 2017)Russell Sloboda
 
Ux e o Inconsciente - Odair Faléco
Ux e o Inconsciente - Odair FalécoUx e o Inconsciente - Odair Faléco
Ux e o Inconsciente - Odair FalécoOdair Faléco
 
Exceptional Design is Emotional Design
Exceptional Design is Emotional DesignExceptional Design is Emotional Design
Exceptional Design is Emotional DesignMarissa Louie
 
Writing code you won't hate tomorrow
Writing code you won't hate tomorrowWriting code you won't hate tomorrow
Writing code you won't hate tomorrowRafael Dohms
 
Anti-Money Laundering (AML) Risk Assessment Process
Anti-Money Laundering (AML) Risk Assessment ProcessAnti-Money Laundering (AML) Risk Assessment Process
Anti-Money Laundering (AML) Risk Assessment Processaccenture
 
5 Presentation design trends 2017
5 Presentation design trends 20175 Presentation design trends 2017
5 Presentation design trends 2017Annova Studio
 
SACON - Cyber Risk Assessment Using Bayesian Network (R Venkat)
SACON - Cyber Risk Assessment Using Bayesian Network (R Venkat)SACON - Cyber Risk Assessment Using Bayesian Network (R Venkat)
SACON - Cyber Risk Assessment Using Bayesian Network (R Venkat)Priyanka Aash
 

Andere mochten auch (13)

James Altucher: 40 Alternatives To College
James Altucher: 40 Alternatives To CollegeJames Altucher: 40 Alternatives To College
James Altucher: 40 Alternatives To College
 
Building websites for all
Building websites for allBuilding websites for all
Building websites for all
 
Making Meaningful Maps: Seeing Geography through Cartography
Making Meaningful Maps: Seeing Geography through CartographyMaking Meaningful Maps: Seeing Geography through Cartography
Making Meaningful Maps: Seeing Geography through Cartography
 
Integration of Risk Assessment and Chemical Characterization (MD&M Minn. 2017)
Integration of Risk Assessment and Chemical Characterization (MD&M Minn. 2017)Integration of Risk Assessment and Chemical Characterization (MD&M Minn. 2017)
Integration of Risk Assessment and Chemical Characterization (MD&M Minn. 2017)
 
Audit Risk Assessment Chapter 9
Audit Risk Assessment Chapter 9Audit Risk Assessment Chapter 9
Audit Risk Assessment Chapter 9
 
Ux e o Inconsciente - Odair Faléco
Ux e o Inconsciente - Odair FalécoUx e o Inconsciente - Odair Faléco
Ux e o Inconsciente - Odair Faléco
 
Exceptional Design is Emotional Design
Exceptional Design is Emotional DesignExceptional Design is Emotional Design
Exceptional Design is Emotional Design
 
Writing code you won't hate tomorrow
Writing code you won't hate tomorrowWriting code you won't hate tomorrow
Writing code you won't hate tomorrow
 
Risk management
Risk managementRisk management
Risk management
 
Designing for Context
Designing for ContextDesigning for Context
Designing for Context
 
Anti-Money Laundering (AML) Risk Assessment Process
Anti-Money Laundering (AML) Risk Assessment ProcessAnti-Money Laundering (AML) Risk Assessment Process
Anti-Money Laundering (AML) Risk Assessment Process
 
5 Presentation design trends 2017
5 Presentation design trends 20175 Presentation design trends 2017
5 Presentation design trends 2017
 
SACON - Cyber Risk Assessment Using Bayesian Network (R Venkat)
SACON - Cyber Risk Assessment Using Bayesian Network (R Venkat)SACON - Cyber Risk Assessment Using Bayesian Network (R Venkat)
SACON - Cyber Risk Assessment Using Bayesian Network (R Venkat)
 

Ähnlich wie PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic

Iterator - a powerful but underappreciated design pattern
Iterator - a powerful but underappreciated design patternIterator - a powerful but underappreciated design pattern
Iterator - a powerful but underappreciated design patternNitin Bhide
 
Building APIs in an easy way using API Platform
Building APIs in an easy way using API PlatformBuilding APIs in an easy way using API Platform
Building APIs in an easy way using API PlatformAntonio Peric-Mazar
 
Puppet Fundamentals Talk at DevOps Dubai by Hameedullah Khan
Puppet Fundamentals Talk at DevOps Dubai by Hameedullah KhanPuppet Fundamentals Talk at DevOps Dubai by Hameedullah Khan
Puppet Fundamentals Talk at DevOps Dubai by Hameedullah KhanHameedullah Khan
 
Puppet - The IT automation software
Puppet - The IT automation softwarePuppet - The IT automation software
Puppet - The IT automation softwareagenedy
 
Node collaboration - sharing information between your systems
Node collaboration - sharing information between your systemsNode collaboration - sharing information between your systems
Node collaboration - sharing information between your systemsm_richardson
 
Introduction to Puppet Scripting
Introduction to Puppet ScriptingIntroduction to Puppet Scripting
Introduction to Puppet ScriptingAchieve Internet
 
Why do they call it Linked Data when they want to say...?
Why do they call it Linked Data when they want to say...?Why do they call it Linked Data when they want to say...?
Why do they call it Linked Data when they want to say...?Oscar Corcho
 
Python indroduction
Python indroductionPython indroduction
Python indroductionFEG
 
PuppetConf 2016: Writing Custom Types to Manage Web-Based Applications – Tim ...
PuppetConf 2016: Writing Custom Types to Manage Web-Based Applications – Tim ...PuppetConf 2016: Writing Custom Types to Manage Web-Based Applications – Tim ...
PuppetConf 2016: Writing Custom Types to Manage Web-Based Applications – Tim ...Puppet
 
Rapid API Development ArangoDB Foxx
Rapid API Development ArangoDB FoxxRapid API Development ArangoDB Foxx
Rapid API Development ArangoDB FoxxMichael Hackstein
 
Pragmatic REST APIs
Pragmatic REST APIsPragmatic REST APIs
Pragmatic REST APIsamesar0
 
Performance and Abstractions
Performance and AbstractionsPerformance and Abstractions
Performance and AbstractionsMetosin Oy
 
Writing Custom Puppet Types and Providers to Manage Web-Based Applications
Writing Custom Puppet Types and Providers to Manage Web-Based ApplicationsWriting Custom Puppet Types and Providers to Manage Web-Based Applications
Writing Custom Puppet Types and Providers to Manage Web-Based ApplicationsTim Cinel
 

Ähnlich wie PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic (20)

Iterator - a powerful but underappreciated design pattern
Iterator - a powerful but underappreciated design patternIterator - a powerful but underappreciated design pattern
Iterator - a powerful but underappreciated design pattern
 
Building APIs in an easy way using API Platform
Building APIs in an easy way using API PlatformBuilding APIs in an easy way using API Platform
Building APIs in an easy way using API Platform
 
Puppet Fundamentals Talk at DevOps Dubai by Hameedullah Khan
Puppet Fundamentals Talk at DevOps Dubai by Hameedullah KhanPuppet Fundamentals Talk at DevOps Dubai by Hameedullah Khan
Puppet Fundamentals Talk at DevOps Dubai by Hameedullah Khan
 
Puppet - The IT automation software
Puppet - The IT automation softwarePuppet - The IT automation software
Puppet - The IT automation software
 
L12: REST Service
L12: REST ServiceL12: REST Service
L12: REST Service
 
API Docs with OpenAPI 3.0
API Docs with OpenAPI 3.0API Docs with OpenAPI 3.0
API Docs with OpenAPI 3.0
 
Node collaboration - sharing information between your systems
Node collaboration - sharing information between your systemsNode collaboration - sharing information between your systems
Node collaboration - sharing information between your systems
 
Introduction to Puppet Scripting
Introduction to Puppet ScriptingIntroduction to Puppet Scripting
Introduction to Puppet Scripting
 
Why do they call it Linked Data when they want to say...?
Why do they call it Linked Data when they want to say...?Why do they call it Linked Data when they want to say...?
Why do they call it Linked Data when they want to say...?
 
Introduction_to_Python.pptx
Introduction_to_Python.pptxIntroduction_to_Python.pptx
Introduction_to_Python.pptx
 
Python indroduction
Python indroductionPython indroduction
Python indroduction
 
PuppetConf 2016: Writing Custom Types to Manage Web-Based Applications – Tim ...
PuppetConf 2016: Writing Custom Types to Manage Web-Based Applications – Tim ...PuppetConf 2016: Writing Custom Types to Manage Web-Based Applications – Tim ...
PuppetConf 2016: Writing Custom Types to Manage Web-Based Applications – Tim ...
 
Rapid API Development ArangoDB Foxx
Rapid API Development ArangoDB FoxxRapid API Development ArangoDB Foxx
Rapid API Development ArangoDB Foxx
 
Automation using Puppet 3
Automation using Puppet 3 Automation using Puppet 3
Automation using Puppet 3
 
Pragmatic REST APIs
Pragmatic REST APIsPragmatic REST APIs
Pragmatic REST APIs
 
Performance and Abstractions
Performance and AbstractionsPerformance and Abstractions
Performance and Abstractions
 
Writing Custom Puppet Types and Providers to Manage Web-Based Applications
Writing Custom Puppet Types and Providers to Manage Web-Based ApplicationsWriting Custom Puppet Types and Providers to Manage Web-Based Applications
Writing Custom Puppet Types and Providers to Manage Web-Based Applications
 
Chef for openstack
Chef for openstackChef for openstack
Chef for openstack
 
Api crash
Api crashApi crash
Api crash
 
Api crash
Api crashApi crash
Api crash
 

Mehr von Puppet

Puppet camp2021 testing modules and controlrepo
Puppet camp2021 testing modules and controlrepoPuppet camp2021 testing modules and controlrepo
Puppet camp2021 testing modules and controlrepoPuppet
 
Puppetcamp r10kyaml
Puppetcamp r10kyamlPuppetcamp r10kyaml
Puppetcamp r10kyamlPuppet
 
2021 04-15 operational verification (with notes)
2021 04-15 operational verification (with notes)2021 04-15 operational verification (with notes)
2021 04-15 operational verification (with notes)Puppet
 
Puppet camp vscode
Puppet camp vscodePuppet camp vscode
Puppet camp vscodePuppet
 
Modules of the twenties
Modules of the twentiesModules of the twenties
Modules of the twentiesPuppet
 
Applying Roles and Profiles method to compliance code
Applying Roles and Profiles method to compliance codeApplying Roles and Profiles method to compliance code
Applying Roles and Profiles method to compliance codePuppet
 
KGI compliance as-code approach
KGI compliance as-code approachKGI compliance as-code approach
KGI compliance as-code approachPuppet
 
Enforce compliance policy with model-driven automation
Enforce compliance policy with model-driven automationEnforce compliance policy with model-driven automation
Enforce compliance policy with model-driven automationPuppet
 
Keynote: Puppet camp compliance
Keynote: Puppet camp complianceKeynote: Puppet camp compliance
Keynote: Puppet camp compliancePuppet
 
Automating it management with Puppet + ServiceNow
Automating it management with Puppet + ServiceNowAutomating it management with Puppet + ServiceNow
Automating it management with Puppet + ServiceNowPuppet
 
Puppet: The best way to harden Windows
Puppet: The best way to harden WindowsPuppet: The best way to harden Windows
Puppet: The best way to harden WindowsPuppet
 
Simplified Patch Management with Puppet - Oct. 2020
Simplified Patch Management with Puppet - Oct. 2020Simplified Patch Management with Puppet - Oct. 2020
Simplified Patch Management with Puppet - Oct. 2020Puppet
 
Accelerating azure adoption with puppet
Accelerating azure adoption with puppetAccelerating azure adoption with puppet
Accelerating azure adoption with puppetPuppet
 
Puppet catalog Diff; Raphael Pinson
Puppet catalog Diff; Raphael PinsonPuppet catalog Diff; Raphael Pinson
Puppet catalog Diff; Raphael PinsonPuppet
 
ServiceNow and Puppet- better together, Kevin Reeuwijk
ServiceNow and Puppet- better together, Kevin ReeuwijkServiceNow and Puppet- better together, Kevin Reeuwijk
ServiceNow and Puppet- better together, Kevin ReeuwijkPuppet
 
Take control of your dev ops dumping ground
Take control of your  dev ops dumping groundTake control of your  dev ops dumping ground
Take control of your dev ops dumping groundPuppet
 
100% Puppet Cloud Deployment of Legacy Software
100% Puppet Cloud Deployment of Legacy Software100% Puppet Cloud Deployment of Legacy Software
100% Puppet Cloud Deployment of Legacy SoftwarePuppet
 
Puppet User Group
Puppet User GroupPuppet User Group
Puppet User GroupPuppet
 
Continuous Compliance and DevSecOps
Continuous Compliance and DevSecOpsContinuous Compliance and DevSecOps
Continuous Compliance and DevSecOpsPuppet
 
The Dynamic Duo of Puppet and Vault tame SSL Certificates, Nick Maludy
The Dynamic Duo of Puppet and Vault tame SSL Certificates, Nick MaludyThe Dynamic Duo of Puppet and Vault tame SSL Certificates, Nick Maludy
The Dynamic Duo of Puppet and Vault tame SSL Certificates, Nick MaludyPuppet
 

Mehr von Puppet (20)

Puppet camp2021 testing modules and controlrepo
Puppet camp2021 testing modules and controlrepoPuppet camp2021 testing modules and controlrepo
Puppet camp2021 testing modules and controlrepo
 
Puppetcamp r10kyaml
Puppetcamp r10kyamlPuppetcamp r10kyaml
Puppetcamp r10kyaml
 
2021 04-15 operational verification (with notes)
2021 04-15 operational verification (with notes)2021 04-15 operational verification (with notes)
2021 04-15 operational verification (with notes)
 
Puppet camp vscode
Puppet camp vscodePuppet camp vscode
Puppet camp vscode
 
Modules of the twenties
Modules of the twentiesModules of the twenties
Modules of the twenties
 
Applying Roles and Profiles method to compliance code
Applying Roles and Profiles method to compliance codeApplying Roles and Profiles method to compliance code
Applying Roles and Profiles method to compliance code
 
KGI compliance as-code approach
KGI compliance as-code approachKGI compliance as-code approach
KGI compliance as-code approach
 
Enforce compliance policy with model-driven automation
Enforce compliance policy with model-driven automationEnforce compliance policy with model-driven automation
Enforce compliance policy with model-driven automation
 
Keynote: Puppet camp compliance
Keynote: Puppet camp complianceKeynote: Puppet camp compliance
Keynote: Puppet camp compliance
 
Automating it management with Puppet + ServiceNow
Automating it management with Puppet + ServiceNowAutomating it management with Puppet + ServiceNow
Automating it management with Puppet + ServiceNow
 
Puppet: The best way to harden Windows
Puppet: The best way to harden WindowsPuppet: The best way to harden Windows
Puppet: The best way to harden Windows
 
Simplified Patch Management with Puppet - Oct. 2020
Simplified Patch Management with Puppet - Oct. 2020Simplified Patch Management with Puppet - Oct. 2020
Simplified Patch Management with Puppet - Oct. 2020
 
Accelerating azure adoption with puppet
Accelerating azure adoption with puppetAccelerating azure adoption with puppet
Accelerating azure adoption with puppet
 
Puppet catalog Diff; Raphael Pinson
Puppet catalog Diff; Raphael PinsonPuppet catalog Diff; Raphael Pinson
Puppet catalog Diff; Raphael Pinson
 
ServiceNow and Puppet- better together, Kevin Reeuwijk
ServiceNow and Puppet- better together, Kevin ReeuwijkServiceNow and Puppet- better together, Kevin Reeuwijk
ServiceNow and Puppet- better together, Kevin Reeuwijk
 
Take control of your dev ops dumping ground
Take control of your  dev ops dumping groundTake control of your  dev ops dumping ground
Take control of your dev ops dumping ground
 
100% Puppet Cloud Deployment of Legacy Software
100% Puppet Cloud Deployment of Legacy Software100% Puppet Cloud Deployment of Legacy Software
100% Puppet Cloud Deployment of Legacy Software
 
Puppet User Group
Puppet User GroupPuppet User Group
Puppet User Group
 
Continuous Compliance and DevSecOps
Continuous Compliance and DevSecOpsContinuous Compliance and DevSecOps
Continuous Compliance and DevSecOps
 
The Dynamic Duo of Puppet and Vault tame SSL Certificates, Nick Maludy
The Dynamic Duo of Puppet and Vault tame SSL Certificates, Nick MaludyThe Dynamic Duo of Puppet and Vault tame SSL Certificates, Nick Maludy
The Dynamic Duo of Puppet and Vault tame SSL Certificates, Nick Maludy
 

Kürzlich hochgeladen

UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
Introduction to Quantum Computing
Introduction to Quantum ComputingIntroduction to Quantum Computing
Introduction to Quantum ComputingGDSC PJATK
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
Babel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxBabel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxYounusS2
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataSafe Software
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesThousandEyes
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 

Kürzlich hochgeladen (20)

UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
Introduction to Quantum Computing
Introduction to Quantum ComputingIntroduction to Quantum Computing
Introduction to Quantum Computing
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
Babel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxBabel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptx
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 

PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic

  • 1. Tyler Langlois, October 12th 2017 Software Engineer, Elastic @leothrix, github: tylerjl Custom Types and Providers: Modeling Modern REST Interfaces and Beyond
  • 2. 2 Obligatory “About Me” Slide • Been with company since 2014 • Co-maintainer of Elastic Puppet modules (primarily Elasticsearch and Kibana) • Puppet-ing in one way or another over my whole professional career • Brought too many Elastic stickers that need to be given away (please partake) • Talk to me about Elasticsearch/Logstash/Kibana/Beats! Infrastructure/Operations/Software Engineer @ Elastic
  • 3. 3 Who is This Presentation For? Developers who work with Puppet modules Puppet users who want to dip into native type/provider development “What in the %@#$ is the Elasticsearch module doing” Operators who want to automate against APIs Hopefully empowers you to implement custom resources on your own
  • 4. 4 WHAT DOES THIS TALK’S TITLE EVEN MEAN?
  • 5. 5
  • 6. 6 Managing Resources with Raw APIs Example: CloudFormation • Pro: • Infrastructure resources are data • Extensible • Con: • Managing changes • Grokking huge chunks of JSON
  • 7. 7 Modeling Resources with Raw APIs Example: Terraform • Pro: • Readable • Manageable • Lifecycle + changes • Interoperability between other systems
  • 8. 8 Modeling Resources in Puppet A DSL to Model Disparate Resources A Graph to Manage Relationships A Concept of Changes to Manage Lifecycles ls, stat, chmod, chown sysv, systemd, upstart deb, rpm, pkg }
  • 9. 9 Modeling Resources in Puppet Abstraction is Powerful file { “/tmp/foo”: source => “puppet:///foo”, } -> package { “foo”: source => “/tmp/foo” } ~> service { “foo”: ensure => “running”, }
  • 10. 10 Modeling Resources in Puppet ? Elasticsearch Logstash Other REST APIs }
  • 11. 11 Modeling Resources in Puppet Extending the idea to APIs elasticsearch::template { “logstash”: content => { “template” => “*”, “settings” => { “number_of_replicas” => 0 } } } -> service { “es-app”: ensure => “running” }
  • 12. 12 Modeling Resources in Puppet • State Changes • Instead of comparing changes with GET responses and template files, compare during a no-op • A change in state can form dependencies and refresh events into other resources • Trickling changes up via reports lends better visibility Benefits
  • 13. 13 Modeling Resources in Puppet • State Changes • More finely-grained control • Most resources can be represented as Puppet hashes, so Hiera can be fully leveraged • Communicating via full Ruby HTTP libraries means CA files, auth, and more are easier to control • TESTS! Benefits
  • 14. 14 Modeling Resources in Puppet • State Changes • More finely-grained control • Some existing API-based resources: • Kubernetes module (swagger-generated) • Google Cloud • Following examples will be low-level (i.e. with just native Ruby HTTP libraries) • …hopefully, will help you write your own for $system
  • 15. 15 Let’s (briefly) talk about Puppet Types and Providers
  • 16. 16 Types, Providers, and their Resources Underlying Resource Puppet Provider Puppet Type • Has some way to change a property • Its state is introspectable and discoverable • Uniquely identified • How Ruby interacts with actual commands/system properties • Knows how to discover the properties of resources • Normalized provider API to Puppet DSL • Somewhat typed, catalog compilation • Abstraction over providers
  • 17. 17 Types, Providers, and their Resources: service • systemctl/service/rc commands • Startup visibility with enable/ chkconfig/etc. • Primarily shell-based for state • One provider for each init system • Ruby knows which shell commands to invoke to start, stop, enable, etc. • Unified API to start, enable, and restart a general service resource • Abstraction over provider- specific implementations • What we see in a manifest Underlying Resource Puppet Provider Puppet Type
  • 18. 18 Types, Providers, and their Resources: elasticsearch • REST API endpoints • Objects modeled in JSON • Individual endpoints via _template, _ingest, etc. • One provider base class, one provider per resource type • Using native Ruby HTTP APIs are high-level enough • Better alternative than `exec { “curl”:` • Resource properties expressed in Puppet DSL hashes • We don’t make API calls, we declare desired state Underlying Resource Puppet Provider Puppet Type
  • 24. 24 Ingest Pipelines • All pipelines are uniquely identified by a name (like defined or native types!) • Endpoints to manage pipelines: • GET to retrieve JSON object enumerating all pipelines • Note: can also retrieved based by name alone • PUT to create with JSON body • Note that we’re using unauthenticated APIs right now Key observations
  • 26. 26 Ingest Pipelines: Puppet Type (Implementation)
  • 27. 27 Ingest Pipelines: Puppet Type (Implementation) …what the included abstraction does
  • 28. 28 Ingest Pipelines: Puppet Provider (Implementation)
  • 29. 29 Ingest Pipelines: Puppet Provider (details) …what the parent class does
  • 30. 30 Ingest Pipelines: Puppet Provider (details) …what the parent class does
  • 32. 32 Ingest Pipelines • That’s most of it! • Test-driven development + rspec makes it smooth • Bulk is abstracted; the beefy parts are in parent classes and reused by templates, indices, etc. • Native types and providers ≠ scary Summary
  • 33. 33 Fitting REST Resources Into Puppet Considerations `exists?` versus `prefetch` Leveraging type-level tools HTTP API availability 1 2 3 4
  • 34. 34
  • 35. 35 An Example of Returning a Hash to Prefetch Automatically Gathering Resources uri = URI(“http://localhost:9200/_template”) http = Net::HTTP.new uri.host, uri.port req = Net::HTTP::Get.new uri.request_uri response = http.request req JSON.parse(response.body).map do |object_name, api_object| { :name => object_name, :ensure => :present, :content => api_object, :provider => name } end
  • 36. 36 Advantages • puppet resource functionality • Minimizes chatter with API endpoints • i.e., checking for existence versus properties, etc. • Call flush only when necessary • Additional API freebies (i.e., centralized access in flush(), etc.) Prefetching resources versus vanilla exists?
  • 37. 37
  • 38. 38 Fitting REST Resources Into Puppet Considerations `exists?` versus `prefetch` Leveraging type-level tools HTTP API availability 1 2 3 4
  • 39. 39 Response Content vs. Request Content Usually never 1:1 mappings { "logstash": { "order": 0, "version": 60001, "index_patterns": [ "logstash-*" ], . . . elasticsearch::template { 'logstash': content => { 'template' => '*', 'settings' => { . . . vs.
  • 40. 40 Types To the Rescue • A resource’s desired state is almost never the plain response for a query against the resource • Example: kubernetes Deployment versus the state of a Deployment • munge can help unify the resource versus JSON for comparability • insync? can be enhanced to understand which fields are being explicitly controlled by a user • e.g., I want {“foo”: “bar”} set, I don’t care about what’s in {“another”: “field”} • Used pretty heavily in puppet-elasticsearch Managing response data
  • 41. 41 Example: Setting Default Fields Elasticsearch template # Set default values for templates munge do |value| { 'order' => 0, 'aliases' => {}, 'mappings' => {} }.merge(value) end
  • 42. 42 Example: Unifying Formatting Elasticsearch template # Normalize then compare the Puppet hash and json def insync?(is) Puppet_X::Elastic.deep_implode(is) == Puppet_X::Elastic.deep_implode(should) end { “foo”: { “bar”: “value” } } { “foo.bar”: “value” }
  • 43. 43 Fitting REST Resources Into Puppet Considerations `exists?` versus `prefetch` Leveraging type-level tools HTTP API availability 1 2 3 4
  • 45. 45 HTTP In Providers • Native HTTP libraries let us more easily control and pass: • TLS certificate authorities and verification booleans • HTTP basic auth credentials • Failure cases (timeouts, 4xx/5xx response codes, etc.) • In this case with Elasticsearch, error responses can return JSON messages for more helpful Puppet failures
  • 46. 46 Fitting REST Resources Into Puppet Considerations `exists?` versus `prefetch` Leveraging type-level tools HTTP API availability 1 2 3 4
  • 47. 47 API Availability • What happens if: • An API-based REST resource requires an API to be up, not just a daemon? • A resource should block until one is available? • An unrelated resource needs that API as well? Weird edge cases when controlling APIs as opposed to hosts
  • 48. 48 API Availability • es_instance_conn_validator doesn’t resolve until a connection can be made
  • 49. Some observations after a couple years…
  • 50. 50 Results From the Field • One parent class makes creating more easy • Supported REST-based resources include: • indices • templates • pipelines • + more Extensibility • rspec + webmock for great testing • ES docs + specs first have made some implementations first try successes • Good mocks make some acceptance tests unnecessary (faster CI!) Reliability • Much easier to extend to new OS’s (i.e., Windows) • Greater control has made some tasks (like 3.x → 4.x module update) smooth + more