The Grand Puppet Sub-Systems Tour - Nicholas Fagerlund, Puppet Labs

1. 2014
Presented by
The Grand Puppet
Sub-Systems Tour
Nick Fagerlund
Technical Writer / Weird Bugs | Puppet
Labs
@nfagerlund

2. Presented by
PIPES

3. Presented by
What does Puppet DO?
• Gather system information
• Compile a catalog
• Apply catalog
• Report

4. Presented by
What does Puppet DO?
AGENT
• Gather system
information
• Apply catalog
• Report
MASTER
• Compile a catalog

5. STOP 1: Command Line and the
Presented by

6. Presented by
Someone ran puppet agent

7. Presented by
???
• /opt/puppet/bin/puppet
• Puppet::Util::CommandLine.new.execute
• Puppet::Application.find
• Puppet::Application::Agent.new
• Puppet::Application::Agent#setup
• Puppet::Application::Agent#run_command

8. Presented by
#run_command
#
lib/puppet/application/agent.rb
def
run_command
if
options[:fingerprint]
fingerprint
else
#
It'd
be
nice
to
daemonize
later,
but
we
have
to
daemonize
before
#
waiting
for
certificates
so
that
we
don't
block
daemon
=
daemonize_process_when(Puppet[:daemonize])
!
wait_for_certificates
!
if
Puppet[:onetime]
onetime(daemon)
else
main(daemon)
end

9. Presented by
onetime() and main()
#
lib/puppet/application/agent.rb
begin
exitstatus
=
daemon.agent.run
rescue
=>
detail
Puppet.log_exception(detail)
end
!
daemon.stop(:exit
=>
false)
!
if
not
exitstatus
exit(1)
elsif
options[:detailed_exitcodes]
then
exit(exitstatus)
else
exit(0)
end
#
lib/puppet/application/agent.rb
def
main(daemon)
if
Puppet[:listen]
setup_listen(daemon)
end
Puppet.notice
"Starting
Puppet
client
version
#{Puppet.version}"
!
daemon.start
end

10. Presented by
Obtaining the daemon
#
lib/puppet/application/agent.rb
def
daemonize_process_when(should_daemonize)
daemon
=
Puppet::Daemon.new(Puppet::Util::Pidlock.new(Puppet[:pidfile]))
daemon.argv
=
@argv
daemon.agent
=
@agent
!
daemon.daemonize
if
should_daemonize
!
daemon
end

11. STOP 2: The Daemon and the
Presented by

12. Presented by
DAEMON LAND?!
Nope
Bye

13. Presented by
Puppet::Agent
• Run a thing

14. Presented by
Puppet::Daemon
• Run a thing that runs the other thing

15. Lots of layers of abstraction??
• Historical reasons!
Presented by

16. Presented by
STOP 3: The Configurer

17. Presented by
Puppet::Configurer

18. Puppet::Configurer#run_internal
Presented by
#
lib/puppet/configurer.rb
def
run_internal(options)
#
We
create
the
report
pre-­‐populated
with
default
settings
for
#
environment
and
transaction_uuid
very
early,
this
is
to
ensure
#
they
are
sent
regardless
of
any
catalog
compilation
failures
or
#
exceptions.
options[:report]
||=
Puppet::Transaction::Report.new("apply",
nil,
@environment,
@transaction_uuid)
report
=
options[:report]
init_storage
!
Puppet::Util::Log.newdestination(report)
…
…
…

19. Puppet::Configurer#run_internal
Presented by
#
lib/puppet/configurer.rb
…
query_options
=
get_facts(options)
unless
query_options
…

20. Puppet::Configurer#run_internal
Presented by
#
lib/puppet/configurer/fact_handler.rb
…
facts
=
Puppet::Node::Facts.indirection.find(Puppet[:node_name_value],
:environment
=>
@environment)
…

21. Puppet::Configurer#run_internal
Presented by
#
lib/puppet/configurer.rb
…
unless
catalog
=
prepare_and_retrieve_catalog(options,
query_options)
return
nil
end
…

22. Puppet::Configurer#run_internal
Presented by
#
lib/puppet/configurer.rb
…
apply_catalog(catalog,
options)
…

23. Presented by
Where next?
Retrieve Apply

24. STOP 4: The Transaction
Presented by

25. Presented by
(Puppet::Resource::Catalog)

26. Presented by
Puppet::Resource::Catalog
Catalog
Resource Other
Stuff

27. Presented by
Puppet::Configurer (again)
#
lib/puppet/configurer.rb
def
convert_catalog(result,
duration)
catalog
=
result.to_ral
catalog.finalize
catalog.retrieval_duration
=
duration
catalog.write_class_file
catalog.write_resource_file
catalog
end

28. Presented by
RAL Catalog

29. Presented by
RAL Catalog
#
lib/puppet/resource/catalog.rb
def
to_catalog(convert)
result
=
self.class.new(self.name,
self.environment_instance)
result.version
=
self.version
map
=
{}
resources.each
do
|resource|
next
if
virtual_not_exported?(resource)
next
if
block_given?
and
yield
resource
newres
=
resource.copy_as_resource
newres.catalog
=
result
if
convert
!=
:to_resource
newres
=
newres.to_ral
end
#
We
can't
guarantee
that
resources
don't
munge
their
names
#
(like
files
do
with
trailing
slashes),
so
we
have
to
keep
track
#
of
what
a
resource
got
converted
to.
map[resource.ref]
=
newres
result.add_resource
newres
end

30. Presented by
RAL Catalog
Puppet::Resource
!
↓
!
(something) << Puppet::Type

31. Presented by
RAL Catalog
???

32. Presented by
RAL Catalog
#
lib/puppet/configurer.rb
def
apply_catalog(catalog,
options)
report
=
options[:report]
report.configuration_version
=
catalog.version
!
benchmark(:notice,
"Finished
catalog
run")
do
catalog.apply(options)
end
!
report.finalize_report
report
end

33. Puppet::Resource::Catalog#apply
Presented by
#
lib/puppet/resource/catalog.rb
def
apply(options
=
{})
Puppet::Util::Storage.load
if
host_config?
!
transaction
=
create_transaction(options)
!
begin
transaction.report.as_logging_destination
do
transaction.evaluate
end
rescue
Puppet::Error
=>
detail
!

34. Presented by
Transaction
???

35. Presented by
Transaction
• Ral catalog
• Prioritizer
• Report

36. Presented by
Transaction
• Ral catalog
• Prioritizer
• Report
• Event manager
• Resource harness
• Relationship graph

37. Presented by
Transaction
• Ral catalog
• Prioritizer
• Report
• Event manager
• Resource harness
• Relationship graph
• relationship_graph.traverse

38. Presented by
Transaction
• Ral catalog
• Prioritizer
• Report
• Event manager
• Resource harness
• Relationship graph
• relationship_graph.traverse
• *ollies out*

39. Presented by
STOP 5: The Indirector

40. Presented by
Back to Puppet::Configurer
• #prepare_and_retrieve_catalog

41. Presented by
Back to Puppet::Configurer
#
lib/puppet/configurer.rb
result
=
Puppet::Resource::Catalog.indirection.find(
Puppet[:node_name_value],
query_options.merge(:ignore_cache
=>
true,
:environment
=>
@environment,
:fail_on_404
=>
true)
)

42. Presented by
The Indirector
!
Puppet::Resource::Catalog.indirection.find()

43. Presented by
The Indirector
←
↑
↓
?

44. Presented by
The Indirector
• Globally configured
• Abstract
• Gets stuff for you

45. Presented by
The Indirector
#
lib/puppet/resource/catalog.rb
extend
Puppet::Indirector
indirects
:catalog,
:terminus_setting
=>
:catalog_terminus

46. Presented by
The Indirector
• find
• search
• save
• destroy

47. Presented by
The Indirector
• find
• search
• save
• destroy
• terminus_class
• cache_class
• lib/puppet/indirector/…

48. Presented by
Puppet Agent
• Catalog => REST
• Report => REST
• Facts => Facter

49. Presented by
Puppet Apply
• Catalog => Compiler
• Report => Processor
• Facts => Facter

50. Presented by
Avoiding the cache
#
lib/puppet/configurer.rb
result
=
Puppet::Resource::Catalog.indirection.find(
Puppet[:node_name_value],
query_options.merge(:ignore_cache
=>
true,
:environment
=>
@environment,
:fail_on_404
=>
true)
)

51. Presented by
The Indirector
• Why?

52. Presented by
The Indirector
• Why?
• Global? Rigid?

53. Presented by
The Indirector
• Why?
• Global? Rigid?
• IT MAKES SENSE IN CONTEXT

54. Presented by
The Indirector
• …Catalog.find( <FACTS> )
• REST terminus
• HTTP POST request to master

55. STOP 6: The Puppet Master’s
Presented by

56. Presented by
Rack
Web server
↓
Rack server
↓
“App” object that responds to #call()
↓
Actual application logic

57. Presented by
config.ru
$0
=
"master"
!
ARGV
<<
"-­‐-­‐rack"
!
ARGV
<<
"-­‐-­‐confdir"
<<
"/etc/puppet"
ARGV
<<
"-­‐-­‐vardir"
<<
"/var/lib/puppet"
!
require
'puppet/util/command_line'
run
Puppet::Util::CommandLine.new.execute
!

58. Presented by
Starting master w/ Rack
#
lib/puppet/application/master.rb
def
start_rack_master
require
'puppet/network/http/rack'
!
announce_start_of_master
!
return
Puppet::Network::HTTP::Rack.new()
end

59. Presented by
The app object
#
lib/puppet/network/http/rack.rb
class
Puppet::Network::HTTP::Rack
def
call(env)
request
=
Rack::Request.new(env)
response
=
Rack::Response.new
Puppet.debug
'Handling
request:
%s
%s'
%
[request.request_method,
request.fullpath]
!
begin
Puppet::Network::HTTP::RackREST.new.process(request,
response)
rescue
=>
detail
#
Send
a
Status
500
Error
on
unhandled
exceptions.
response.status
=
500
response['Content-­‐Type']
=
'text/plain'
response.write
'Internal
Server
Error:
"%s"'
%
detail.message
#
log
what
happened
Puppet.log_exception(detail,
"Puppet
Server
(Rack):
Internal
Server
Error:
Unhandled
Exception:
"%s""
%
detail.message)
end
response.finish

60. Presented by
#process
#
lib/puppet/network/http/handler.rb
def
process(request,
response)
new_response
=
Puppet::Network::HTTP::Response.new(self,
response)
!
request_headers
=
headers(request)
request_params
=
params(request)
request_method
=
http_method(request)
request_path
=
path(request)
…
!

61. Presented by
#process
#
lib/puppet/network/http/handler.rb
if
route
=
@routes.find
{
|route|
route.matches?(new_request)
}
route.process(new_request,
new_response)

62. lib/puppet/network/http/api/v1.rb
Presented by

63. lib/puppet/network/http/api/v1.rb
• GET => indirection.find()
• POST => indirection.find() (for catalogs)
• (There was an issue with some servers rejecting GET
parameters that were too long. Funny story about that.)
• DELETE => indirection.destroy()
• PUT => indirection.save()
• GET to magical plural name => indirection.search()
Presented by

64. Presented by
BIG PICTURE
Configurer calls catalog.indirection.find()
↓
HTTP POST request to master
↓
Request handler calls catalog.indirection.find()
↓
Compiler terminus handles it
↓

65. Presented by
STOP 7: The Compiler

66. Presented by
MOSTLY SKIPPING THIS
Lexer
↓
Parser
↓
AST
↓
Evaluator
↓

67. Presented by
THIS IS THE TEST
“Compiler terminus to the catalog
indirection”
!
???

68. Presented by
THIS IS THE TEST
GOOD WORK
EVERYBODY!

69. Presented by
Questions?

70. Presented by
THANKS!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
•Hope we’ll see you next year in
Portland!