Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Functional Operations - Susan Potter

444 Aufrufe

Veröffentlicht am

Technical operations is plagued with an unhealthy infatuation of typically untested, imperative code with a high reliance on shared mutable state using dynamically typed languages such as Ruby, Python, Bash, and - ugh - remember Perl? :) In an age where building reliable infrastructure to elastically scale applications and services are paramount to business success, we need to start rethinking the infrastructure engineer’s toolkit and guiding principles. This talk will take a look at applying various functional techniques to building and automating infrastructure. From functional package management and congruent configuration to declarative cloud provisioning we’ll see just how practical these techniques typically used in functional programming for applications can be used to help build more robust and predictable infrastructures. While specific code examples will be given, the emphasis of the talk will be on guiding principles and functional design.

Veröffentlicht in: Daten & Analysen
  • Als Erste(r) kommentieren

  • Gehören Sie zu den Ersten, denen das gefällt!

Functional Operations - Susan Potter

  1. 1. Functional Operations #dmconf1521 November 2015 Susan Potter Lookout twitter: @SusanPotter github: mbbx6spp
  2. 2. % whoami Figure: From developer to (dev)ops engineer
  3. 3. Agenda 1 Motivation for reasoning 2 Review functional programming 101 3 Illustrate forms of reasoning 4 Case study: Reimagine package management
  4. 4. Edsger Dijkstra
  5. 5. Reliability “Those who want really reliable software will discover that they must find means of avoiding the majority of bugs to start with, and as a result the programming process will become cheaper. If you want more effective programmers, you will discover that they should not waste their time debugging, they should not introduce the bugs to start with.”[?]
  6. 6. Why care now? 1 Economic factors necessity of distributed systems
  7. 7. Why care now? 1 Economic factors necessity of distributed systems 2 Human factors high churn/turnover, low quality of ops life
  8. 8. Why care now? 1 Economic factors necessity of distributed systems 2 Human factors high churn/turnover, low quality of ops life 3 Technological factors programmable infrastructure & FP no longer just for academics
  9. 9. The Problem. . .
  10. 10. Application Delivery 1 Provision infrastructure 2 Configure nodes 3 Orchestrate services
  11. 11. Need to support more 1 Application services 2 Environments 3 Data services 4 Distributed services
  12. 12. Optimize for 1 Scalability solved by on-demand ”cloud”
  13. 13. Optimize for 1 Scalability solved by on-demand ”cloud” 2 Reliability solved by . . .
  14. 14. So what yields reliability?
  15. 15. Reason The required techniques of effective reasoning are pretty formal, but as long as programming is done by people that don’t master them, the software crisis will remain with us and will be considered an incurable disease. [?]
  16. 16. Functions 101
  17. 17. Functions have inputs (Ruby) 1 # Two input arguments here 2 def add(x, y) 3 x + y 4 end 5 6 # One input argument here 7 def len(s) 8 s.size 9 end
  18. 18. Functions have inputs (Scala) 1 object Functions { 2 // Two typed input arguments here 3 def add(x: Int , y: Int) = x + y 4 5 // One typed input argument here 6 def len(s: String) = s.size 7 }
  19. 19. Functions return a result 1 scala > add(5, 6) 2 res0: Int = 11 3 4 scala > len("Hello ,␣Barcelona") 5 res1: Int = 16
  20. 20. Only depend on inputs 1/2 1 scala > val defaultTimeout : Int = 30 2 scala > val timeout1: Option[Int] = Some (15) 3 scala > val timeout2: Option[Int] = None 4 scala > :paste 5 def defaulter[A](a: => A, ma: Option[A]) = 6 ma match { 7 case Some(x) => x 8 case None => a 9 }
  21. 21. Only depend on inputs 2/2 1 scala > timeout1 2 timeout1: Option[Int] = Some (15) 3 4 scala > timeout2 5 timeout2: Option[Int] = None 6 7 scala > defaulter(defaultTimeout , timeout1) 8 res0: Int = 15 9 10 scala > defaulter(defaultTimeout , timeout2) 11 res1: Int = 30
  22. 22. Return same result given same inputs 1 scala > len("Hello ,␣Barcelona") 2 res0: Int = 16 3 4 scala > len("Hello ,␣Barcelona") 5 res1: Int = 16 6 ... 7 scala > len("Hello ,␣Barcelona") 8 res3333333333: Int = 16 9 10 scala > // Always!
  23. 23. The Big idea Referential Transparency Given same inputs, return same result. Always.
  24. 24. Functions can use other values 1 // type aliasing a function 2 type Pred[A] = A => Boolean 3 4 // Passing a function as an input argument 5 def is[A](p: Pred[A])(a: A) = p(a) 6 7 // This uses already defined function +is+ 8 def not[A](p: Pred[A])(a: A) = !is(p)(a)
  25. 25. Values can be functions 1 // Returning a function as a value 2 def lessThanN(n: Int): Pred[Int] = _ < n 3 4 // Using two in scope functions 5 def islessThanN(n: Int)(x: Int) = 6 is(ltN(n))(x) 7 8 // Those values can be functions :)
  26. 26. Another important idea Higher Order Functions Build useful functions from simpler functions!
  27. 27. Questions so far? Figure: Awake?
  28. 28. Function Composition
  29. 29. UNIX Pipes 1/4 1 sh > echo -n "Hello ~~~" 2 | sed ’s/~//g’ 3 | tr ’[: lower :]’ ’[:upper :]’ 4 | wc -c 5 5
  30. 30. UNIX Pipes 2/4 1 sh > alias sanitize=’sed "s/~//g"’ 2 sh > alias toUpper=’tr "[: lower :]" "[: upper :]"’ 3 sh > alias len=’wc -c’ 4 5 sh > alias myfun0=’sanitize | toUpper ’ 6 sh > alias myfun1=’myfun0 | wc -c’
  31. 31. UNIX Pipes 3/4 1 sh > echo -n "Hello ~~~" 2 | sed ’s/~//g’ 3 | tr ’[: lower :]’ ’[:upper :]’ 4 | wc -c 5 5 6 7 sh > echo -n "Hello ~~~" | myfun1 8 5
  32. 32. UNIX Pipes 4/4 • Character-based • File descriptors
  33. 33. Function Composition 1/3 1 def toUpper = (s: String) => s.toUpperCase 2 def len = (s: String) => s.size 3 def sanitize = "~".r replaceAllIn (_, "")
  34. 34. Function Composition 2/3 1 scala > def myfun0 = sanitize andThen toUpper 2 scala > def myfun1 = myfun0 andThen len 3 4 scala > myfun0 "Hello ~~~" 5 res0 = HELLO 6 7 scala > myfun1 "Hello ~~~" 8 res1: Int = 5
  35. 35. Function Composition 3/3 • Value-based • Functions
  36. 36. Questions so far? Figure: Awake?
  37. 37. Functions as building blocks Figure: Build solid walls/foundations from simple, regular building blocks
  38. 38. Reasoning
  39. 39. Equational Reasoning • Simpler testing; reduce test suite complexity • Substitute repeated expression with name • Refactoring is trivial; not error prone
  40. 40. Testing non-RT code Figure: Side effecting, non-RT code requires complex testing setup.
  41. 41. Equational Reasoning 1 scala > f(v1 , v2 , otherfun) == expected 2 res0: Boolean = true 3 • no setup/teardown complexity to test assertions • no mutation of SUT just for testing
  42. 42. Equational Reasoning 1 scala > val x = "YOLO" 2 x: java.lang.String = YOLO 3 4 scala > val r1 = x.reverse 5 r1: String = OLOY 6 7 scala > "YOLO".reverse 8 res0: String = OLOY 9
  43. 43. Equational Reasoning 1 scala > var age = 27 2 scala > def incr = { age += 1; age } 3 4 scala > incr 5 res0: Int = 28 6 7 scala > incr 8 res1: Int = 29 9
  44. 44. Axiomatic Reasoning • Axioms are basic assumptions • Allows us to specify properties • Infer potential problems due to properties • What can we guarantee? What can we not?
  45. 45. Axiomatic Reasoning 1 class Vpc < AwsResource 2 def initialize(conf) 3 @cidr = conf[’vpc.cidr ’] 4 end 5 end 6 # Is the Vpc instance created with an 7 # empty conf Hash guaranteeing the 8 # premise of the initialize contract? 9
  46. 46. Axiomatic Reasoning 1 // validate inputs before instantiating 2 case class Vpc(cidr: Cidr) 3 4 def createVpc(ipStr: String , pfx: String) = 5 for { 6 ip <- IpAddress.toIpAddress(ipStr) 7 prefix <- Cidr.toPrefix(pfx) 8 } yield Vpc(Cidr(ip , prefix )) 9
  47. 47. Axiomatic Reasoning 1 // if fun is 1-to -1 or bijective 2 // we know inverse must exist 3 def toLower(u: UpperCaseChar ): LowerCaseChar = ?? 4 def toUpper(l: LowerCaseChar ): UpperCaseChar = ?? 5 6 // idempotency typically important in ops 7 f(f(x)) == f(x) 8
  48. 48. Axiomatic Reasoning 1 // commutativity important in dist sys 2 // f, binary operator over type A 3 // x and y are of type A then 4 // x @ y = y @ x 5 f(x, y) == f(y, x) 6 7 // associativity important in concurrency 8 // f, binary operator over type A 9 // x, y, and z are of type A then 10 // x @ (y @ z) = (x @ y) @ z 11 f(x, f(y, z)) == f(f(x, y), z) 12
  49. 49. Axiomatic Reasoning - Distributed Systems • Simple causal consistency (partial ordering) • CRDTs (semilattice algebra) • Invariant confluence (coordination free execution)
  50. 50. Axiomatic Reasoning - Concurrency • MVars, TVars, LVars have algebraic properties • Design for liveness via axiomatic reasoning
  51. 51. Axiomatic Reasoning - More Exhaustive Testing • Property-based testing 1 // untested - uses scalacheck 2 def inverse[A]( implicit a: Action[A]) = 3 Prop.forAll { (x: A) => 4 x === a.inverse(a.inverse(x)) 5 } 6 7 implicit def actionArbitrary [A] 8 (implicit a: Arbitrary[A]) = Arbitrary { /* 9
  52. 52. Generic Reasoning • Reasoning on generic type functions • Less specific the types the more we know about the function • Find design-time bugs
  53. 53. Generic Reasoning 1 def f0[A](a: A): A = ??? 2
  54. 54. Generic Reasoning 1 // Only definition 2 def f0[A](a: A): A = a 3
  55. 55. Generic Reasoning 1 def f1[A](a: A)(ma: Option[A]): A = ??? 2
  56. 56. Generic Reasoning 1 def f1[A](a: A)(ma: Option[A]): A = a 2 1 def f1[A](a: A)(ma: Option[A]): A = ma match { 2 case Some(x) => x 3 case None => a 4 } 5
  57. 57. Generic Reasoning 1 def f2[A](as: List[A]): A = ??? 2
  58. 58. Generic Reasoning 1 // what about an empty list? Yikes! 2 def f2[A](as: List[A]): A = ??? 3
  59. 59. Generic Reasoning 1 // Assuming we want the head element! 2 // A more sensible type signature design 3 def f3[A](as: List[A]): Option[A] = as match { 4 case Nil => None 5 case a :: rest => a 6 } 7
  60. 60. Generic Reasoning 1 def f4[A](ma: Option[A]): A = ??? 2
  61. 61. Review
  62. 62. Reasoning Process • Assume as little as possible • Define all inputs • Build only on top of RT • Derive properties from domain • Infer properties from generic types
  63. 63. Methods / Techniques • Encode normal error cases in results • Strive for termination • Strive for RT where ever possible
  64. 64. Iteration / Improvement • So much to learn • Start out informal; add formal reasoning as needed/learned • The more we do the easier it gets
  65. 65. Example in the wild
  66. 66. Mainstream Package Management Based on shared + mutable state (filesystem)
  67. 67. Violates RT
  68. 68. Alternatives • shared + immutable • private + mutable • expensive coarse grained locks • hybrid without the expense
  69. 69. Define all inputs • Force clean build env (chroot) • Requires explicit inputs • Full dependency definition
  70. 70. Ensure RT • Use private mutable space • Different inputs, different result • Symlink unique results (atomic op)
  71. 71. Different inputs give unique results • Shared + immutable provides RT • New inputs result in different path • Install many versions/configurations • Means we can do binary substitution
  72. 72. RT foundation • Update existing node more reliably • Supports immutable infrastructure • Supports congruent configuration
  73. 73. Recap 1 Referential Transparency allows us to reason 3 HOFs & Composition is the motar 4 Logical reasoning provides solid foundation
  74. 74. Recap 1 Referential Transparency allows us to reason 2 Functions are simple, regular building blocks 3 HOFs & Composition is the motar 4 Logical reasoning provides solid foundation
  75. 75. Don’t fear the maths! Figure: Domain specific language of the sciences and engineering
  76. 76. Questions Figure: Heckle me @SusanPotter later too.

×