1. Generating Example Data For Dataflow Programs Chris Olston Shubham Chopra Utkarsh Srivastava Research

3. Lots of queries and programs to analyze that data

4. New data flow languages

5. Map-Reduce, Pig Latin, Dryad

6. Other data flow systems

8. Iterative Process LOAD (user, url) LOAD (url, pagerank) Joining on right attribute? JOIN on url GROUP on user TRANSFORM user, canonicalize(url) TRANSFORM user, AVG(pagerank) Bug in UDF canonicalize? Everything being filtered out? FILTER avgPR> 0.5 No Output 

9. How to do test runs? Run with real data Too inefficient (TBs of data) Create smaller data sets (e.g., by sampling) Empty results due to joins [Chaudhuri et. al. 99], and selective filters Biased sampling for joins Indexes not always present

10. Examples to Illustrate Program (www.cnn.com, 0.9) (www.frogs.com, 0.3) (www.snails.com, 0.4) LOAD (user, url) LOAD (url, pagerank) (Amy, cnn.com) (Amy, http://www.frogs.com) (Fred, www.snails.com/index.html) JOIN on url (Amy, www.cnn.com, 0.9) (Amy, www.frogs.com, 0.3) (Fred, www.snails.com, 0.4) GROUP on user TRANSFORM user, canonicalize(url) (Amy, www.cnn.com, 0.9) (Amy, www.frogs.com, 0.3) (Fred, www.snails.com, 0.4) ) ( Amy, ( Fred, ) TRANSFORM user, AVG(pagerank) (Amy, www.cnn.com) (Amy, www.frogs.com) (Fred, www.snails.com) (Amy, 0.6) (Fred, 0.4) FILTER avgPR> 0.5 (Amy, 0.6)

11. Value Addition From Examples Examples can be used for Debugging Understanding a program written by someone else Learning a new operator, or language

12. Outline Formalization of good examples Example Generation Algorithm Performance Evaluation

13. Good Examples: Consistency LOAD (user, url) LOAD (url, pagerank) (Amy, cnn.com) (Amy, http://www.frogs.com) (Fred, www.snails.com/index.html) JOIN on url GROUP on user 0. Consistency TRANSFORM user, canonicalize(url) TRANSFORM user, AVG(pagerank) output example = operator applied on input example (Amy, www.cnn.com) (Amy, www.frogs.com) (Fred, www.snails.com) FILTER avgPR> 0.5

14. Good Examples: Realism LOAD (user, url) LOAD (url, pagerank) (Amy, cnn.com) (Amy, http://www.frogs.com) (Fred, www.snails.com/index.html) JOIN on url GROUP on user 1. Realism TRANSFORM user, canonicalize(url) TRANSFORM user, AVG(pagerank) (Amy, www.cnn.com) (Amy, www.frogs.com) (Fred, www.snails.com) Formalization: Fraction of examples that are real or are derived from real records FILTER avgPR> 0.5

15. Good Examples: Completeness LOAD (user, url) LOAD (url, pagerank) 2. Completeness JOIN on url Demonstrate the salient properties of each operator, e.g., FILTER GROUP on user TRANSFORM user, canonicalize(url) TRANSFORM user, AVG(pagerank) (Amy, 0.6) (Fred, 0.4) FILTER avgPR> 0.5 (Amy, 0.6)

16. Good Examples: Completeness (www.cnn.com, 0.9) (www.frogs.com, 0.3) (www.snails.com, 0.4) LOAD (user, url) LOAD (url, pagerank) JOIN on url (Amy, www.cnn.com, 0.9) (Amy, www.frogs.com, 0.3) (Fred, www.snails.com, 0.4) GROUP on user TRANSFORM user, canonicalize(url) 2. Completeness TRANSFORM user, AVG(pagerank) (Amy, www.cnn.com) (Amy, www.frogs.com) (Fred, www.snails.com) Demonstrate the salient properties of each operator, e.g., JOIN FILTER avgPR> 0.5

18. Each equivalence class demonstrates one property of the operator.

20. Formalizing Completeness Operator Completeness: Fraction of equivalence classes that have at least one example record. Overall Completeness: Average of per-operator completeness.

21. Good Examples: Conciseness LOAD (user, url) LOAD (url, pagerank) 3. Conciseness (Amy, cnn.com) (Amy, http://www.frogs.com) (Fred, www.snails.com/index.html) JOIN on url Operator Conciseness: # equivalence classes # example records GROUP on user TRANSFORM user, canonicalize(url) Overall Conciseness: Average of per-operator conciseness TRANSFORM user, AVG(pagerank) (Amy, www.cnn.com) (Amy, www.frogs.com) (Fred, www.snails.com) FILTER avgPR> 0.5

22. Outline Formalization of good examples Example Generation Algorithm Performance Evaluation

23. Related Work Related Areas: Reverse Query Processing Database Testing Software and Hardware Verification Differences Realism not a concern Notion of conciseness is different Intermediate result size is immaterial

24. Strawman I: Downstream Propagation Take some portion of input data and run the program over it. 1. Realism 2. Completeness 3. Conciseness

25. Strawman II: Upstream Propagation Start from what output is desired, and work backwards 1. Realism 2. Completeness 3. Conciseness

26. Our Algorithm Algorithm Passes Downstream Pruning Upstream Pruning

27. Our Algorithm Algorithm Passes Downstream Pruning Upstream Pruning Take a subset of input and propagate through the program. (Jack, 30) LOAD (user, age) UNION FILTER age>18 LOAD (user, age) FILTER udf(user) (Amy, 20) (Fred, 25) (Jack, 30) (Amy, 20) (Fred, 25) (Jack, 30) (Amy, 20) (Fred, 25) (Amy, 20) (Fred, 25)

28. Our Algorithm Algorithm Passes Downstream Pruning Upstream Pruning Prune redundant examples, i.e., improve conciseness without hurting completeness. (Jack, 30) LOAD (user, age) UNION FILTER age>18 LOAD (user, age) FILTER udf(user) (Amy, 20) (Fred, 25) (Jack, 30) (Amy, 20) (Fred, 25) (Jack, 30) (Amy, 20) (Fred, 25) (Amy, 20) (Fred, 25)

29. Our Algorithm Algorithm Passes Downstream Pruning Upstream Pruning Prune redundant examples, i.e., improve conciseness without hurting completeness. (Jack, 30) LOAD (user, age) UNION FILTER age>18 LOAD (user, age) FILTER udf(user) (Amy, 20) (Fred, 25) (Jack, 30) (Amy, 20) (Fred, 25) (Jack, 30) (Amy, 20) (Fred, 25) (Amy, 20) (Fred, 25)

30. Our Algorithm Algorithm Passes Downstream Pruning Upstream Pruning Prune redundant examples, i.e., improve conciseness without hurting completeness. (Jack, 30) LOAD (user, age) UNION FILTER age>18 LOAD (user, age) FILTER udf(user) (Amy, 20) (Jack, 30) (Amy, 20) (Jack, 30) (Amy, 20) (Amy, 20)

31. Formalization of Pruning Example Records Elements Equivalence Classes Sets Pick minimum #records to cover every equivalence class Set-Cover Problem More involved because completeness of other operators must be maintained; details in paper

32. Our Algorithm Algorithm Passes Downstream Pruning Upstream Pruning Enhance completeness by inserting constraint records (best effort; details in paper) (Jack, 30) LOAD (user, age) UNION FILTER age>18 LOAD (user, age) FILTER udf(user) (Amy, 20) (Jack, 30) (Amy, 20) (Jack, 30) (Amy, 20) (Amy, 20)

33. Our Algorithm Algorithm Passes Downstream Pruning Upstream Pruning Enhance completeness by inserting constraint records (best effort; details in paper) (Jack, 30) LOAD (user, age) UNION FILTER age>18 LOAD (user, age) FILTER udf(user) (Amy, 20) (Jack, 30) (Amy, 20) (Jack, 30) (--, 17) (Amy, 20) (Amy, 20)

34. Our Algorithm Algorithm Passes Downstream Pruning Upstream Pruning Enhance completeness by inserting constraint records (best effort; details in paper) (Jack, 30) (--, 17) LOAD (user, age) UNION FILTER age>18 LOAD (user, age) FILTER udf(user) (Amy, 20) (Jack, 30) (Amy, 20) (Jack, 30) (--, 17) (Amy, 20) (--, 17) (Amy, 20) (Bill, 17)

35. Our Algorithm Algorithm Passes Downstream Pruning Upstream Pruning Enhance completeness by inserting constraint records (best effort; details in paper) (Jack, 30) (Bob, 17) LOAD (user, age) UNION FILTER age>18 LOAD (user, age) FILTER udf(user) (Amy, 20) (Jack, 30) (Amy, 20) (Jack, 30) (Bill, 17) (Bob, 17) (Amy, 20) (Bill, 17) (Amy, 20) (Bill, 17)

36. Our Algorithm Algorithm Passes Downstream Pruning Upstream Pruning Prune redundant examples (as in Pass 2). Favor real examples over synthetic ones. (Jack, 30) (Bob, 17) LOAD (user, age) UNION FILTER age>18 LOAD (user, age) FILTER udf(user) (Amy, 20) (Jack, 30) (Amy, 20) (Jack, 30) (Bill, 17) (Bob, 17) (Amy, 20) (Bill, 17) (Amy, 20) (Bill, 17)

37. Our Algorithm Algorithm Passes Downstream Pruning Upstream Pruning Prune redundant examples (as in Pass 2). Favor real examples over synthetic ones. (Jack, 30) (Bob, 17) LOAD (user, age) UNION FILTER age>18 LOAD (user, age) FILTER udf(user) (Amy, 20) (Jack, 30) (Amy, 20) (Jack, 30) (Bill, 17) (Bob, 17) (Amy, 20) (Bill, 17) (Amy, 20) (Bill, 17)

38. Our Algorithm Algorithm Passes Downstream Pruning Upstream Pruning Prune redundant examples (as in Pass 2). Favor real examples over synthetic ones. (Jack, 30) LOAD (user, age) UNION FILTER age>18 LOAD (user, age) FILTER udf(user) (Jack, 30) (Jack, 30) (Bill, 17) (Bill, 17) (Bill, 17)

39. Implementation Status Available as ILLUSTRATE command in open-source release of Pig Available as Eclipse Plugin (PigPen)

40. PigPen Snapshot

41. Performance Evaluation Program I: (Web Search Result Viewing Statistics) LOAD FILTER by compound arithmetic expression GROUP TRANSFORM using built-in aggregate function

42. Performance on Program I

43. Performance Evaluation Program II: (Web Advertising Activity) LOAD table A FILTER A by compound logical expression JOIN with table B (highly selective) TRANSFORM using 4 string manipulation UDFS (non-invertible)

44. Performance on Program II

45. Running Time

47. Actual dataset too large for test runs.

48. Our algorithm can automatically generate examples that illustrate the program through:

49. Realism

50. Conciseness

51. Completeness