The document discusses automating machine learning workflows and feature engineering. It proposes domain-specific languages like Flatline and WhizzML to provide abstraction and automation. Flatline allows declarative feature engineering expressions. WhizzML provides abstraction, reusable procedures, and handles workflows and algorithms more efficiently by executing remotely on servers. Challenges remain around error management, resumable workflows, and optimizing data access across distributed executions.

1. Automating Machine Learning
Features and Workflows
jao@bigml.com
PAPIs Connect Valencia, 2016

2. Outline
Introduction: ML as a System Service
Feature Engineering Automation
Workflow Automation
Challenges and Outlook

3. Outline
Introduction: ML as a System Service
Feature Engineering Automation
Workflow Automation
Challenges and Outlook

4. Machine Learning as a System Service
The goal
Machine Learning as a system
level service
The means
APIs: ML building blocks
Abstraction layer over
feature engineering
Abstraction layer over
algorithms
Automation

5. Machine Learning Workflows
Dr. Natalia Konstantinova (http://nkonst.com/machine-learning-explained-simple-words/)

6. Machine Learning Workflows for real
Jeannine Takaki, Microsoft Azure Team

7. Machine Learning Automation Today
from bigml.api import BigML
api = BigML()
project = api.create_project({’name’: ’ToyBoost’})
orig_source =
api.create_source(source,
{"name": "ToyBoost",
"project": project[’resource’]})
api.ok(orig_source)
orig_dataset =
api.create_dataset(orig_source, {"name": "Boost"})
api.ok(orig_dataset)
trainset = api.get_dataset(trainset)
for loop in range(0,10):
api.ok(trainset)
model = api.create_model(trainset, {
"name": "ToyBoost - Model%d" % loop,
"objective_fields": ["letter"],
"excluded_fields": ["weight"],
"weight_field": "100011"})
api.ok(model)
batchp =
api.create_batch_prediction(model, trainset, {
"name": "ToyBoost - Result%d" % loop,
"all_fields": True,
"header": True})
api.ok(batchp)
batchp = api.get_batch_prediction(batchp)
batchp_dataset =
api.get_dataset(batchp[’object’])

8. Machine Learning Automation Today
Problems of current solutions
Complexity Lots of details outside the problem domain
Reuse No inter-language compatibility
Scalability Client-side workflows hard to optimize

9. Machine Learning Automation Today
Problems of current solutions
Complexity Lots of details outside the problem domain
Reuse No inter-language compatibility
Scalability Client-side workflows hard to optimize
Not enough abstraction

10. Machine Learning Automation Tomorrow
Solution: Domain-specific languages

11. Outline
Introduction: ML as a System Service
Feature Engineering Automation
Workflow Automation
Challenges and Outlook

12. Domain-specific Expressions (sexps)
(if (missing? "height")
(random-value "height")
(field "height"))
(window "income" 10)
(within-percentiles? "age" 0.5 0.95)
(cond (> (field "score") (mean "score")) "above average"
(= (field "score") (mean "score")) "below average"
"mediocre")

13. Domain-specific Expressions (JSON)
["if", ["missing?", "height"],
["random-value", "height"],
["field", "height"]]
["window", "income", 10]
["within-percentiles?", "age", 0.5, 0.95]
["cond", [">", ["field", "score"], ["mean", "score"]], "above av
["=", ["field", "score"], ["mean", "score"]], "below av
"mediocre"]

14. Domain-specific Expressions (sexps)
(if (missing? "height")
(random-value "height")
(field "height"))
(window "income" 10)
(within-percentiles? "age" 0.5 0.95)
(cond (> (field "score") (mean "score")) "above average"
(= (field "score") (mean "score")) "below average"
"mediocre")

15. Abstraction via the Language
;; (if (missing? "height")
;; (random-value "height")
;; (field "height"))
(ensure-value "height")
(window "income" 10)
(within-percentiles? "age" 0.5 0.95)
;; (cond (> (field "score") (mean "score")) "above average"
;; (= (field "score") (mean "score")) "below average"
;; "mediocre")
(discretize "score" "above above" "below average" "mediocre")

16. Abstraction via the User Interface

17. Remote for efficiency and reuse, local for discoverability

18. Flatline: A DSL for Feature Enginering
Domain-specific: new fields from an input sliding window as
declarative expressions
Simple syntax: JSON → s-expressions
Efficient: full server-side implementation
Discoverable: in-browser client-side implementation
Reusable: the same expressions usable from any language
binding.
Bonus: applicable to filtering

19. Outline
Introduction: ML as a System Service
Feature Engineering Automation
Workflow Automation
Challenges and Outlook

20. Machine Learning Workflows
A DSL for Machine Learning
Workflows?

21. Machine Learning Workflows
A DSL for Machine Learning
Workflows? Absolutely!

22. Machine Learning Workflows
Same problems, only worse. . .
Complexity Hairy logic and control-flow
Reuse More complex algorithms and behaviour very hard to
port to other languages
Scalability Lots of iterations and intermediate resources very
hard to make efficient on the client side

23. Machine Learning Workflows
WhizzML, same solution, only better. . .

24. WhizzML: A sexp-based, domain-specific language
(define apple
"https://s3.amazonaws.com/bigml-public/csv/nasdaq_aapl.csv")
(define source (create-and-wait-source {"remote" apple
"name" "whizz"}))
(define dataset (create-and-wait-dataset {"source" source}))
(define anomaly (create-and-wait-anomaly {"dataset" dataset}))
(define input {"Open" 275 "High" 300 "Low" 250})
(define score
(create-and-wait-anomalyscore {"anomaly" anomaly
"input_data" input}))
(get (fetch score) "score")

25. WhizzML vs Flatline (as languages)
A better language:
Better data structures (dictionaries, sets. . . )
Better control-flow: (tail) recursion, iteration, loops
Better abstraction: procedures

26. WhizzML: Lambda Abstraction
Abstraction
(define (score-stock name input)
(let (base "https://s3.amazonaws.com/bigml-public/csv"
stock (str base "/" name)
source (create-and-wait-source {"remote" stock})
dataset (create-and-wait-dataset {"source" source})
anomaly (create-and-wait-anomaly {"dataset" dataset}))
(create-and-wait-anomalyscore {"anomaly" anomaly
"input_data" input})))

27. WhizzML: Reusable Procedures
Abstraction
(score-stock "aapl" {"Open" 275 "High" 300 "Low" 250})

28. WhizzML: Server-side fortes
A better server-side:
Better reusability: scripts, executions and libraries as
first-class ML resources
Higher efficiency gains: automatic parallelism
More opportunities for UI extensions

29. WhizzML Source Code as a Machine Learning Resource
{"library":{
"imports":["12343addb343f2890f23492d"],
"source_code": "(define (mu2) (mu (g 3 8)))",
"exports": [{"name": "mu2", "signature": []}]}}
{"script":{
"parameters": [{"name": "remote_uri", "type": "string"},
{"name": "timeout", "type": "number",
"default": 10000}],
"source_code":
"(define id (create-source {"remote" remote_uri}))
(wait id timeout)",
"outputs": [{"name": "id", "type": "source-id"}]}}
Rich metadata, reuse and shareability of WhizzML code

30. Executions as a Machine Learning Resource
{"execution": {"script_id": "1a2232bf3498f95dde",
"username": "bittwidler",
"tlp": 4,
"resource_limits": {"total": 50,
"source": 10,
"dataset": 5,
"model": 10},
"max_exection_time": 3600,
"max_execution_steps": 10000,
"max_recursion_depth": 1024}}

31. Executions as a Machine Learning Resource
{"execution": {"script_id": "1a2232bf3498f95dde",
"username": "bittwidler",
"tlp": 4,
"resource_limits": {"total": 50,
"source": 10,
"dataset": 5,
"model": 10},
"max_exection_time": 3600,
"max_execution_steps": 10000,
"max_recursion_depth": 1024}}

32. WhizzML: Client-side fortes
A better client-side:
Better interactive experience: read-eval-print loop
Scripts usable from the user’s machine
Interoperability: Java, JavaScript and NodeJS REPLs
Challenge: behaviourial coherence between server and client
sides

33. Outline
Introduction: ML as a System Service
Feature Engineering Automation
Workflow Automation
Challenges and Outlook

34. Challenges
Solved
Local REPL and remote shared implementation
Automatic parallelization
Error reporting
Traceability: stack traces and stepwise execution
Open
Better error management (dynamic typing, type inferencer)
Resumable workflows
Data locality: optimizing repeated access to the same datasets