Our fall 2017 release presents operating thresholds, available from the BigML Dashboard, API, and WhizzML for automation. The ability to set an operating threshold is a key feature that allows you to fine tune the performance of your classification models, and lets you be more or less aggressive when predicting a class for an instance. Applying operating thresholds is especially useful in domains like fraud detection or medical diagnosis, where the consequences of some classifications may have prohibitive costs associated. Additionally, BigML is bringing organizations, a convenient collaborative space where several customers can work on the same projects, within the same Dashboard, at different levels of privileges. Organizations break down silos and make it easy and efficient for any company to adopt Machine Learning across their entire corporate structure.

1. Introducing
Operating Thresholds
and Organizations
BigML Fall 2017 Release

2. BigML, Inc 2BigML Release Webinar: Operating Thresholds and Organizations
BigML Release
CHARLES PARKER, PH.D. - VP of Machine
Learning Algorithms
Please enter questions into chat box – We will
answer some via chat and others at the end of the
session
https://bigml.com/releases/fall-2017
ATAKAN CETINSOY - VP of Predictive Applications
Resources
Moderator
Speaker
Contact support@bigml.com
Twitter @bigmlcom
Questions

3. BigML, Inc 3BigML Release Webinar: Operating Thresholds and Organizations
• You have a model! What will you do with it?
• Make Predictions
• About what? And what happens afterwards?
• You probably need an operating threshold
• Share it
• With whom?
• Use organizations
Actually using a model

4. BigML, Inc 4BigML Release Webinar: Operating Thresholds and Organizations
• The model has no idea what you want!
• It’s optimizing its own version of loss
• There are some ways to tell the model
to learn differently
• Class weights / Balance objective
• Weight column
• A lot of times, you can apply your
preferences after learning, but only if
you know what they are
• You can discover them using an
evaluation
The Dirty Little Secret of ML

5. BigML, Inc 5BigML Release Webinar: Operating Thresholds and Organizations
Evaluation Curves

6. BigML, Inc 6BigML Release Webinar: Operating Thresholds and Organizations
• The evaluation tool gives you the opportunity to see the effect of setting different
thresholds for a particular class on your data
• If the probability of this class is over the threshold, BigML predicts it
• Else BigML chooses the next-highest probability class
The Threshold Slider

7. BigML, Inc 7BigML Release Webinar: Operating Thresholds and Organizations
• Suppose the costs of false positives and
false negatives are different
• Medical Diagnosis:
• The positive class is rare, and so will not
be predicted often
• The positive class is very important!
• Use the evaluation tool to set the tradeoff
between the error types
Case 1: Medical Diagnosis

8. BigML, Inc 8BigML Release Webinar: Operating Thresholds and Organizations
• Sometimes you know precisely the cost of making a classification
• Predicting failure for a machine:
• Each positive prediction requires you to send a tech out into the field ($)
• Each false negative costs some amount of downtime ($$$), plus the
cost of the tech
• True negatives cost nothing
• Solution: Use WhizzML on the evaluation
• Coming soon to an interface near you!
Case 1-1: Failure Prediction

9. BigML, Inc 9BigML Release Webinar: Operating Thresholds and Organizations
• Predict: When a customer is going to leave a subscription service; take
some action to retain them if they are
• Sometimes you have a constraint on the number of times you can
apply a class
• In churn prediction, you want to call the people most likely to churn
• But you only have so much money to pay for it
• Set the threshold according to your budget; use the evaluation tool to
estimate performance
Case 2: Churn Prediction

10. BigML, Inc 10BigML Release Webinar: Operating Thresholds and Organizations
• Sometimes you have a fixed constraint, but
the input domain changes
• Suppose you have a lending operation
where your funding allows you to approve
half of the applications that come in
• What happens if I suddenly start getting
twice as many applications?
• Use the original evaluation to reset the threshold
(assuming the same application quality)
Case 2-2: Credit Default

11. BigML, Inc 11BigML Release Webinar: Operating Thresholds and Organizations
• Now that you have your threshold, what do you do?
• BigML makes predicting with a threshold BigML easy
• Thresholds available for predictions, batch predictions, through the BigML
API, and WhizzML for automation
Now What?

12. BigML, Inc 12BigML Release Webinar: Operating Thresholds and Organizations
• Some models have support for confidence, which is an instance-dependent
lower bound on the probability of the model
• If the model supports confidence, you can threshold on that as well
Using Confidence

13. BigML, Inc 13BigML Release Webinar: Operating Thresholds and Organizations
• Machine learning often involves a lot of
duplicated work
• Preprocessing data
• Updating models
• It can be ridiculous to have this duplicated in
multiple places in the same company /
working group
• BigML organizations is a fix for this problem
Organizations

14. BigML, Inc 14BigML Release Webinar: Operating Thresholds and Organizations
• Create an organization
• Organizations are separate from the
user’s individual account
• Other users can be invited to the
organization
Creating an Organization

15. BigML, Inc 15BigML Release Webinar: Operating Thresholds and Organizations
Managing Users
• Each user in the organization has a certain role, which controls their ability
to access and create resources

16. BigML, Inc 16BigML Release Webinar: Operating Thresholds and Organizations
• Each project can have access settings per user (read, create, etc.)
• The organization can have many projects
Assigning Permissions

17. BigML, Inc 17BigML Release Webinar: Operating Thresholds and Organizations
More Info
https://bigml.com/releases/fall-2017

18. Questions?
@bigmlcom support@bigml.com