An overview of data science in the social sector, and how to use your data science skills for good. Includes tips and tricks from winning data science competitions on www.drivendata.org.

1. Using Your Powers
for Good

2. PETER BULL
ISAAC SLAVITT
@drivendataorg
[peter | isaac]@drivendata.org
www.drivendata.org

5. When
it
comes
to
data,
nonprofits
don’t
know
what
they
don’t
know.

6. THE DATA CAPACITY GAP
• McKinsey predicts
a 140k – 180k
shortage of data
scientists
• Average salary of
data
scientist:$118,709
• Average salary of
the Executive
Director of a
nonprofit (budget
$500k – $5m):
• $133,000

8. L
I
T
E
R
A
C
Y
C
A
P
A
C
I
T
Y

9. “Finding ways to make big data useful to
humanitarian decision makers is one of
the great challenges and opportunities of
the network age.”
-­‐ UN Office for the Coordination of
Humanitarian Affairs

10. What is data
for good?

11. The Eric & Wendy Schmidt Data Science for Social Good Summer Fellowship 2014
Brew, Loewi, Majumdar, Reece, Rozier.
Buildings: 197,157
Time: 76 years
Money: $98 million
Buildings: 42,695
Time: 16.4 years
Money: $21.3 million
Buildings: 378
Time: 2 months
Money: $189,000
Prediction Saves Time & Money
No Prediction Current Model Model Forecast
Lead Paint Inspections

12. Abelson, Varshney.
http://www.datakind.org/blog/using-satellite-images-to-understand-poverty/
Quantifying Poverty

13. Protect the Environment

18. "The muffins are great...esp
the blueberry! I have never
had that good a blueberry
muffin...its not super sickey
sweet like most...."

19. "Food was soggy and cold by
the time I got it and they
messed up my order. Better
food and better service from
other places nearby."

21. Sentiment Analysis
𝑁"##$ − 𝑁&'$
𝑁(#)$*
brainy
rockstars
agility
unreal
fascinate
problem-solver
steadiness
greatness
works
commendably
gawk
cons
disorder
inane
martyrdom
irrecoverable
plasticky
madman
decrement

22. Topic Modeling (LDA)

23. Using the winning algorithm, Boston
could catch the same number of
health violations with 40 percent
fewer inspections, simply by better
targeting city resources at what
appear to be dirty-kitchen hotspots.
- Mike Luca, Harvard Business School

24. Education Resource Strategies
Budget
We are
budget-
ing our
things
$
Expenditures
Monies
for
making
students
good
Beware
there be
dollars
here

25. Lots of Labels!
PETRO-VEND FUEL AND FLUIDS
MAINT MATERIALS
SATELLITE COOK
UPPER EARLY INTERVENTION PROGRAM
Regional Playoff Hosts
Supp.- Materials
ITEMGH EXTENDED DAY
FURNITURE AND FIXTURES
NON-CAPITALIZED AV
Water and Sewage *
Instructional Materials
Food Services - Other Costs
Capital Assets - Locally Defined Groupings

27. All about the features!

28. Text features in scikit-­‐learn
from sklearn.feature_extraction.text import CountVectorizer
vec = CountVectorizer()
vectorized_data = vec.fit_transform(text_data)

29. Processing: Tokenize on Punctuation
PETRO-VEND FUEL AND FLUIDS
PETRO VEND FUEL AND FLUIDS
PETRO-
VEND
FUEL AND FLUIDS

30. Processing: Tokenize on Punctuation
from sklearn.feature_extraction.text import CountVectorizer
alpha_tokens = lambda text: re.split("[^a-z]", text.lower())
vec = CountVectorizer(tokenizer=alpha_tokens)
vectorized_data = vec.fit_transform(text_data)

31. Processing: 2-grams and 3-grams
PETRO VEND FUEL AND FLUIDS
1-grams
2-grams
3-grams

32. from sklearn.feature_extraction.text import CountVectorizer
alpha_tokens = lambda text: re.split("[^a-z]", text.lower())
vec = CountVectorizer(tokenizer=alpha_tokens,
ngram_range=(1, 3))
vectorized_data = vec.fit_transform(text_data)
Processing: 2-grams and 3-grams

33. Computational: Hashing Trick
PETRO VEND FUEL AND FLUIDS
2954 9384 4569 1197 8947

34. Computational: Hashing Trick
from sklearn.feature_extraction.text import HashingVectorizer
alpha_tokens = lambda text: re.split("[^a-z]", text.lower())
vec = HashingVectorizer(tokenizer=alpha_tokens,
ngram_range=(1, 3))
vectorized_data = vec.fit_transform(text_data)

35. Statistical: Pairwise Interaction Terms

36. from sklearn.feature_extraction.text import HashingVectorizer
from sklearn.preprocessing import PolynomialFeatures
alpha_tokens = lambda text: re.split("[^a-z]", text.lower())
vec = HashingVectorizer(tokenizer=alpha_tokens,
ngram_range=(1, 3))
vectorized_data = vec.fit_transform(text_data)
prep = PolynomialFeatures(degree=2,
interaction_only=True,
include_bias=False)
preprocessed_data = prep.fit_transform(vectorized_data)
Statistical: Pairwise Interaction Terms

37. Education Resource Strategies
Saving about
300 staff-hours
per year

38. How do I get involved in
data for good?

39. 1
By joining data for
good organizations

40. www.datakind.org

41. www.codeforboston.org

42. meetup.com/Data-Science-for-Social-Good

43. 2
By doing #data4good
in your spare time

44. blog.datalook.io/openimpact/

45. data.cityofboston.gov

46. www.drivendata.org

47. 3
By participating in a
fellowship program

48. dssg.io

49. dssg-atl.io

50. escience.washington.edu

51. www.codeforamerica.org

52. www.bayesimpact.org

53. 4
By attending data for
good events

54. www.kdd.org

55. www.bloomberg.com/company/d4gx

56. dogooddata.com

57. http://socialgoodtech.org/

58. 5
By getting involved
professionally

59. Look for local non-profits!

60. Get your company involved!

61. Questions?
@drivendataorg