Penny is a simple tool to help us understand what wealth and poverty look like to artificial intelligence algorithms. The tool lets you change the landscape of a city, by adding and removing urban features like buildings, parks and freeways to high-resolution satellite imagery. With this interface, you can explore what different kinds of features make a place look wealthy or poor to an AI.

1. What does money
look like from space?
(here comes the neighborhood)

2. The Training Model
Satellite Imagery From GBDX

3. The Training Model
Census Tracts (2013)

4. Quartiles
The Training Model
Quartile 0
< $34,176
Quartile 1
$34,177 - $49,904
Quartile 2
$49,905 - $71,875
Quartile 3
$71,876+

5. Satellite Imagery & Census Data
The Training Model

6. Centroids
The Training Model

7. Centroid Outlines
The Training Model

8. Satellite Imagery & Census Data
The Training Model (Resnet 50)

9. Satellite Imagery & Census Data Neural Network
The Training Model (Resnet 50)

10. Satellite Imagery & Census Data Output
=
Neural Network
The Training Model (Resnet 50)
Q0 = 91%
Q1 = 5.64%
Q2 = 2.55%
Q3 = .41%

11. Satellite Imagery & Census Data Output
=
Neural Network
The Training Model (Resnet 50)
The model carries
what it has learned and
repeats the process.
Q0 = 91%
Q1 = 5.64%
Q2 = 2.55%
Q3 = .41%

12. The Training Model (Resnet 50)
Input Normalization Normalization Normalization MergeActivationActivationConvolution Convolution Convolution
Neural Network
The data is fed into the model.

13. Input Normalization Normalization Normalization MergeActivationActivationConvolution Convolution Convolution
The Training Model (Resnet 50)
Neural Network
A number of filters are applied to the image.

14. Input Normalization Normalization Normalization MergeActivationActivationConvolution Convolution Convolution
The Training Model (Resnet 50)
Neural Network
The resulting new values are normalized to be within
learned mean and standard deviations of the dataset.

15. Input Normalization Normalization Normalization MergeActivationActivationConvolution Convolution Convolution
The Training Model (Resnet 50)
Neural Network
Separates out features from important and non important ones.

16. Input Normalization Normalization Normalization MergeActivationActivationConvolution Convolution Convolution
The Training Model (Resnet 50)
Neural Network
Repeat.

17. Input Normalization Normalization Normalization MergeActivationActivationConvolution Convolution Convolution
The Training Model (Resnet 50)
Neural Network
Results are merged with previous iteration.

18. The Training Model (Resnet 50)
TensorBoard data from CMU

19. So now you have a Classifier

20. Centroid Outlines

21. Classified Approximate

22. Satellite Imagery & Census Data

23. What is the classifier seeing?

24. NYC Areas Classified as Q0

25. NYC Areas Classified as Q1 & Q2

26. NYC Areas Classified as Q3

27. Classification
Confidence
97% Q0
3% Q1
0% Q2
0% Q3

28. Classification
Confidence
92% Q0
7% Q1
0% Q2
0% Q3

29. Classification
Confidence
91% Q0
5% Q1
2% Q2
.4% Q3

30. Classification
Confidence
2% Q0
2% Q1
6% Q2
90% Q3

31. Classification
Confidence
0% Q0
.1% Q1
0% Q2
99% Q3

32. “Despite this encouraging process, there is still little
insight into the internal operation and behavior of
these complex models, or how they achieve such
good performance. From a scientific standpoint, this is
deeply unsatisfactory. Without clear understanding of
how and why they work, the development of better
models is reduced to trial-and-error.”
Visualizing and Understanding Convolutional Networks - Matthew D. Zeiler, Dept. of Computer Science, Courant
Institute, New York University - Rob Fergus, Dept. of Computer Science, Courant Institute, New York University

33. Baseball Field Experiment
Confidence
91% Q0
5% Q1
2% Q2
.4% Q3
Original Image

34. Baseball Field Experiment
Confidence
79% Q0 (-12)
12% Q1 (+7)
7% Q2 (+5)
.8% Q3 (+.4)
Added Trees

35. Baseball Field Experiment
Confidence
63% Q0 (-28)
19% Q1 (+14)
13% Q2 (+11)
3% Q3 (+2.6)
Added More Trees

36. Baseball Field Experiment
Confidence
68% Q0 (-23)
17% Q1 (+12)
11% Q2 (+9)
2% Q3 (+1.6)
And Added More Trees

37. Baseball Field Experiment
Confidence
44% Q0 (-47)
12% Q1 (+7)
23% Q2 (+21)
18% Q3 (+17.6)
Added All The Trees

38. Tree Experiment
Confidence
79% Q0
10% Q1
5% Q2
5% Q3
Original Image

39. Tree Experiment
Confidence
79% Q0
10% Q1
5% Q2
5% Q3

40. Tree Experiment
Confidence
77% Q0 (-2)
10% Q1 (0)
6% Q2 (+1)
5% Q3 (0)

41. Tree Experiment 2
Confidence
89% Q0
10% Q1
0% Q2
0% Q3
Original Image

42. Tree Experiment 2
Confidence
89% Q0
10% Q1
0% Q2
0% Q3

43. Tree Experiment 2
Confidence
87% Q0 (-2)
11% Q1 (+1)
0% Q2 (0)
0% Q3 (0)

44. Trump Tower Experiment
Confidence
0% Q0
.1% Q1
0% Q2
99% Q3
Original Image

45. Trump Tower Experiment
Confidence
16% Q0 (+16)
25% Q1 (+24.9)
11% Q2 (+11)
45% Q3 (-54)
Added Grass

46. Trump Tower Experiment
Confidence
34% Q0 (+34)
47% Q1 (+46.9)
6% Q2 (+6)
10% Q3 (-89)
Built a Wall

47. Conclusions:
• It's possible to predict income levels
from space
• Underlying data can provide valuable
assistance to complex neural networks
• Human-based empirical inquiry has legs
• Teasing out why it knows what it knows
is interesting

48. Questions:
• What does this thing do when you point
it at other cities?
• What are the similarities and differences
between cities from space?
• Can we construct a model to account for
seasonal variance?
• Can we construct a model to account for
architectural difference?

49. cmu.edu
Who
stamen.com amantiwari.comgbdx.geobigdata.io
Carnegie MellonStamen Design Aman TiwariDigital Globe

50. Thanks
(@stamen rules)