How we use AI as a tool to increase human productivity by a factor of 10? By augmenting our fluid intelligence.

1. AI Tools for Creative Work
Lydia B. Chilton
DUB Seminar
2 October 2019

3. AI can do things it couldn’t do before…
But it still struggles at ill-defined tasks.

4. It looks like you’re trying to
solve world peace.
Do you need assistance?
AI can do things it couldn’t do before…
But it still struggles at ill-defined tasks.

5. Long history of creativity support tools in HCI
Jonas Frich, Lindsay MacDonald Vermeulen, Christian Remy, Michael Mose Biskjaer, and Peter Dalsgaard. 2019.
Mapping the Landscape of Creativity Support Tools in HCI. CHI 2019

6. How does AI change the future of creative work?
Gatys, Ecker, Bethge. A Neural Algorithm of Artistic Style. http://arxiv.org/abs/1508.06576 2015

7. Outline: AI Tools for Creative Work
• Introduce A Creative Problem: Visual Metaphors
• Can AI do it?
• VisiBlends: A Workflow for Prototyping Visual Metaphors
• Symbol Finder: Adding AI to the workflow
• VisiFit: AI Tools for Iteratively Improving Visual Blends
• The Future of AI and creative work

8. AdvertisementsNews Public Service Announcements
Earth + MeltBrazil + Takes Off Tabasco + Hot
Visual Metaphors: image that convey a message symbolically

9. How can we create visual blends for a message?

10. Outline: AI Tools for Creative Work
• Introduce A Creative Problem: Visual Metaphors
• Can AI do it?
• VisiBlends: A Workflow for Prototyping Visual Metaphors
• Symbol Finder: Adding AI to the workflow
• VisiFit: AI Tools for Iteratively Improving Visual Blends
• The Future of AI and creative work

11. With successes in deep learning,
what is the future of creative work?

12. Visual Metaphor:
Lego are Summer Vacation Fun
Artist blend:

13. Deep Style Transfer
Artist:AI:

14. Visual Metaphor:
Burgers are Healthy, like apples
Artist:AI:

15. Visual Metaphor:
Burgers are Healthy, like apples
Artist:AI:
Blending objects requires more than just blending “style.”
You need more context about the objects - how to take them apart, blend the
pieces, think about how they’ll be perceived.

16. Outline: AI Tools for Creative Work
• Introduce A Creative Problem: Visual Metaphors
• Can AI do it?
• VisiBlends: A Workflow for Prototyping Visual Metaphors
• Symbol Finder: Adding AI to the workflow
• VisiFit: AI Tools for Iteratively Improving Visual Blends
• The Future of AI and creative work

17. But there is no simple pattern for visual blends
We need to break up the problem into tasks.

18. 1. Two objects are integrated into one object
2. Both objects are individually identifiable
Visual Blending Structure:
Whole-to-Part Shape Matching
Abstract problem structure:
Single Shape Mapping

19. Why blend based on shape?
Shape is the primary feature the human visual system uses to recognize objects.
Colors, silhouette and details and secondary.

20. 1. Two objects are integrated into one object
2. Both objects are individually identifiable
Visual Blending Structure:
Whole-to-Part Shape Matchingby using a design pattern to make it a search problem
We can solve a creative design problem
Earth Melt
Visual Blending Structure:
Whole-to-Part Shape Matching

21. Starbucks + Summer

22. Prototype
blend
Evaluate
prototype
Iterate
Output:
A visual blend
Two objects are integrated into one object
Both objects are identifiable
Prototype
blend
Evaluate
prototype
Two objects are integrated into one object
Both objects are identifiable

23. Prototype
blend
Evaluate
prototype
Iterate
Output:
A visual blend
Two objects are integrated into one object
Both objects are identifiable
Prototype
blend
Evaluate
prototype
Two objects are integrated into one object
Both objects are identifiable

24. Prototype
blend
Evaluate
prototype
Iterate
Output:
A visual blend
Two objects are integrated into one object
Both objects are identifiable

25. Prototype
blend
Evaluate
prototype
Iterate
Output:
A visual blend
Two objects are integrated into one object
Both objects are identifiable
e
e
Output:
A visual blend
grated into one object
ntifiable

26. A) Brainstorm
for summer
B) Find and annotate images
for summer
C) Matching algorithm
Starbucks + summer
D) Automatic blends + human evaluation
Starbucks + summer
matches = []
for a in summer_symbols:
for b in starbucks_symbols:
a_ratio = a.height / a.width
b_ratio = b.height / b.width
ratios = sort(a_ratio, b_ratio)
if (a.shape == b.shape) and
(a.coverage != coverage) and
(ratios[0] >= 0.5*ratios[1]):
matches.push([a, b])
return matches
VisiBlends:
A web interface to collaboratively make blends

27. Blends made by groups at
Columbia University

28. Football
+
dangerous

29. Joe’s Coffee
+
Night

30. Wash your hands.
It’s the smart move.

31. Forget the knives and forks: the humble hamburger is in
the midst of a renaissance. From Melbourne to Milan,
restaurateurs have helped elevate this classic piece of
American folklore into an iconic dish. Every day, more
and more enthusiasts join the search for the perfect

33. Philosophy
+
Christmas

34. Does VisiBlends make people better?
• 13 undergraduates (novice designers)
• Each made 6 blends (3 with VisiBlends, 3 without)
• 5 minutes for each blend
• Metric: How many successful blends can 5 minutes?

35. Novices are 10x better with VisiBlends
Control-first
Avg. Number of
blends made
VisiBlends-first
Without VisiBlends With VisiBlends
Without VisiBlendsWith VisiBlends
Avg. Number of
blends made
0.56 5.56 (t(18)=4.88, p<0.001)
0.675.67 (t(21)=5.84, p<0.001)

36. Bicycle + Smart
VisiBlendsUsers Unassisted
VisiBlends helps them have
many ideas, and pick the best.
Novices get stuck “improving”
their first idea.

37. New York City + Night
Users Unassisted
Novices pick “perfect” symbols,
but sacrifice shape-fit.
VisiBlends helps novices meet all
the criteria for a visual blend
(especially shape-fit).
VisiBlends

38. Do we want “perfect” symbols for Visual Blends?
By meeting more constraints,
Visual Metaphors are more
novel and interesting

39. Novices are 10x better with VisiBlends
10x improvement in novices ability
VisiBlendsUnassisted
Why?

40. Football + Dangerous

41. Football Dangerous
Brainstorm
associations
Find Images
of objects
Annotate
shapes
Annotate
shape coverage
Are both objects identifiable?
Are two objects integrated into one object?
Evaluate
Blend
Shape covers
All of object
Shape covers
Part of object

42. Football Dangerous
Brainstorm
associations
Find Images
of objects
Annotate
shapes
Annotate
shape coverage
Are both objects identifiable?
Are two objects integrated into one object?
Evaluate
Blend
Shape covers
All of object
Shape covers
Part of object

43. Collect pieces
Combine them
Re-combine them
Collect pieces
Lots of individual pieces under your finger;
it’s easy to recombine them.

44. Fluid
Intelligence
Crystallized
Intelligence
Having the resources and tools you need
(memory, routing procedures, etc)
Having the tools at hand so you can
fluidly combine them

45. Crystallized Intelligence
Crystallized Intelligence
Fluid Intelligence
Fluid Intelligence
VisiBlends augments fluid intelligence
Collect pieces
Combine them
Re-combine them
Collect pieces

46. NYC + Healthy

47. NYC Healthy
Brainstorm
associations
Find Images
of objects
Annotate
shapes
Annotate
shape coverage
Evaluate
Blend
No shape matches

48. NYC Healthy
Brainstorm
associations
Find Images
of objects
Annotate
shapes
Annotate
shape coverage
Are both objects identifiable?
Are two objects integrated into one object?
Evaluate
Blend

49. Orange + Healthy

50. Orange Healthy
Brainstorm
associations
Find Images
of objects
Annotate
shapes
Annotate
shape coverage
Shape covers
Part of object
Shape covers
All of object
Are both objects identifiable?
Are two objects integrated into one object?
Evaluate
Blend

51. Orange Healthy
Brainstorm
associations
Find Images
of objects
Annotate
shapes
Annotate
shape coverage
Shape covers
All of object
Shape covers
Part of object
Evaluate
Blend
Exercise equipment

52. Orange Healthy
Brainstorm
associations
Find Images
of objects
Annotate
shapes
Annotate
shape coverage
Shape covers
All of object
Shape covers
Part of object
Are both objects identifiable?
Are two objects integrated into one object?
Evaluate
Blend
Exercise equipment

53. Iteration requires fluidity –
putting together pieces in new ways.
No matches
Improve object fit
Objects are
not identifiable + =
+ =
Within same search space,
meet other constraints.
Find versions of an object
with different
color, style, aspect ratio
Focus on meeting a
specific constraint:
Find symbols with
a different shape
Search in a new
subspace
Find symbols with
a different shape

54. Crystallized Intelligence
Crystallized Intelligence
Fluid Intelligence
Fluid Intelligence
VisiBlends augments fluid intelligence
Collect pieces
Combine them
Re-combine them
Collect pieces

55. Outline: AI Tools for Creative Work
• Introduce A Creative Problem: Visual Metaphors
• Can AI do it?
• VisiBlends: A Workflow for Prototyping Visual Metaphors
• Symbol Finder: AI tools for finding symbols
• VisiFit: AI Tools for Iteratively Improving Visual Blends
• The Future of AI and creative work

56. Prototype
blend
Evaluate
prototype
Iterate
Output:
A visual blend
Two objects are integrated into one object
Both objects are identifiable
pe
e
pe
Output:
A visual blend
egrated into one object
entifiable

57. Rules for finding images are complex

58. Simply searching for abstract words doesn’t return
simple, iconic symbols that we can blend.

59. SymbolFinder uses AI + KBs
to find concrete images for an abstract term
Savvas Petridis
PhD Student
word2vec, synonyms, ConceptNet, concreteness classifiers

60. Prototype
blend
Evaluate
prototype
Iterate
Output:
A visual blend
Two objects are integrated into one object
Both objects are identifiable
pe
e
pe
Output:
A visual blend
egrated into one object
entifiable

61. Outline: AI Tools for Creative Work
• Introduce A Creative Problem: Visual Metaphors
• Can AI do it?
• VisiBlends: A Workflow for Prototyping Visual Metaphors
• Symbol Finder: AI tools for finding symbols
• VisiFit: AI Tools for Iteratively Improving Visual Blends
• The Future of AI and creative work

62. How often do VisiBlend prototypes need editing?
Output:
A visual blend
object
4 of 27 were ok to print on social media
(according to 2 evaluators who are graphic designers)

63. How do we improve the rest of them?

64. Fully automatic approaches fail
Artist:AI:
Design Principle #1 :
Instead of pursuing fully automatic approaches,
break up the objects into visual components that can each be blended.

65. Design Principle #2:
When improving a blend,
Recombine secondary visual aspects: color, silhouette, details.
Co-Design with 2 Graphic Designers

66. Formative study with novice designer
Design Principle #3:
Translating intent into action is a barrier for novices.
Use AI tools to automate actions or
provide more direct mapping between intent and action
“I want to make the metro card the
same shape as the purse.”

67. VisiFit: AI Tools to Iteratively Improve Visual Blends
Each visual aspect has it’s own AI tool for novices to apply

68. AI tools to improve each visual dimension
Color blending
Crop to extracted
silhouette
Deep Object
Segmentation
Interactive
Object
Segmentation
Interactive
Object
Segmentation

69. Evaluation
Improved:
97.5% of cases
11 Novice designers (8 female)
1 hr study
11 blends to extract and improve
Publishable quality:
65.3% of the time.
Experts Publishable:
87.0% of the time.
121 total blends produced
Outputs rated by 2 designers:
Improved, publishable.

70. Non-Publishable blends
1. Grabcut failed to extract details 2. No tool to fill parts of the bottom object.
(Add a fill tool)

71. Response of Professional Visual Blend Designers
1. Amazed by the AI tools
They did try things no novice did
(even if it didn’t work).
2. Much easier than “pixel pushing”.
Enabled them to explore more designs quickly and
easily.
But did suggest more PhotoShop features to all,
like Context Aware Fill
Wanted it as a plugin to sketch.
Everyone else Designer

72. Outline: AI Tools for Creative Work
• Introduce A Creative Problem: Visual Metaphors
• Can AI do it?
• VisiBlends: A Workflow for Prototyping Visual Metaphors
• Symbol Finder: AI tools for finding symbols
• VisiFit: AI Tools for Iteratively Improving Visual Blends
• The Future of AI and creative work

73. Crystallized
Intelligence
Crystallized
Intelligence
Fluid
Intelligence
Fluid
Intelligence
AI Tools can augment fluid intelligence
Collect pieces
Combine them
Combine them
Collect pieces
by making it easy to collect and combine pieces

74. If we identify the key aspects of the problem
VisiBlends collects and combines the
shape of blended objects
VisiFit extracts and combines the
color, silhouette, details of the objects
AI tools can help collect and combine elements for each aspect

75. AI Tools for writers:
Key aspects of writing:
Substance, structure, and style.
Metaphoria (CHI 2019):
Within the metaphor structure,
helps generate substance.
Style Thesaurus (C&C 2019):
Same substance, new style
(word level)
Language Style Transfer (INLG 2019):
Same substance, new style
(sentence level)
Katy Gero
PhD Candidate

76. Characteristic motion
Reference video
Static image
Future Work: Animated Blends
Use AI to extract key aspects of motion (speed),
and reapply them to the blend (use a different path).

77. Ecenaz Jen Ozmen
Columbia CC ‘21
Claire Chen
Columbia CC ‘21
Maria Kogan
Columbia SEAS ‘21
Sam Ross
Barnard ‘21
Annie Sui
Columbia CC ‘22
Maneesh Agrawala
Stanford
Katy Gero
PhD Candidate
Savvas Petridis
PhD Candidate
Collaborators

78. Summary

79. Future of Creative Work:
AI should augment fluid intelligence
AI shouldn’t try to understand complexities of culture and working context.
AI can provide people the tools and resources
to more easily turn users’ intent into action
AI Artist Novice with AI Tools

80. VisiBlends: AI Tools for Prototyping Visual Blends
VisiBlendsUnassisted
10x improvement in novices’ ability

81. VisiFit: AI Tools to Iteratively Improve Visual Blends
97.5% of cases improved, 65% to publishable quality

82. Crystallized
Intelligence
Crystallized
Intelligence
Fluid
Intelligence
Fluid
Intelligence
VisiBlends augments fluid intelligence
Collect pieces
Combine them
Combine them
Collect pieces
by making it easy to collect and combine pieces

83. Characteristic motion
Reference video
Static image
Future Work: Animated Blends
Use AI to extract key aspects of motion,
and reapply them to the blend.

84. Iteratively Improving: one key element at a time
Prototyping: collect and recombined parts Brainstorming: easily collect parts
AI Tools for Creative Work
Repurposing: Elements of a Reference Object