Images have the power to convey messages in striking and memorable ways. Although constructing visual messages is currently too hard for computers or novice users, by combining the intelligence of people and computers we can create compelling visual messages computationally. In this talk, we present VisiBlends, a flexible workflow for creating visual blends that follows the design process with steps involving brainstorming, synthesis, and iteration. An evaluation of the workflow shows that (1) decentralized groups of people can generate blends in independent microtasks, (2) co-located groups can collaboratively make visual blends for their own messages, and (3) VisiBlends improves novices’ ability to make visual blends.
We discuss how to decompose other complex tasks so that people and computers can collaborate in generating novel, useful and creative solutions to problems.
4. AdvertisementsNews Public Service Announcements
Earth + MeltBrazil + Takes Off Tabasco + Hot
Visual Metaphors
How can we systemically create visual metaphors?
5. Visual Metaphors are a creative design problem.
There is no simple recipe.
Design process:
• Brainstorm,
• Prototype many ideas,
• Select the best ones
• Iterate to improve designs
Ideate
Evaluate
Prototype
6. Outline: Crowdsourcing Visual Ads
• Illustration of the VisiBlends Pipeline
• 3 Evaluations:
• Decomposing the design process: Can users work on steps independently?
• Can groups collaborate to make visual blends?
• Can we improve novices’ ability to make Visual Blends?
• Can we decompose other creative tasks?
8. 1. Two objects are integrated into one object
2. Both objects are individually identifiable
Visual Blending Structure:
Whole-to-Part Shape Matching
Design Pattern: Single Shape Mapping
9.
10.
11. 1. Two objects are integrated into one object
2. Both objects are individually identifiable
Visual Blending Structure:
Whole-to-Part Shape Matching
A creative problem is now a search problem:
For two concepts, search for symbols that meet this constraint
28. A) Brainstorm
for summer
B) Find and annotate images
for summer
C) Matching algorithm
Starbucks + summer
D) Automatic blends + human evaluation
Starbucks + summer
matches = []
for a in summer_symbols:
for b in starbucks_symbols:
a_ratio = a.height / a.width
b_ratio = b.height / b.width
ratios = sort(a_ratio, b_ratio)
if (a.shape == b.shape) and
(a.coverage != coverage) and
(ratios[0] >= 0.5*ratios[1]):
matches.push([a, b])
return matches
VisiBlends:
A web interface to collaboratively make blends
29. Outline
• Illustration of the VisiBlends Pipeline
• 3 Evaluations:
• Decomposing the design process: Can users work on steps independently?
• Can groups collaborate to make visual blends?
• Can we improve novices’ ability to make Visual Blends?
• Can we decompose other creative tasks?
30. Decompose the pipeline Five people
Five people
One person
One person
One person
One person
One person
Computer: matching and
blending algorithms
33. To fix it, we trained people on the pipeline
steps backwards.
Now they know why they need to pick
Simple, iconic objects with a single main shape.
Because they’re seen how it affects the pipeline
34. Independent people can make blends for
random concept pairs.
NYC + FashionMcDonald’s + HealthyBicycle + Fall McDonald’s + Energy
35. When we taught the process backwards,
users were successful
36. Outline
• Illustration of the VisiBlends Pipeline
• 3 Evaluations:
• Decomposing the design process: Can users work on steps independently?
• Can groups collaborate to make visual blends?
• Can we improve novices’ ability to make Visual Blends?
• Can we decompose other creative tasks?
37. Study 2: Joe’s Coffee + Night
“Joe’s Coffee: Open Late”
Ad
44. Football Dangerous
Brainstorm
associations
Find Images
of objects
Annotate
shapes
Annotate
shape coverage
Are both objects identifiable?
Are two objects integrated into one object?
Evaluate
Blend
Shape covers
All of object
Shape covers
Part of object
45. Football Dangerous
Brainstorm
associations
Find Images
of objects
Annotate
shapes
Annotate
shape coverage
Are both objects identifiable?
Are two objects integrated into one object?
Evaluate
Blend
Shape covers
All of object
Shape covers
Part of object
53. Orange Healthy
Brainstorm
associations
Find Images
of objects
Annotate
shapes
Annotate
shape coverage
Shape covers
Part of object
Shape covers
All of object
Are both objects identifiable?
Are two objects integrated into one object?
Evaluate
Blend
55. Orange Healthy
Brainstorm
associations
Find Images
of objects
Annotate
shapes
Annotate
shape coverage
Shape covers
All of object
Shape covers
Part of object
Are both objects identifiable?
Are two objects integrated into one object?
Evaluate
Blend
Exercise equipment
56. When do we need to iterate?
No matches
Improve object fit
Objects are
not identifiable + =
+ =
Within same search space,
meet other constraints.
Find versions of an object
with different
color, style, aspect ratio
Focus on meeting a
specific constraint:
Find symbols with
a different shape
Search in a new
subspace
Find symbols with
a different shape
57. Outline
• Illustration of the VisiBlends Pipeline
• 3 Evaluations:
• Decomposing the design process: Can users work on steps independently?
• Can groups collaborate to make visual blends?
• Can we improve novices’ ability to make Visual Blends?
• Can we decompose other creative tasks?
58. Controlled Study
• 13 undergraduates
• Each made 6 blends
• Control-first condition
• 3 blends without VisiBlends
• 3 blends with VisiBlends
• System-first condition
• 3 blends with VisiBlends
• 3 blends without VisiBlends
• How many successful blends could they make?
59. Blend-making success with and without
VisiBlends
Control-first
Avg. Number of
blends made
System-first
Without VisiBlends With VisiBlends
Without VisiBlendsWith VisiBlends
Avg. Number of
blends made
60. Blend-making success with and without
VisiBlends
Control-first
Avg. Number of
blends made
System-first
Without VisiBlends With VisiBlends
Without VisiBlendsWith VisiBlends
Avg. Number of
blends made
0.56 5.56 (t(18)=4.88, p<0.001)
0.675.67 (t(21)=5.84, p<0.001)
61. Why is this task so hard?
There are a lot of constraints.
Novices without the system focused on
meeting one constraint
at the expense of others.
Usually they, found symbols, and then forced them
together.
62. Outline
• Illustration of the VisiBlends Pipeline
• 3 Evaluations:
• Decomposing the design process: Can users work on steps independently?
• Can groups collaborate to make visual blends?
• Can we improve novices’ ability to make Visual Blends?
• Can we decompose other creative tasks?
67. 1. Two objects are integrated into one object
2. Both objects are individually identifiable
Visual Blending Structure:
Whole-to-Part Shape Matching
by using an
abstract design pattern to turn it into a search problem.
We can decompose design problems
68. Independent people can make blends for
random concept pairs.
NYC + FashionMcDonald’s + HealthyBicycle + Fall McDonald’s + Energy
But…
69. Knowledge of the pipeline is needed to
motivate the rules, and fill in gaps.
Now they know why they need to pick
Simple, iconic objects with a single main shape.
Because they’re seen how it affects the pipeline1
2
3
71. Iteration is necessary to:
Fix small problems like
aesthetic refinement
Fix bigger problems by
refining the search for different shapes
Fix failure cases with unforeseen problems by
re-directing the search for different symbols
74. Future Work
Adding computational assistance to:
• Brainstorming
• Finding symbols
• Annotation
• Evaluation
Improving the blend quality
75. Crowdsourcing Visual Ads
Lydia Chilton
chilton@cs.columbia.edu
Columbia CS + DSI
We can use design patterns to
turn design problems into search
problems.
Iteration is necessary to
meet all search constraints
Groups can collaborate on visual blends
10x improvement to novices ability
1. Two objects are integrated into one object
Visual Blending Structure:
Whole-to-Part Shape Matching
Hinweis der Redaktion
Visual blends are a way to convey a message visually.
If you have a message you want to convey, like “Washing your hands is smart”
We can make a image that blends together two objects related to that message.
Visual Ads are everywhere.
Commercial and crass
They make a lot of money
Nobody knows how they work (but they do work)
To me, ads are interesting because they convey a message visually.
Convey a message
How do we make things like this?
I once wondered, what the considered the most creative ads.
“Creative ads”
Many of the results are visual metaphors.
This technical is used in new, advertising, and public service announcements to
Visual depict symbols of
Brazil + taking off
Tabasco + hot
Earth + melt
No surface level pattern:
Different colors, different objects, different meanings
But there is a more abstract pattern.
How do we get two objects to appear blended, but still individually identifiable?
We find objects that share a shape. Like a circle. When we blend on the circle, we get visual cues of both objects.
Here is an example of a tool we built to help people prototype with blends.
If we look only at the shapes of the objects,
The tabasco bottle is a cylinder.
The fire extinguisher is also mainly a cylinder.
If we combine the objects on their shape, they create a visual blend:
The tabasco label clearly identifies the tabasco bottle, and the hose outside the main shape clearly identifies the fire extinguisher
The light switch is a rectangle.
And the iPhone is also a rectangle
However, if we cover the entire iPhone, it will not be visible in the blend.
Instead, we cover only part of the iPhone with a rectangle.
Now both objects are blended and both objects are recognizable.
How do we get two objects to appear blended, but still individually identifiable?
We find objects that share a shape. Like a circle. When we blend on the circle, we get visual cues of both objects.
Let's go through the steps of the pipeline to create a blend that associates:
Starbucks and Summer.
….
The system automatically matches images from each concept with the
same shape and blends them.
Users then evaluate the blend by checking whether it has integrated two objects into one, and if both objects are identifiable.
Users can then modify color to improve the blend’s aesthetics.
The system automatically matches images from each concept with the
same shape and blends them.
Users then evaluate the blend by checking whether it has integrated two objects into one, and if both objects are identifiable.
Users can then modify color to improve the blend’s aesthetics.
And finally, if they are unhappy with the prototype, they can iterate.
This is a pipeline,
But it’s also fairly open.
People can move between step as they see fit.
Anytime you put in a new image, the matching algorithm will run, and it will look through all pairs for new matches.
This blend was made for the advertisement: “Joe’s Coffee: Open Late”
Visual blends are useful for conveying messages visually. To see if our pipeline could make blends for messages, we had trained groups of people use the pipeline to create visual blends for messages.
This blend was made for the public service announcement: “Wash your hands. It’s the smart thing to do.”
This blend was made for the advertisement: “Panel Discussion: Women in Computer Science”
This blend was made for the news headline: “Football linked to lasting brain damage”
This blend was made for the advertisement:
“Join the Philosophy Dept’s Holiday Celebration”
Football + Dangerous: We know which constraint is tight and look for that constrant.
Oragne + healthy: We need to search in a new subspace.
People naturally iterating.
They like linear steps that guarantee a result.
Evaluating intermediate steps is challenging, but it can be taught.
Four chord song:
https://www.youtube.com/watch?v=5pidokakU4I
I once wondered, what the considered the most creative ads.
“Creative ads”
Many of the results are visual metaphors.
Four chord song:
https://www.youtube.com/watch?v=5pidokakU4I
Visual blends are a way to convey a message visually.
If you have a message you want to convey, like “Washing your hands is smart”
We can make a image that blends together two objects related to that message.
