Thesis defence_rev4

CED RClemson Engineering Design Applications and Research
Varun Kumar
kumar3@clemson.edu
Committee:
Dr. Gregory Mocko (Chair)
Dr. Joshua Summers
Dr. Cameron Turner
UNDERSTANDING THE ROLE AND IMPORTANCE
OF DESIGN PROBLEMS IN CREATIVITY RESEARCH

2 of 43
kumar3@clemson.eduCED RClemson Engineering Design Applications and Research
 Key contributions
 Research questions and tasks
 Design problem reuse pattern
 Design problem similarity
o Based on structural elements
o Based on Latent Semantic Analysis
 Effect of problems on example interventions
 Research conclusions
 Research overview
 Recommendations and limitations
OUTLINE
Contributions RQs Problem reuse Structural sim. LSA sim. Concl.Regression Overview

3 of 43
KEY CONTRIBUTIONS
1. Identified pattern of design problem usage
• Problems not reused often
• Evaluated 15 years published literature
2. Provided two methods to compare conceptual problems
• Structural similarity based on count of elements within the
problem
• Semantic similarity based on LSA
3. Identified design problems as potential moderator in effectiveness of
examples as intervention
• Meta-regression to study problem size on results
4. Initiated creation of experiment design problem repository
• Developed structure and DB for experiment and problems

4 of 43
 Statements of need, requirement or function desired [1]
 Conceptual problems used in creativity research
WHAT ARE DESIGN PROBLEMS
Conceptual design problem example [2]
Design and build a low-cost, easy to manufacture peanut
shelling machine that will increase the productivity of
the African peanut farmers. Target throughput is
approximately 50 Kg per hour. The goals include: a)
Must remove the shell with minimal damage to peanuts
b) Electrical outlets are not available as a power source
c) A large quantity of peanuts must be quickly shelled.

5 of 43
RESEARCH QUESTIONS AND TASKS
RQ1: How can the pattern
of design problem usage in
creativity research be
identified?
Task 1:
Graph representation of
design problems
RQ2: How can design problem
be assessed for similarity
based on the problem
statement?
Task 2: Similarity assessment
based on structural elements
Task 3: Similarity assessment
based on Latent Semantic
Analysis

6 of 43
RESEARCH QUESTIONS AND TASKS
RQ3: How can natural
language representation be
used to compare problems
for similarity?
Task 2: Similarity
comparison based on
structural elements
Task 3: Similarity
comparison based on
Latent Semantic Analysis
RQ4: Does the choice of
design problem influence
effectiveness of interventions
(presenting examples)?
Task 4:
Meta-regression analysis of
user studies

7 of 43
DESIGN PROBLEM REUSE: GATHERING STUDIES
1. Creativity + engineering design 644,000
2. Creativity + engineering design + ideation 17,600
3.
Creativity + engineering design + ideation +
experiment
11,400
4. Between 2000 and 2014 8080
5. Design studies, design society, ASME 392
6. Abstract review for presence of experiment 129
7. Reported design problem 93
8. Removal of case studies 41
9. Removal of duplicate experiments 34
Contributions RQs Problem reuse Struct. sim LSA sim. Concl.Regression Overview

8 of 43
DESIGN PROBLEM REUSE: GRAPH CODING
Study Author Problem name
S1 Toh (Student), Miller (Faculty) Milk frothing device (DP11)
S2 Linsey (Faculty), Markman (Faculty), Wood (Faculty) Peanut shelling machine (DP9)
S3 Daly (Faculty), Christian (Faculty) Solar device (DP12)
S4
Glier (Faculty), Schmidt (Student)
Linsey (Faculty), McAdams (Faculty)
Peanut shelling machine (DP9)
S6 Acuna (Faculty), Sosa (Faculty) Counter top stand (DP13)
….. …… ……

9 of 43
Legends
Researcher
Problem Used Design problem-researcher use
• Some problems used
multiple times.
E.g: 59 (Peanut sheller)
used by 10 (Linsey) 7
times
• Problem choice depends
on researcher’s
requirements
• Problem reuse not
widespread across
researchers
DESIGN PROBLEM REUSE: GRAPH

10 of 43
• Is it possible to draw conclusions between studies when design
problems are different?
• Can we control one source of difference between studies?
• Can we compare problems to enable reuse?
DESIGN PROBLEM REUSE

11 of 43
STRUCTURAL SIMILARITY
• Question originated from work by Durand and coauthors [2]
o Size of the problem in terms of functional units
o Connectedness of the problem
o Participants' familiarity with the design problem
o Participants' familiarity with the design solutions
o Size (number of variables) of the potential solution space
o Are there assumed constraints due to known solutions
o The effort required to solve the problem
o The degree to which existing solutions will cause fixation
o The domain of the design problem

12 of 43
STRUCTURAL SIMILARITY
Design
Problem
Goals of the problem
Functional
Requirements
Non-functional
requirements
Information about end
user
Reference to an
existing product
• 5 elements of identified based on
o Literature
o Assessment of design
problems collected

13 of 43
STRUCTURAL SIMILARITY: PROTOCOL
Characteristic Questions asked
Number of goals • What is the final objective of the problem
statement?
• Does the problem statement ask only to design
one object or more?
Number of functional
requirements
• How many primary functions can you find?
• How many action verbs can you identify?
Number of non functional
requirements
• How many non-action and non-functional
aspects can you identify?
• How many performance and usability aspects
can you identify?

14 of 43
STRUCTURAL SIMILARITY: PROTOCOL
Characteristic Questions asked
user
• Can you identify who is going to use the
product?
Reference to existing
product
• Do you know if the product that needs to be
designed exists?

15 of 43
STRUCTURAL SIMILARITY: PROCEDURE
Adapted from [3]

16 of 43
STRUCTURAL SIMILARITY: OUTCOME
Design problems Goal FR NFR
Ref. existing
product User
Sketch Ideas for subway improvement. 1 1 0 1 0
There is a need of designing a new drawing
table……..
1 1 2 1 1
Tubular map cases consist of a system for…….. 1 4 0 1 1
Design a new system for gathering together………. 1 2 3 1 1
It is asked to design a new table for offices………… 1 1 1 1 1
Design a wearable binocular which satisfies……….. 1 1 3 0 0
Design an urban (bi or tri) cycle for use by “white-
collar workers”.
1 1 0 1 1
Design a water lifting device. 1 1 0 1 0
Design and build a low-cost, easy to
manufacture………
1 3 4 1 1
…….. ….. ….. ….. ….. ….
Krippendorff’s
Alpha
1.0 1.0 0.50 0.18 0.60

17 of 43
STRUCTURAL SIMILARITY: COMPARISON
Design problems Goal FR NFR Ref User
DP18 Devise an innovative solution for a remote controller
that meets……. 1 12 3 0 0
DP17 The Air Force wants to increase the functionality of
their Unmanned Air Vehicle technology…….. 1 11 1 1 1
DP3 Tubular map cases consist of a system for storage
and transportation of maps…………
1 4 0 1 1
DP13 Design a counter top stand to display and dispense
candy and chocolate snacks……... 1 4 5 1 1
DP9 Design and build a low-cost, easy to manufacture
peanut shelling machine that………………….
1 3 4 1 1
DP26 Design a machine that registers a bottle to a capping
station, caps it, and allows……………….
1 3 0 0 0
DP4 Design a new system for gathering together and
hiding the wires of the electronic
equipment………………….
1 2 3 1 1
…… ………….
…. …. …. …. ….

18 of 43
 5 elements analogous to a vector space
 Cosine between vectors gives separation
 Average structural similarity of 0.74
STRUCTURAL SIMILARITY: COSINES
DP10 DP11 DP12 DP13 DP14
DP10 1 0.70 0.80 0.80 0.70
DP11 0.70 1 0.61 0.60 1
DP12 0.80 0.61 1 0.99 0.61
DP13 0.80 0.60 0.99 1 0.60
DP14 0.70 1 0.61 0.60 1

19 of 43
 Extracts contextual meanings from large text corpus.
 Generates similarity scores between -1 and 1
 37 design problem statements tested for LSA similarity
SEMANTIC SIMILARITY: LSA

20 of 43
SEMANTIC SIMILARITY: RESULTS
DP10 1 0.05 0.14 0.09 0.07
DP11 0.05 1 0.43 0.42 0.12
DP12 0.14 0.43 1 0.38 0.33
DP13 0.09 0.42 0.38 1 0.28
DP14 0.07 0.12 0.33 0.28 1
A snippet of the LSA result matrix

21 of 43
STRUCTURAL AND SEMANTIC SIMILARITY
A1 LSA A1 LSA A1 LSA A1 LSA A1 LSA
DP10 1 1 0.70 0.05 0.80 0.14 0.80 0.09 0.70 0.07
DP11 0.70 0.05 1 1 0.61 0.43 0.60 0.45 1 0.12
DP12 0.80 0.14 0.61 0.43 1 1 0.99 0.38 0.61 0.33
DP13 0.80 0.09 0.60 0.45 0.99 0.38 1 1 0.60 0.28
DP14 0.70 0.07 1 0.12 0.61 0.33 0.60 0.28 1 1

22 of 43
DP12
Develop products that utilize sunlight for heating and cooking food. The products
should be portable and made of inexpensive materials. It should be able to be
used by individual families, and should be practical for adults to set up in a sunny
spot. Specific materials requirements for a targeted temperature can be
postponed to a later stage [4]
DP13
Design a counter top stand to display and dispense candy at convenience
stores. The requirements of this task are: a. The stand must be easy to use both
by the final user to grab the product and by the shop attendant to refill the
product. b. The stand must contain and visually identify one specific target brand
and product presentation. c. The stand must be built in one single material to
choose between cardboard or laminated plastic (PVC, PS or PETG).d. The stand
must be innovative, yet simple to manufacture and assemble [5].
Structural LSA
0.99 0.38

23 of 43
Design an automatic clothes-ironing machine for use in hotels. The purpose of
the device is to press wrinkled clothes as obtained from clothes dryers and fold
them suitably for the garment type. You are free to choose the degree of
automation. At this stage of the project, there is no restriction on the types and
quantity of resources consumed or emitted. However, an estimated 5 minutes
per garment is desirable [6]
Design an automatic recycling machine for household use. The device should
sort plastic bottles, glass containers, aluminum cans, and tin cans. The sorted
materials should be compressed and stored in separate containers. The amount
of resources consumed by the device and the amount of space occupied are not
limited. However, an estimated 15 seconds of recycling time per item is desirable
[6]
Structural LSA
0.81 0.52

24 of 43
Structural LSA
0.81 0.52

25 of 43
Alarm clocks are essential for college students, however often times they will wake up a
roommate and those around them as well. Design an alarm clock for individual use that will
not disturb others. The clock should be portable for use in a variety of situations such as on
the bus, in the library, or in a classroom. Customer Needs: a) Must wake up individual with
no disturbance to others b) Must be portable and lightweight c) Electrical outlets are not
available as a constant power source d) Low cost [2]
Corn is currently the most widely grown crop in the Americas with the United States
producing 40% of the world’s harvest. However, only the loose corn kernels are used when
bought canned or frozen in grocery stores. An ear of corn has a protective outer covering
of leaves, known as the husk, and strands of corn silk threads run between the husk and
the kernels. The removal of husk and silk to clean the corn is known as shucking corn.
Design a device that quickly and cheaply shucks corn for mass production. Customer
Needs: a) Must remove husk and silk from corn cob with minimal damage to kernels b) A
large quantity of corn must be shucked quickly. c) Low cost [2]
Structural LSA
0.70 0.30

26 of 43
Structural LSA
0.70 0.30
• Durand and coauthors conclusions:
o Quantity: No difference
o Quality: Alarm has higher quality solutions than corn
o Novelty: Corn higher than alarm
o Variety: No difference

27 of 43
 Used for identifying potential covariates
 Relationship between effect of intervention and covariate
 Use of examples in user studies
 Does design problem have a relationship with this intervention?
META-REGRESSION
Experiment
Design problem
Examples Results

28 of 43
STEPS FOR META-REGRESSION
Collect studies
Extract information
Calculate effect size and variance
Generate regression model
between effect size and covariate
Verify significance of model
• 9 studies out of 34 contained examples as
interventions.
• Additional search conducted using keywords ‘design
+ creativity + examples + experiment’ in Google
Scholar
• Filtering was done to collect between-subject
experiments
• Study should report statistics
• Studies which measure quantity or fluency, quality,
variety or novelty
• English language publications between 1995 - 2015
• Total N = 17 studies

29 of 43
• Method of assignment to condition
• Mean, standard deviation, sample size
• Design problem used and its problem
statement.
• Different datasets for 4 metrics
Collect studies
Extract information

30 of 43
Collect studies
Extract information
Generate regression model between
effect size and covariate
𝑆𝑀𝐷 =
𝑋𝑡𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡 − 𝑋𝑐𝑜𝑛𝑡𝑟𝑜𝑙
𝑠 𝑝
𝑠 𝑝 =
𝑛 𝐺1 − 1)𝑠 𝐺1
2
+ (𝑛 𝐺2 − 1)𝑠 𝐺2
2
𝑛 𝐺1 + 𝑛 𝐺2 − 2
𝑆𝑀𝐷′ = 1 −
3
4𝑁 − 9
× 𝑆𝑀𝐷
𝑆𝐸′ =
𝑛 𝐺1 + 𝑛 𝐺2
𝑛 𝐺1 𝑛 𝐺2
+
𝑆𝑀𝐷′ 2
)2(𝑛 𝐺1 + 𝑛 𝐺2

31 of 43
Study SMD’ SE’ Problem size
Tsenn -0.61 0.41 8
Smith,S 0.07 0.21 3
Shorachi 0.86 0.38 3
Lujun -0.21 0.33 2
Cardoso 0.15 0.33 2
Agogue -0.39 0.22 2
Chan -0.46 0.24 4
Goncalves -0.01 0.33 2
... … … …
Collect studies
Extract information
Generate regression model between
effect size and covariate
𝑷𝒓𝒐𝒃𝒍𝒆𝒎 𝒔𝒊𝒛𝒆 = 𝑵𝒐. 𝒐𝒇 𝒈𝒐𝒂𝒍𝒔 + 𝑵𝒐. 𝒐𝒇 𝑭𝑹𝒔 + 𝑵𝒐. 𝒐𝒇 𝑵𝑭𝑹𝒔

32 of 43
Collect studies
Extract information
where 𝛿𝑖 = predicted SMD
𝛽0, 𝛽1 = regression coefficients
εi = within study variance
ηi = residual variance
𝛿𝑖 = 𝛽0 + 𝛽1 × 𝑠𝑖𝑧𝑒 + 𝜀𝑖 + 𝜂𝑖

33 of 43
RESULTS OF META-REGRESSION: QUANTITY
Variable Coefficient SE 95% CI p
DP size -0.09 0.02 -0.14 to -0.04 <0.01
Constant 0.34 0.11 0.12 to 0.56 <0.01

34 of 43
RESULTS OF META-REGRESSION: QUALITY
DP size -0.07 0.07 -0.21 to 0.08 0.38
Constant 0.80 0.45 -0.08 to 1.67 0.08

35 of 43
RESULTS OF META-REGRESSION: NOVELTY
DP size 0.11 0.10 -0.08 to 0.30 0.25
Constant -0.27 0.39 -1.04 to 0.49 0.48

36 of 43
SUMMARY OF REGRESSION RESULTS
Creativity metrics
Quantity Quality Novelty
Number of sample points 44 13 34
tau 0.15 0.56 0.95
R2 48.02% 0% 0.59%
Coefficient of problem size -0.09* -0.07 0.11
* p < 0.05

37 of 43
 Structural similarity
o Offers a way for comparing two problem structures
o Five characteristics between problems can be compared
o Compares problems based on representation
 LSA
o Similarity assessed based on contextual meaning
o No human interpretation
 Element similarity and LSA
o Can be used together to generate a better similarity estimate
RESEARCH CONCLUSIONS

38 of 43
 Meta-regression
o Problem size affects effectiveness of examples presented
o Other differences can also affect effectiveness of examples
o Regression results should be verified through experiments
o Need for verification and validation of experimental findings
RESEARCH CONCLUSIONS

39 of 43
RESEARCH OVERVIEW
Graph of
problem and
author
Structural
similarity
Meta-
regression
Semantic
similarity
RQ1
Problem usage
pattern
RQ2
Using
natural
language for
similarity
RQ4
Impact of
problem
choice
RQ3
Similarity
comparison
between
problems
Low problem
reuse
Similarity between
problems
Problems can
moderate
effectiveness of
intervention
ContributionsResearch
tasks
Research
questions

40 of 43
 Problems similar to existing ones can be chosen
 LSA and five elements can be used for similarity comparison
 Repository of benchmarked problems with similarity scores
 Validation framework needed
 Common methodology for reporting studies
RECOMMENDATIONS

41 of 43
 Similarity assessment
o Do not address problem solvability
o LSA similarity and engineer’s interpretation may be different
o Element similarity needs robust protocol for identification
o Cosine similarity shows high similarity for most problems
 Meta-regression
o Hypothesis generating tool
o Results depend on reported data
LIMITATIONS

42 of 43
[1] Cross, N., 2008, “The nature of design,” Engineering design methods: strategies for
product design, John Wiley & Sons, Chippenham, Wiltshire, pp. 11–12.
[2] Durand, F., Helms, M. E., Tsenn, J., McAdams, D. A., and Linsey, J. S., 2015, “In
search of effective design problems for design research,” IDETC & CIE, American
Society of Mechanical Engineers, Boston, pp. 1–10.
[3] Cardoso, C., and Badke-Schaub, P., 2011, “The influence of different pictorial
representations during idea generation,” J. Creat. Behav., 45(2), pp. 130–146.
[4] Daly, S., and Christian, J., 2012, “Assessing design heuristics for idea generation in an
introductory engineering course,” Int. J. Eng. Educ., Vol. 28(2), pp. 1–11.
[5] Acuna, A., and Sosa, R., 2011, “The Complementary Role of Representations in Design
Creativity: Sketches and Models,” Design Creativity 2010, Springer, pp. 265–270.
[6] Patel, A., Kramer, W., Summers, J. D., and Shuffler-Porter, M., 2016, “Function Modeling:
A Study of Model Sequential Completion Based on Count and Chaining of
Functions,” International Design Engineering Conferences and Computers in
Engineering Conference (ASME IDETC/CIE, ASME-AMER SOC MECHANICAL
ENG, Charlotte, NC, pp. DETC2016–59860.
References

CED RClemson Engineering Design Applications and Research
Questions?

44 of 43
Maximum LSA score
DP29
Design the next generation of breakthrough alarm clock.
DP31
People generally use their mobile phone or a traditional alarm
clock in order to wake themselves up every morning. Yet, they
are not always effective and can sometimes cause
oversleeping. What would be other possible ways to wake up
in the morning in a certain time without using any form of
alarm clock? Generate as many ideas as possible.
LSA
0.53

45 of 43
Low LSA score
DP28
Redesign an electric toothbrush for increased portability.
DP30
Design a litter collection system for use by student groups in
volunteer "Adopt-A-highway" activities.
LSA
0

46 of 43
Appendix A: Design problem used for experimental verification
of protocol
Problem 1: Design a new system for gathering together and hiding the wires of the electronic equipment in an
office table. Currently the work in the field of design, architecture and engineering needs of a personal computer,
printers, and scanners. Each of these devices needs of electrical supply and the wires on table surface are annoying.
Actually, there are simple solutions to gather them, but it is difficult to extract or introduce a wire, or they leave the
wires hanging behind the table.
Problem 2: Design an automatic recycler device that can automatically sort plastic bottles, glass containers,
aluminum cans, and tin cans. The major differentiation between types of materials lay with the given dimensions of
the products: plastic bottles are the tallest, glass containers are very short and heavy, and aluminum cans are
lightweight. Devices are given strict requirements to adhere to such as volume and weight constraints, safety
requirements, and most importantly, have to operate autonomously once a master shutoff switch is toggled.
Problem 3: A mechanical system is required which, in the event of a fire, will enable people to escape from a
six-storey building by lowering themselves to the ground from windows. The system, which might make use of a 5
mm diameter steel cable, must be capable of lowering either a small child or a heavy adult at approximately the
same constant speed.
Problem 4: Design and develop an artifact to facilitate grocery shopping in a typical French/Singapore
Chinatown fresh market. The artifact should facilitate carrying of groceries from fresh market to home.

47 of 43
S.No. Characteristic Definition Scoring system
1.
Reference to existing
product
Whether or not the problem statement contains a
reference to an existing product.
Questions to be asked
 Does the problem statement contain reference to
an existing product?
 Does the problem statement ask you to re-design
an existing product?
If a reference to existing product exists in the
problem statement, assign a score of 1, else 0
2.
Functional
requirements (FR)
These are the things which the product needs to do, or
the tasks that you want the product to perform without
any reference to how it should be done. Judgement
should be based only on explicitly stated texts, and not
on the implied meaning of a sentence. To identify FRs,
look for:
1. Action verbs like move, work etc. associated with
objects (objects include nouns on which the
action verbs act like throw stones, gather fruits
etc.)
2. Primary functions of the design (eg. move
objects, lift, transport etc.).
3. These could also be nouns derived from verbs
(eg. washing machine, toaster etc.)
4. If there are two objects associated with one
primary function, count it as 2 separate FRs (eg.
move object X & object Y is counted as 2 FRs)
 What are the primary functions of the product?
 What are the expected outputs/tasks which the
product needs to perform?
Count the number of functional requirements
given in the problem statement. There can be 2
cases:
1: When a new product design is desired: In
this case, FR should be specified in the
problem statement. Else, give a score of 0.
2: When a re-design or a new design for an
existing product is desired: FR count in this
case is already 1 to start off, since atleast 1
product function is known.
Appendix B: Element definitions

48 of 43
S.No. Characteristic Definition Scoring system
3.
Non-functional
requirements
These are 'non-functional' requirements, which do not determine
the primary functions of the product, but cast a bound on the
overall shape, size, cost, operation and selection of the design.
Judgement should be based only on explicitly stated texts, and not
on the implied meaning of a sentence. Typical NFRs include:
1. Any restrictions on what the product or system shall or shall
not do apart from the primary functions (eg. should not
overheat, should be easy to use, should be manufacturable
etc.)
2. Any restriction on how the product shall fulfil its intended
functions (eg. device should move using rollers, device
should work using sliding mechanism etc.)
3. Any qualities that the product must possess (eg. easy to
make, easy to use, cheap, etc.)
• What things cast a limit or a bound on the solution space?
 What are the qualities that the product should possess as a
whole?
 How the overall product 'shall be' like?
Count the number of Non-functional requirements
given in the problem statement.
4. Number of goals
These are goals or final objectives associated with the design
problem.
 What is the final objective of the problem statement, or
 Does the problem statement ask only to design or do
something else?
Count the number of goals or objectives mentioned
in the problem statement.
5.
user
Information about who is going to use the product or who is the
customer. It should be explicitly stated in the problem statement.
 Who is going to be the end user of the product?
Check the problem statement to see if any
information about the end user is provided or not. If yes,
give a score of 1, else a 0
Appendix B: Element definitions

49 of 43
Problem statements
DP10
Redesign a traffic light using LED instead of an incandescent bulb. Ideas
generated must address the barred vision due to accumulation of snow as well
maintain the energy savings of LEDs.
DP11
Develop concepts for a new, innovative, product that can froth milk in a short
amount of time. The product should be able to be used by the consumer with
minimal instruction.
DP12
Develop products that utilize sunlight for heating and cooking food. The
products should be portable and made of inexpensive materials. It should be
able to be used by individual families, and should be practical for adults to set
up in a sunny spot. Specific materials requirements for a targeted temperature
can be postponed to a later stage.
DP13
Design a counter top stand to display and dispense candy and chocolate
snacks at convenience stores. The requirements of this task are:a. The stand
must be easy to use both by the final user to grab the product and by the shop
attendant to refill the product.b. The stand must contain and visually identify one
specific target brand and product presentation.c. The stand must be built in one
single material to choose between cardboard or laminated plastic (PVC, PS or
PETG).d. The stand must be innovative, yet simple to manufacture and
assemble.
DP14
Design a device to transport a ping-pong ball the farthest distance powered only
by a standard issue compression spring. The device is to be constructed with a
limited set of given materials (e.g. balsa wood, wire and Styrofoam).

50 of 43
Regression assumptions
Quantity
Quality

51 of 43
Novelty

52 of 43
Quantity
Quality

53 of 43
Novelty

54 of 43
 List of studies
 Meta regression dataset
Links to different data tables

Thesis defence_rev4

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Andere mochten auch

Andere mochten auch (13)

Ähnlich wie Thesis defence_rev4

Ähnlich wie Thesis defence_rev4 (20)

Thesis defence_rev4

Hinweis der Redaktion