In our thinking and acting, natural language plays a central part. This language defines a structure even before we form something, and it can be regarded as the architecture of design. To be able to grasp the expression of these structures in HCI, we chose the perspective of linguistics and semiotics. The semiotics perspective in the context of HCI is increasingly popular in presenting a different approach to UX. In our paper, we take this perspective to build a set of semiotic heuristics which we then used to evaluate a complex UI example. We present a semiotic evaluation method and report the results of our in-depth investigation.

1. Semiotics of Interaction:
Towards a UI Alphabet
Jan Brejcha, Charles University, Prague
Aaron Marcus, Aaron Marcus and Associates, Inc., Berkeley
HCII 2013, Las Vegas 26 July 2013

2. Semiotic Foundation
• Semiotics: theory of signs
• Sign: “something that stands for someone or some-thing in some respect or
capacity” (C.S. Peirce)
• Four semiotic dimensions:
• Pragmatic: psychological, biological, and sociological impacts of using
signs
• Semantic: meaning, reference, or action of signs
• Syntactic: relations among signs (syntax)
• Lexical: physical constraints of generating signs
• Pragmatics regarding UI analysis:
user-UI and user-UI-designer relationships

3. UI Language Components
• UI grammar is composed of basic elements: interaction sentence,
interaction games, rhetorical tropes, interaction phases, and patterns. The
grammar elements concern both the noun and verb phrase of a sentence.
• Discrete elements are the smallest elements to have a meaning.
• Interaction sentence is a meaningful unit describing a task in a user's
interaction. A set of interaction sentences with the same goal form an
interaction game.
• Narrative in UI is made both by the designer’s meta-communication and the
temporal aspect of perceiving UI elements.
• Rhetorical tropes are devices of persuasion and emphasis, often
presented as metaphors.
• Patterns are typical configurations of UI language components in different
settings

4. UI Language Components: Summary
• Grammar sets rules regarding interaction chains:
• What can be chained in a cause-effect interaction unit
• How it can be chained
• For what purpose it can be chained.
• Studying interaction grammar helps to improve UI and interaction
consistency
• By promoting consistency, interaction grammar helps to build
expectations
• Constraints reinforce interaction grammar rules
• Physical (affordances), logical (reasoning), and cultural (conventions)

5. UI Analysis and Annotation Carried Out in
Study
• To analyze visible and interaction relations in set of UIs, one must extract
intrinsic grammar forming the UI “language”
• Semiotic analysis uses structured interaction transcript based on interaction
sentences
• Resulting transcript serves as input for follow-on expert evaluation

6. Evaluation Methods
• Expert evaluation methods available:
• Cognitive walkthrough
• Heuristic evaluation (HE, our choice)
• Expert inspection
• Semiotic analysis (SA, our choice)
• Evaluation process of initial study:
• Study used two complex graphic design applications:
• Adobe Photoshop
• GIMP
• Evaluate two UIs using both HE and SA methods
• Compare data resulting from each

7. Heuristic Evaluation (HE)
• Semiotic analysis focuses on signs in the UI, extracts underlying codes that
make/break UI’s meaning.
• Primary objective:
• Establish underlying conventions
• Identify significant differences and oppositions
• Model system categories, relations, connotations, distinctions and rules of
combination employed. (Chandler)
• Benefits:
• Provides more insight, captures more compatible data, at lower cost
• More amendable to cross-cultural analysis
• More accessible/understandable to stakeholders
• Much previous data and analysis examples exist
• Low overlap with heuristic evaluation

8. Heuristic Evaluation: Criteria
• Aesthetic integrity and minimalist design
• Consistency and standards
• Direct manipulation/See and point
• Error prevention
• Feedback/Visible system status
• Fitt’s Law
• Flexibility and efficiency of use
• Help and documentation
• Help users recognize, diagnose, and recover from errors
• Information legibility/Density
• Match between system and real world
• Modelessness
• Perceived stability
• Recognition rather than recall
• User control and freedom
• Visible interfaces/WYSIWYG

9. Semiotic Analysis (SA)
• Semiotic analysis focuses on signs in the UI, extracts underlying codes that
make/break UI’s meaning.
• Primary objective:
• Establish underlying conventions
• Identify significant differences and oppositions
• Model system categories, relations, connotations, distinctions and rules of
combination employed. (Chandler)
• Benefits:
• Provides more insight, captures more compatible data, at lower cost
• More amendable to cross-cultural analysis
• More accessible/understandable to stakeholders
• Much previous data and analysis examples exist
• Low overlap with heuristic evaluation

10. Semiotic Analysis: Heuristics
• Actors, audience and paradigm. Defining who the UI users are, who the
intended audience of the UI is and what the leading interaction paradigm is.
• Symbols. Different kinds of symbols connote different semantic spaces, cultural
backgrounds and address different audiences.
• Syntax. Signs should be used in any given context only once and should not be in
conflict with its context.
• Rhetorical tropes. The most common rhetorical tropes in the UI are devices of
substitution: metaphor, metonymy, prosopopoeia, and synecdoche.
• Interaction phases. The interaction should form meaningful temporal units.
• Patterns. UI language components form different kinds of patterns.

11. UI Corpus: Selection for Initial Study
• To get a workable UI set, study must select sample body of material: UI
corpus
• Study compared UI of Adobe Photoshop CS2 and GIMP 2.6.7, both running
on Mac OS X 10.5.8.
• Corpus must be as homogeneous as possible (in substance and time)
• Study chose features that were commercially advertised, as well as basic
core functions which all users are likely to perform

12. UI Corpus: Functions Analyzed
• Study found five shared functions in both UIs:
• Reduce red-eye: One-click red-eye tool vs. Red-eye removal
• Barrel distortion: Optical lens correction vs. Perspective clone tool
• Eliminate an object: Healing brush vs. Clone tool
• Clone objects in perspective: Vanishing pt. vs. Perspective clone tool
• Customize UI: Keyboard shortcuts and menu customization vs. Configure
keyboard shortcuts
• Study process analyzed these functions:
• Transcribed step-by-step process involved (interaction sentences)
• Carried out HE and SA for each UI

13. UI Corpus: Interaction Sentence Example
for Adobe Photoshop, 1/2
• (0) Open picture to adjust
• (1) Find proper function in menu or tool palette
• (a) Sub-task: look in toolbox for button resembling intended action
• Nothing like that was found
• (b) Alternate sub-task: look through menu items (especially in what seemed
most related (Image -> Adjustments, and Filter) for relative command
• Found under Filter -> Vanishing point…
• Window called “Vanishing Point” appears
• Window shows live preview, “Create Plane Tool“, “Clone Tool", etc.
• (2) Click four corners according to information provided:
• “Click the four corners of a perspective plane or object in the image to
create an editing plane. Tear off perpendicular planes from the stretch
nodes of existing…)”

14. UI Corpus: Interaction Sentence Example
for Adobe Photoshop, 2/2
• (3) Select Clone tool
• (4) Option-click in plane to set
source
• Opt+click in plane to set source
point for clone
• Once source point is set,
click+drag to paint or clone
• Shift+click to extend stroke to last
click
• (5) Click-drag (to paint) several
times to clone in perspective
• See proposed results in preview.
• (6) Click “OK“ to apply changes
• (7) Save changes to file

15. UI Corpus: Results of HE vs SA
• HE results:
• More concise; of 16 heuristics, only small number applied each time
• Could be used not only on interaction-sentence level, but also as general
analysis of entire UI
• SA results:
• SA tended to be more verbose
• All elements could be applied every time
• Could seem repetitive in some instances
• Provided solid context of analysis
• Enables comparison of UIs from different time-periods to show changes in
paradigm, syntax, and related SA criteria

16. Summary
• We applied the semiotic and linguistic theories to an UI corpus of graphic manipulation
applications. At the same time we analyzed the corpus using heuristic evaluation.
• We presented the transcript of interaction sentences forming language games, that
served together with the actual UI as a basis for Heuristic evaluation (HE) and Semiotic
analysis (SA). Moreover, the transcript served as input for defining the different elements
involved in the interaction language.
• The SA provided the expected kind of data, that gathered a wider context than those from
HE. That said, SA can be used to complement the widely used expert evaluation
methods, but could possibly be defined to have a higher overlap with HE. In the latter
case, SA would need to be evaluated hand in hand with the interaction sentences.
• In summary, our study demonstrated the depth of investigation and breadth of insight that
SA can achieve in HCI and how this could enhance the current practice. Both methods
could be merged to provide a best-of-both solution.

17. Future Work Plan
• Develop a pattern visualization for UI language elements. Such visualization
would help quickly compare interaction structures with different kinds of UI in
one culture, or between two or more cultures.
• Compare the SA with other semiotic methods.
• Focus SA to UI pragmatics (persuasion, ideologies).

18. Semiotics of Interaction:
Towards a UI Alphabet
Jan Brejcha, Doctoral Candidate
Charles University in Prague
U Kříže 8, 15800 Praha 5 – Jinonice, Czech Republic
Tel: +420 602 277 588
Email: Jan@Brejcha.name
Web: Jan.Brejcha.name
Aaron Marcus, President
Aaron Marcus and Associates, Inc. (AM+A)
1196 Euclid Avenue, Suite 1F. Berkeley, California 94708-1640, USA
Tel: +1-510-601-0994, Fax: +1-510-527-1994
Email: Aaron.Marcus@AMandA.com
Web: www.AMandA.com