People use search engines to find answers to questions related to their health, finances, or other socially relevant issues. However, most users are unaware that search results are considerably influenced by search engine marketing (SEM). SEM measures are driven by commercial, political, or other motives. Due to these motivations, two questions arise: What information quality is mediated through SEM? And how is collecting documents of different quality affecting user knowledge gain? Both questions are not considered by existing models of information behavior. Hence, the doctoral research project described in this paper aims to develop and empirically test an information search behavior model on the influences of SEM on user knowledge gain and thereby contribute to the search as learning body of research. Presentation at CHIIR 2023 Doctoral Consortium.

1. HOW SEARCH ENGINE MARKETING INFLUENCES USER KNOWLEDGE GAIN:
DEVELOPMENT AND EMPIRICAL TESTING OF AN INFORMATION SEARCH
BEHAVIOR MODEL
Sebastian Schultheiß
Hamburg University of Applied Sciences
ACM SIGIR Conference on Human Information
Interaction and Retrieval (CHIIR ’23)
March 19–23, 2023, Austin, TX, USA
Supervisors
Prof. Vivien Petras, PhD, Humboldt University of Berlin
Prof. Dr. Dirk Lewandowski, University of Duisburg-Essen and Hamburg University of Applied Sciences

2. 1 INTRODUCTION

3. CHIIR ’23
Sebastian Schultheiß
INTRODUCTION
 Search engines are integral to the everyday life of their users (Haider & Sundin, 2019).
 They are used for learning purposes and to make important decisions, e.g., on political or health topics
(e.g., Epstein & Robertson, 2015; Ray, 2020).
 When interacting with search engines, users focus on prominently placed results and select them more often
(e.g., Cutrell & Guan, 2007). 2
Asthma at night
Users

4. CHIIR ’23
Sebastian Schultheiß
INTRODUCTION
3
Paid search
marketing
Search engine
optimization
Content
producers
Asthma at night
 The order on the result page is not shaped by the search engine providers alone (Röhle, 2010).
 Content producers act hand in hand with search engine marketing to foster the visibility of results:
• Paid search marketing (PSM): sponsored results labeled with “Ad“ (Li et al., 2014)
• Search engine optimization (SEO): measures to improve the ranking within organic results (Li et al., 2014)
 Consequently, users mostly come across documents related to marketing activities during their search.
Users
Search engine
providers

5. CHIIR ’23
Sebastian Schultheiß
INTRODUCTION
4
Paid search
marketing
Search engine
optimization
Content
producers
Search engine
providers
Users
Asthma at night
 Search engine marketing measures are driven by commercial, political, or other motives (Röhle, 2010).
 Motivations like commercial interests influence how users think about the objectivity of a page (e.g., Sun et al., 2019).
 Promoting representational information quality (IQ) is a focus of search engine optimization (Searchengineland.com, 2021).
 As a result, search engine marketing measures can be associated with a web page's information quality.
Motivations for
search engine
marketing
Intrinsic IQ:
e.g., objectivity,
believability
(Lee et al., 2002)
Representational IQ:
e.g., ease of understanding,
consistent representation
(Lee et al., 2002)

6. CHIIR ’23
Sebastian Schultheiß
Motivation and objectives of my doctoral project
Questions that I have for you
INTRODUCTION
- Search engine marketing is ubiquitous and has the potential to influence learning and decision
making of searchers.
- Models of information behavior do not yet cover these influences (e.g., Agarwal, 2022; Robson & Robinson, 2013).
- The Objective of my doctoral research project is to develop and empirically test an information
search behavior model on the influence of search engine marketing on user knowledge gain.
5
- What is your perspective on my topic? Is the topic relevant?
- Is the procedure plausible from your point of view? Do you see any potential pitfalls,
shortcomings, or inconsistencies?
- What implications of the research do you see?

7. 2 CENTRAL RESEARCH QUESTIONS

8. CHIIR ’23
Sebastian Schultheiß
CENTRAL RESEARCH QUESTIONS
7
RQ1: What is the relationship between search engine marketing measures and the
representational information quality of a document?
RQ2: What is the relationship between search engine marketing measures and the
intrinsic information quality of a document?
RQ3: How does selected documents' representational and intrinsic quality influence
user knowledge gain?
Representational
information quality
Intrinsic
information quality
Search engine
marketing
Knowledge
gain
User study
Information quality
evaluation
Users

9. 3 USER STUDY
PROCEDURE

10. CHIIR ’23
Sebastian Schultheiß
USER STUDY: PROCEDURE
9
 A sample representative of the German online population will participate in the study (N = 1,000–2,000 subjects).
 The core of the user study are tasks that the subjects will work on using predefined SERPs.
 Before and after each task, user knowledge gain is measured by knowledge tests.
 The tasks are complemented by surveys.

11. 4 USER STUDY
TASKS

12. CHIIR ’23
Sebastian Schultheiß
USER STUDY: TASKS
11
 The user study will include tasks from the socially relevant fields of health, politics, and
environment.

13. CHIIR ’23
Sebastian Schultheiß
USER STUDY: TASKS
12
 For each field, three topics are drawn from lists of terms, such as lists of diseases provided by public authorities
(e.g., Federal Ministry of Health, 2023).
 Topics are not selected randomly but according to assumed familiarity from the subjects’ point of view.

14. CHIIR ’23
Sebastian Schultheiß
USER STUDY: TASKS
13
 For each topic (e.g., “asthma”), 20 topical aspects (e.g., “What helps with asthma at night?”) are collected.
 The topical aspects come from Google’s “people also ask.”

15. CHIIR ’23
Sebastian Schultheiß
USER STUDY: TASKS
14
 Tasks are developed for all topical aspects.
 The tasks need to be formulated easily to be understandable for all subjects, i.e., German Internet users.

16. CHIIR ’23
Sebastian Schultheiß
USER STUDY: TASKS
15
 Task complexity influences user behavior (e.g., Arguello et al., 2012; Roy et al., 2022).
 Therefore, student jurors will assess task complexity using cognitive process dimensions
(e.g., remember, understand, analyze) by Anderson et al. (2001).

17. CHIIR ’23
Sebastian Schultheiß
USER STUDY: TASKS
16
 Three tasks per topic are selected for which the evaluators showed the highest agreement in their complexity
ratings.

18. 5 USER STUDY
SERPS

19. CHIIR ’23
Sebastian Schultheiß
USER STUDY: SERPS
18
 For each task, a set of organic (e.g., N = 200) and paid search results (e.g., N = 10) is collected.

20. CHIIR ’23
Sebastian Schultheiß
USER STUDY: SERPS
19
 The SEO probability of all collected results is determined by the SEO classification tool developed in our
research group (Lewandowski et al., 2021).

21. CHIIR ’23
Sebastian Schultheiß
USER STUDY: SERPS
20
 Optimized and non-optimized organic and paid search results are selected.

22. CHIIR ’23
Sebastian Schultheiß
USER STUDY: SERPS
21
 One SERP is constructed for each task.
 The SERPs contain the previously selected ads and organic results in randomized order.

23. 6 INFORMATION QUALITY EVALUATION

24. CHIIR ’23
Sebastian Schultheiß
INFORMATION QUALITY EVALUATION
23
 Experts from the respective field, e.g., health, evaluate the information quality of all search results of the user study.
 For this purpose, the AIM Quality (AIMQ) questionnaire by Lee et al. (2002) is used.

25. 7 CONCLUSION

26. CHIIR ’23
Sebastian Schultheiß
CONCLUSION
Aim
Developing and empirically testing of an information search behavior model on the influence of search
engine marketing on user knowledge gain.
Methods
User study and information quality evaluation.
Contributions
- Contributing to the search as learning body of research (Hoppe et al., 2018).
- Better understanding of the relationship between search engine marketing and information quality.
- Research data including tasks, SERPs, and quality judgements.
- Content producers can improve user knowledge gain by meeting quality criteria.
Next steps
1. Preparing the SERPs, tasks, and questionnaires for both methods.
2. Acquiring experts for the information quality evaluation.
3. Selecting a market research institute and starting the coordination of the user study.
4. Formulating the theoretical parts of the thesis. 25

27. THANK YOU FOR YOUR ATTENTION!
Sebastian Schultheiß
Hamburg University of Applied Sciences
Research group Search Studies
searchstudies.org/team/schultheiss/
0000-0003-2704-7207
Acknowledgements
I would like to thank my supervisors Prof. Vivien Petras, PhD and Prof. Dr. Dirk Lewandowski for their ongoing support.
I also thank Dr. Maria Gäde, Dr. Johanne Trippas, Dr. Stephann Makri, and the CHIIR DC reviewers for their valuable feedback.
This work is funded by the German Research Foundation (DFG Deutsche Forschungsgemeinschaft), grant number 467027676.
- What is your perspective on my topic? Is the topic relevant?
- Is the procedure plausible from your point of view?
- Do you see any potential pitfalls, shortcomings, or inconsistencies?
- What implications of the research do you see?
- Any other comments or questions?

28. REFERENCES

29. CHIIR ’23
Sebastian Schultheiß
REFERENCES
Agarwal, N. K. (2022). Integrating models and integrated models: Towards a unified model of information seeking behaviour. Information Research: An International Electronic Journal, 27(1),
Article 1. https://doi.org/10.47989/irpaper922
Anderson, L. W., Krathwohl, D. R., Airasian, P. W. ., Cruikshank, K. A., Mayer, R. E., Pintrich, P. R., Raths, J., & Wittrock, M. C. (Eds.). (2001). A taxonomy for learning, teaching, and
assessing: A revision of Bloom’s taxonomy of educational objectives. Longman.
Arguello, J., Wu, W.-C., Kelly, D., & Edwards, A. (2012). Task complexity, vertical display and user interaction in aggregated search. Proceedings of the 35th International ACM SIGIR
Conference on Research and Development in Information Retrieval, 435–444. https://doi.org/10.1145/2348283.2348343
Cutrell, E., & Guan, Z. (2007). What are you looking for? Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI ’07, 407–416.
https://doi.org/10.1145/1240624.1240690
Epstein, R., & Robertson, R. E. (2015). The search engine manipulation effect (SEME) and its possible impact on the outcomes of elections. Proceedings of the National Academy of Sciences,
USA, 112(33), 4512–4521. https://doi.org/10.1073/pnas.1419828112
Federal Ministry of Health. (2023). Conditions A–Z | gesund.bund.de. https://gesund.bund.de/en/conditions-a-to-z
Haider, J., & Sundin, O. (2019). Invisible Search and Online Search Engines. Routledge.
Hoppe, A., Holtz, P., Kammerer, Y., Yu, R., Dietze, S., & Ewerth, R. (2018). Current Challenges for Studying Search as Learning Processes. LILE2018 -- Learning & Education with Web Data,
2–5. https://lile2018.wordpress.com/
Lee, Y. W., Strong, D. M., Kahn, B. K., & Wang, R. Y. (2002). AIMQ: a methodology for information quality assessment. Information & Management, 40(2), Article 2.
https://doi.org/10.1016/S0378-7206(02)00043-5
Lewandowski, D., Sünkler, S., & Yagci, N. (2021). The influence of search engine optimization on Google’s results. 13th ACM Web Science Conference 2021, 12–20.
https://doi.org/10.1145/3447535.3462479
Li, K., Lin, M., Lin, Z., & Xing, B. (2014). Running and Chasing—The Competition between Paid Search Marketing and Search Engine Optimization. In R. H. Sprague (Ed.), 2014 47th Hawaii
International Conference on System Sciences (pp. 3110–3119). IEEE. https://doi.org/10.1109/HICSS.2014.640
Ray, L. (2020). 2020 Google Search Survey: How Much Do Users Trust Their Search Results? https://moz.com/blog/2020-google-search-survey
Robson, A., & Robinson, L. (2013). Building on models of information behaviour: Linking information seeking and communication. Journal of Documentation, 69(2), 169–193.
https://doi.org/10.1108/00220411311300039
Röhle, T. (2010). Der Google-Komplex: Über Macht im Zeitalter des Internets. Transcript. https://doi.org/10.14361/transcript.9783839414781
Roy, N., Maxwell, D., & Hauff, C. (2022). Users and Contemporary SERPs: A (Re-)Investigation. Proceedings of the 45th International ACM SIGIR Conference on Research and Development
in Information Retrieval, 2765–2775. https://doi.org/10.1145/3477495.3531719
Searchengineland.com. (2021). Periodic Table of SEO Factors 2021. https://searchengineland.com/seotable
Sun, Y., Zhang, Y., Gwizdka, J., & Trace, C. B. (2019). Consumer Evaluation of the Quality of Online Health Information: Systematic Literature Review of Relevant Criteria and Indicators.
Journal of Medical Internet Research, 21(5), Article 5. https://doi.org/10.2196/12522 28