How Correctness of Decision Support
Systems Influences User’s Performance
in Production Environments
Defective Still Defle...
RWTH Aachen University, Human-Computer Interaction Center
Context: Part of the Cluster of Excellence
“Integrative Producti...
RWTH Aachen University, Human-Computer Interaction Center
Our research objective:
Optimize cross-company cooperation
 Opt...
RWTH Aachen University, Human-Computer Interaction Center
What determines performance in
Complex Cyber-Physical Production...
RWTH Aachen University, Human-Computer Interaction Center
How can Human and Interface factors be investigated?
Business Si...
RWTH Aachen University, Human-Computer Interaction Center
Business Simulation Games as a
Research Lab for Understanding Sy...
RWTH Aachen University, Human-Computer Interaction Center
Study with Business Simulation Game
Can Interfaces support Users...
RWTH Aachen University, Human-Computer Interaction Center
Follow up studies
Focus on singular decisions, focus on specific...
RWTH Aachen University, Human-Computer Interaction Center
Decision Support Systems (DSS)
 “Decision Support Systems aid i...
RWTH Aachen University, Human-Computer Interaction Center
Research Questions
 Do Decision Support Systems influence
opera...
RWTH Aachen University, Human-Computer Interaction Center
Experimental Setup
 Task in Material Disposition Context
– Comp...
RWTH Aachen University, Human-Computer Interaction Center
Description of the Sample
 20 participants
 Age
21 – 55 years
...
RWTH Aachen University, Human-Computer Interaction Center
Results
Baseline experiment (no Decision Support System)
2016-07...
RWTH Aachen University, Human-Computer Interaction Center
Baseline Experiment (no DSS)
Factor Decision
 Significant main ...
RWTH Aachen University, Human-Computer Interaction Center
Baseline experiment (no DSS):
Factor Table Length
 Significant ...
RWTH Aachen University, Human-Computer Interaction Center
Results
Effect of the Decision Support Systems
2016-07-20 P. Bra...
RWTH Aachen University, Human-Computer Interaction Center
Effect of a Decision Support System of Effectivity and Efficienc...
RWTH Aachen University, Human-Computer Interaction Center
Effect of Decision Support Systems on Reaction Times
Closer exam...
RWTH Aachen University, Human-Computer Interaction Center
Effect of Decision Support Systems on Accuracy
Closer examining ...
RWTH Aachen University, Human-Computer Interaction Center
Research Questions & Answers
 Do Decision Support Systems influ...
RWTH Aachen University, Human-Computer Interaction Center
Thank you four your attention!
Summary
 Industrial Internet lea...
RWTH Aachen University, Human-Computer Interaction Center
 Brauner P, Runge S, Groten M, et al (2013) Human Factors in Su...
Nächste SlideShare
Wird geladen in …5
×

Defective Still Deflective – How Correctness of Decision Support Systems Influences User’s Performancein Production Environments

57 Aufrufe

Veröffentlicht am

The increasing dynamic and complexity of todays global supply chains and the growing amount and complexity of information challenge deci- sion makers in manufacturing companies. Decision Support Systems (DSS) can be a viable solution to address these challenges and increase the overall decision efficiency and effectivity. However, a thought-through design and implementation of these systems is crucial for their efficacy.
This article presents the current state-of-the-art of Decisions Support Systems and highlights their benefits and pitfalls. Also, we present an empirical study in which we compared different levels of decision support and decision automation in a simulated supply chain game environment.
We identify and quantify how human factors influence the decision quality and decision performance in this supply chain scenario. We show that an adequately designed system raises the overall performance. However, insufficiently designed systems have the reverse effect and lead operators to miss severe situations, which can have fatal consequences for manufacturing companies.

Veröffentlicht in: Daten & Analysen
0 Kommentare
0 Gefällt mir
Statistik
Notizen
  • Als Erste(r) kommentieren

  • Gehören Sie zu den Ersten, denen das gefällt!

Keine Downloads
Aufrufe
Aufrufe insgesamt
57
Auf SlideShare
0
Aus Einbettungen
0
Anzahl an Einbettungen
0
Aktionen
Geteilt
0
Downloads
0
Kommentare
0
Gefällt mir
0
Einbettungen 0
Keine Einbettungen

Keine Notizen für die Folie
  • I am Philipp Brauner from the Human-Computer Interaction Center at Aachen University in Germany. I am here to present the article I have written together with Matina Ziefle on an ongoing research project for which we developed a new research methodology based on serious games…

    But first, what is the Human-Computer Interaction Center?
  • Although many processes can be automated, humans still are responsible and make the final decisions. Refrence to the talk of my colleague André from yesterday.
  • Tests in the field: High external validity, low internal validity, and low generalizability
    Test in the lab: High internal validity, low external validity, little transferability

    I also have a lab study in the talk later
  • Game based learning environment effetive tool to
    * increase performance over time through training
    Raise awarness for specific supply chain and disposition topics
  • Trust in Automation, perceptual speed nötig?


    Perceptual speed 18 – 19 points, ∅ 24.7±5.0 pt
    Trust in Automation 2.5 – 5.2 points, ∅ 4.0 ± 0.9 pt
  • Astonishing insofar, as there are some perceptual processes that are not processed linearly, but in total.
  • 1)
    Decsion Support Systems are a good think, as long as they work correctly. => Increase accuracy and decrease reaction times.
    Especially in more complex environments with more data to be considered.

    2)
    Decsision Support Systems can be deflecting if they do not work correctly.
    Despite the participants knowing of the defect (immediate feedback), they partialy relied on the DSS. While the reaction times were just slightly lower compared to the baseline, the accuracy dropped. Especially in more complex environments.

    3)

  • Dies kann zum Deskilling, also Verlernen der eigentlichen Tätigkeit oder zur sogenannten „OOTLUF” führen, zur „out of the loop unfamiliarity”, welche dadurch definiert ist, dass der Entscheider auf sich plötzlich verändernde Rahmenbedingungen nicht adäquat reagieren kann (Wickens et al. 2013: S.388 ff)
  • Defective Still Deflective – How Correctness of Decision Support Systems Influences User’s Performancein Production Environments

    1. 1. How Correctness of Decision Support Systems Influences User’s Performance in Production Environments Defective Still Deflective Philipp Brauner André Calero Valdez Ralf Philipsen Martina Ziefle Human-Computer Interaction Center RWTH Aachen University, Germany Human-Computer Interaction International 2016 Toronto, Canada Philipp Brauner , André Calero Valdez, Ralf Philipsen, Martina Ziefle, Defective Still Deflective – How Correctness of Decision Support Systems Influences User’s Performance in Production Environments, HCI in Business, Government, and Organizations: Information SystemsVolume 9752 of the series Lecture Notes in Computer Science pp 16-27, 978-3- 319-39398-8, Springer International Publishing (2016)
    2. 2. RWTH Aachen University, Human-Computer Interaction Center Context: Part of the Cluster of Excellence “Integrative Production Technology for High-Wage Countries”  Goal: Strengthen competitiveness of high wage countries  Engineering of future production systems – New materials and processes – Improved and smarter machinery – Optimize assembly cells, shop floor, cross-company cooperation  > 25 Institutes, > 100 researchers, > 60M€ page 2 2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems Influences User’s Performance in Production Environments" - HCII 2016
    3. 3. RWTH Aachen University, Human-Computer Interaction Center Our research objective: Optimize cross-company cooperation  Optimize cross-company supply chains (SC) – Technical factors influencing performance of SCs – Human Factors influencing performance of SCs – Interface Factors on SCs performance – Interrelationship of technical, interface, and human factors  Why are humans considered? – Humans make final decision – Overview over not explicitly modelled relationships (e.g., closed-world assumption) – Complexity and uncertainty increases, less time for making decisions Information flow flow of goods Supply Chain page 3 2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems Influences User’s Performance in Production Environments" - HCII 2016 Goal: Understand system and user factors that influence efficiency, effectivity, and user satisfaction in Enterprise Resource Planning Systems, Supply Chains and Quality Management.
    4. 4. RWTH Aachen University, Human-Computer Interaction Center What determines performance in Complex Cyber-Physical Production Systems? 2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems Influences User’s Performance in Production Environments" - HCII 2016 page 4 Domain expertise Personality states & traits Trust, Self-efficacy, Motivation, … Uncertainty, randomness Non-linear interactions Disruptions & Seasonal changes Feedback loops, … Interface Design Visual complexity Information visualization Decision Support, … Efficiency, Effectivity, Profit, Quality, Satisfaction Of Workers and Customers USERSYSTEM INTERFACE PERFORMANCE 
    5. 5. RWTH Aachen University, Human-Computer Interaction Center How can Human and Interface factors be investigated? Business Simulation Games!  Convergence between field and laboratory study  Simplified & controllable (game-based) environments  Experimentally manipulate complexity and interface  Empirical methodology to quantify human performance – Identify and measure influencing personality factors – Identify and measure influencing interface factors – Build a formal model that explains performance  Side-effect: Usable for game-based learning (GBL) in education and professional trainings Test in the field (ecological validity) Controlled experiment in laboratory (internal validity) We are here page 5 2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems Influences User’s Performance in Production Environments" - HCII 2016
    6. 6. RWTH Aachen University, Human-Computer Interaction Center Business Simulation Games as a Research Lab for Understanding System, Interface, and User Factors  Interactive Business simulations – Forrester’s Beer Distribution Game, Goldratt’s Game – Quality Management Game  Several studies – Do System, Interface, and Human factors influence performance?  Questions addressed – Replication of similar studies? ✓ – Raises awareness for Quality Management? ✓ – Do human factors exist that explain performance? ✓ – Which human factors influence performance? ❓ – Do interface aspects influence performance? ❓ – Which interface aspects influence performance? ❓ – How can users be supported to make better decisions? ❓ P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems Influences User’s Performance in Production Environments" - HCII 2016 2016-07-20 page 6
    7. 7. RWTH Aachen University, Human-Computer Interaction Center Study with Business Simulation Game Can Interfaces support Users?  Research question: – Do interfaces influence player’s performance?  Interface optimizations based on user feedback – Better spatial layout (e.g., process-oriented) – Key Performance Indicators (e.g., stock level)  Method – Study (N=40) with old vs. new interface, surveys (new interface randomly present in 1st or 2nd round)  Results – Users preferred revised user interface – Higher profits and higher product quality w. new interface  Conclusion: – Good interfaces crucial for performance (V = 0.263, F1, 38 = 13.548, p = .001 < .05*) revised interface first interface page 7 2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems Influences User’s Performance in Production Environments" - HCII 2016
    8. 8. RWTH Aachen University, Human-Computer Interaction Center Follow up studies Focus on singular decisions, focus on specific elements  Narrow down factors that influence decision quality and decision speed  Focus on single decisions in context of material disposition  Research Questions – Which factors explain performance – Quantify costs of the user interface – Understand interrelationships between factors  Here: Influence of Decision Support Systems (DSS) Assets Drawbacks 2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems Influences User’s Performance in Production Environments" - HCII 2016 page 8
    9. 9. RWTH Aachen University, Human-Computer Interaction Center Decision Support Systems (DSS)  “Decision Support Systems aid in solving problems by automatizing the programmable part of a decision problem” [Gorry & Morton 1971]  Support for operative or strategic tasks  Support by – identifying relevant information – Compile data – Prepare data – Visualize data – Identify actions – Suggest actions – Support action execution 2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems Influences User’s Performance in Production Environments" - HCII 2016 page 9
    10. 10. RWTH Aachen University, Human-Computer Interaction Center Research Questions  Do Decision Support Systems influence operators’ performance? – Reaction Times – Accuracy  Do operators follow defective Decision Support Systems? – Reaction Times – Accuracy  Does the influence of a Decision Support System relate to the task complexity? 2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems Influences User’s Performance in Production Environments" - HCII 2016 page 10
    11. 11. RWTH Aachen University, Human-Computer Interaction Center Experimental Setup  Task in Material Disposition Context – Compare tables with two columns (Stock level and demand) – Press key  Order : If at least in one line a order is necessary (50%)  No Order : Otherwise (50%)  Length of the tables (within-subject) – 2, 6, or 12 lines (short, medium, long)  3 Decision Support Systems (within-subject) – None (baseline), Correct DSS, Defective DSS (wrong in 50% of the trials!)  Measured: – Reaction Times [ms] – Accuracy [%] 2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems Influences User’s Performance in Production Environments" - HCII 2016 page 11 Product In Stock Required Milk 136 98 Sugar 993 124 Flour 245 248 Butter 241 210
    12. 12. RWTH Aachen University, Human-Computer Interaction Center Description of the Sample  20 participants  Age 21 – 55 years 29.6±7.2 years  Gender 8 female, 12 male  Explanatory variables – Perceptual speed [n.s.] – Trust in Automation [n.s.] 2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems Influences User’s Performance in Production Environments" - HCII 2016 page 12
    13. 13. RWTH Aachen University, Human-Computer Interaction Center Results Baseline experiment (no Decision Support System) 2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems Influences User’s Performance in Production Environments" - HCII 2016 page 13
    14. 14. RWTH Aachen University, Human-Computer Interaction Center Baseline Experiment (no DSS) Factor Decision  Significant main effect of the decision Procurements faster, but less accurate than non-procurements  Interpretation: Linear processing of the presented data. Search terminated if procurement is required; search then terminated. 2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems Influences User’s Performance in Production Environments" - HCII 2016 page 14 75% 80% 85% 90% 95% 100% 0 1000 2000 3000 4000 5000 6000 7000 No procurement Procurement Accuracy[%] ReactionTimes[ms] Performance [ms] Accuracy [%] [V=.870,F2,17=56.883,p<.001,η2=.870]
    15. 15. RWTH Aachen University, Human-Computer Interaction Center Baseline experiment (no DSS): Factor Table Length  Significant effect of table length  Reaction times increase with length of table  Accuracy decreases with increasing length 2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems Influences User’s Performance in Production Environments" - HCII 2016 page 15 75% 80% 85% 90% 95% 100% 0 1000 2000 3000 4000 5000 6000 7000 8000 9000 2 lines 6 lines 12 lines Accuracy[%] ReactionTimes[ms] Performance [ms] Accuracy [%] [V=.966,F4,15=100.482,p<.001,η2=.966]
    16. 16. RWTH Aachen University, Human-Computer Interaction Center Results Effect of the Decision Support Systems 2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems Influences User’s Performance in Production Environments" - HCII 2016 page 16
    17. 17. RWTH Aachen University, Human-Computer Interaction Center Effect of a Decision Support System of Effectivity and Efficiency  Correct DSS (compared to baseline) – Reduced Reaction Times – Increased Accuracy  Defective DSS (compared to baseline) – No significant difference – Trend to reduced reaction times – Trend to decreased accuracy ⇒ Closer investigation 2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems Influences User’s Performance in Production Environments" - HCII 2016 page 17 75% 80% 85% 90% 95% 100% 0 1000 2000 3000 4000 5000 6000 Correct DSS no DSS (baseline) Defective DSS Accuracy[%] Performance[ms] Performance [ms] Accuracy [%] [V=.681,F4,15=8.006,p<.001,η2=.681]
    18. 18. RWTH Aachen University, Human-Computer Interaction Center Effect of Decision Support Systems on Reaction Times Closer examining the influence of the Number of Lines  Correct Decision Support System – Small effect for low number of lines – Larger effect for more lines  Defective Decision Support System – Small effect for low number of lines – Larger effect for more lines 2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems Influences User’s Performance in Production Environments" - HCII 2016 page 18 0 1000 2000 3000 4000 5000 6000 7000 8000 9000 Correct DSS no DSS (baseline) Defective DSS ReactionTimes[ms] 2 lines 6 lines 12 lines [V=.718,F2,17=3.495,p<.001,η2=.718]
    19. 19. RWTH Aachen University, Human-Computer Interaction Center Effect of Decision Support Systems on Accuracy Closer examining the influence of the Number of Lines  Correct Decision Support System – Positive influence on accuracy for all three conditions – Highest increase for 12 lines  Defective Decision Support System – Negative influence for all three conditions – Strongest decrease for 12 lines 2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems Influences User’s Performance in Production Environments" - HCII 2016 page 19 75% 80% 85% 90% 95% 100% Correct DSS no DSS (baseline) Defective DSS Accuracy[%] 2 lines 6 lines 12 lines [V=.718,F2,17=3.495,p<.001,η2=.718]
    20. 20. RWTH Aachen University, Human-Computer Interaction Center Research Questions & Answers  Do Decision Support Systems influence operators’ performance? – Correct Decision Support System reduce Reaction Times – Correct Decision Support Systems increase Accuracy  Do operators follow defective Decision Support Systems? – Limited influence on Reaction Times – Defective Decision Support Systems still obeyed and lead to lower accuracy!  Does the influence of a Decision Support System relate to the task complexity? – Effects emerge only for longer tables  Next steps – Larger sample (influence of Perceptual Speed, Trust in Automation) – Validate in Context (Business Simulation Games) 2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems Influences User’s Performance in Production Environments" - HCII 2016 page 20
    21. 21. RWTH Aachen University, Human-Computer Interaction Center Thank you four your attention! Summary  Industrial Internet leads to increased information complexity  Supporting the Human-in-the-Loop gains importance  Business Simulation Games as a Research Tool  Decision Support Systems crucial for performance – Operators easily deflected by defective DSS – Effects only emerge for complex conditions Dipl.-Inform. Philipp Brauner Human-Computer Interaction Center Chair for Communication Science RWTH Aachen University, Germany eMail: brauner@comm.rwth-aachen.de page 21 2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems Influences User’s Performance in Production Environments" - HCII 2016 Funded by the German Research Foundation (DFG) within the Cluster of Excellence “Integrated Production Technology for High Wage Countries” (EXC 128).
    22. 22. RWTH Aachen University, Human-Computer Interaction Center  Brauner P, Runge S, Groten M, et al (2013) Human Factors in Supply Chain Management – Decision making in complex logistic scenarios. In: Yamamoto S (ed) Proceedings of the 15th HCI International 2013, Part III, LNCS 8018. Springer-Verlag Berlin Heidelberg, Las Vegas, Nevada, USA, pp 423–432  Brauner P (2014) Understanding Human Factors in Supply Chains and Quality Management by Using Business Simulations. In: Brecher C, Wesch- Potente C (eds) Proceedings of the Conference of the Cluster Of Excellence “Integrative Production Technology For High Wage Countries” 2014/1, 1st edn. Apprimus Verlag, Aachen, Germany, Aachen, Germany, pp 387–396  Hering N, Meißner J, Runge S, Brauner P (2014) Exzellenzcluster: Was bestimmt die Performance meiner Supply-Chain? – Eine Untersuchung technischer und menschlicher Einflussfaktoren im Hinblick auf die Effizienz von Lieferketten. Unternehmen der Zukunft - Zeitschrift für Betriebsorganisation und Unternehmensentwicklung 27–28.  Philipsen R, Brauner P, Stiller S, et al (2014a) The role of Human Factors in Production Networks and Quality Management. – How can modern ERP system support decision makers? First International Conference, HCIB 2014, Held as Part of HCI International 2014, Heraklion, Crete, Greece, June 22-27, 2014. Proceedings, LNCS 8527. Springer Berlin Heidelberg, pp 80–91  Philipsen R, Brauner P, Stiller S, et al (2014b) Understanding and Supporting Decision Makers in Quality Management of Production Networks. Advances in the Ergonomics in Manufacturing. Managing the Enterprise of the Future 2014 : Proceedings of the 5th International Conference on Applied Human Factors and Ergonomics, AHFE 2014. CRC Press, Boca Raton, pp 94–105  Stiller S, Falk B, Philipsen R, et al (2014) A Game-based Approach to Understand Human Factors in Supply Chains and Quality Management. Procedia CIRP 20:67–73. doi: 10.1016/j.procir.2014.05.033  Brauner P, Ziefle M (2015) Human Factors in Production Systems – Motives, Methods and Beyond. In: Brecher C (ed) Advances in Production Technology. Springer International Publishing, pp 187–199  Mittelstädt V, Brauner P, Blum M, Ziefle M (2015) On the visual design of ERP systems – The role of information complexity, presentation and human factors. 6th International Conference on Applied Human Factors and Ergonomics and the Affiliated Conferences, AHFE 2015. pp 270–277  Calero Valdez A, Brauner P, Schaar AK, et al (2015) Reducing Complexity with simplicity - Usability Methods for Industry 4.0. 19thTriennial Congress of the International Ergonomics Association (IEA 2015).  Ziefle M, Brauner P, Speicher F (2015) Effects of data presentation and perceptual speed on speed and accuracy in table reading for inventory control. Occupational Ergonomics 12:119–129. doi: 10.3233/OER-150229  Brauner, P., Philipsen, R., Fels, A., Fuhrmann, M., Ngo, H., Stiller, S., Schmitt, R., Ziefle, M.: A Game-Based Approach to Meet the Challenges of Decision Processes in Ramp-Up Management. Quality Management Journal. 23, 55–69 (2016).  Calero Valdez, A., Brauner, P., Ziefle, M.: Preparing Production Systems for the Internet of Things The Potential of Socio-Technical Approaches in Dealing with Complexity. In: Dimitrov, D. and Oosthuizen, T. (eds.) Proceedings of the 6th International Conference on Competitive Manufacturing 2016 (COMA ’16). pp. 483–487. CIRP, Stellenbosch, South Africa (2016).  Brauner P, Ziefle M. How to train employees, identify task-relevant human factors, and improve software systems with Business Simulation Games. Procedings of the International Conference on Competitive Manufacturing 2016, COMA ’16. Stellenbosch, SA; 2016. p. 541–6.  Brauner, P., Calero Valdez, A., Philipsen, R., Ziefle, M.: Defective Still Deflective – How Correctness of Decision Support Systems Influences User’s Performance in Production Environments. Proceedings of the Human-Computer Interaction International 2016. (in press)  Brauner, P., Philipsen, R., Ziefle, M.: Projecting Efficacy and Use of Business Simulation Games in the Production Domain using Technology Acceptance Models. Proceedings of the Applied Human Factors and Ergonomics Conference (AHFE 2016). (in press)  Brauner, P., Philipsen, R., Calero Valdez, A., Ziefle, M.: On Studying Human Factors in Complex Cyber-Physical Systems, Workshop HFIDSS 2016, Mensch &Computer 2016 (in press) Publications page 22 2016-07-20 P. Brauner et al. – "Defective Still Deflective – How Correctness of Decision Support Systems Influences User’s Performance in Production Environments" - HCII 2016

    ×