Video: https://youtu.be/GfuAGH3EGE8 Dissertation: https://www.publish0x.com/fajar-purnama-academics/development-of-a-lossy-online-mouse-tracking-method-for-capt-xjjlwxd?a=4oeEw0Yb0B&tid=slideshare

1. 1/66
Doctoral Thesis Open Defense
Development of a Lossy Online Mouse
Tracking Method for Capturing User
Interaction with Web Browser Content
Fajar Purnama D3 1
1
Human Interface Cyber Communication Laboratory (Usagawa
Lab), Computer Science and Electrical Engineering, Kumamoto
University
fajar@hicc.cs.kumamoto-u.ac.jp
July 28, 2020

2. 2/66
Doctoral Thesis Open Defense
Chapter 1 Introduction
Chapter 1 Introduction

3. 3/66
Doctoral Thesis Open Defense
Chapter 1 Introduction
Background
Due to Covid-19 pandemic, all activities are online.

4. 4/66
Doctoral Thesis Open Defense
Chapter 1 Introduction
Problem
Problem 1: How to monitor/follow these activities online?

5. 5/66
Doctoral Thesis Open Defense
Chapter 1 Introduction
Problem
Can conventional web logs / educational data only monitor
complicated online activities? Answer: Cannot

6. 6/66
Doctoral Thesis Open Defense
Chapter 1 Introduction
Problem
Human interface that can generate deeper logs? (Solution)
conventional web logs / educational data +

7. 7/66
Doctoral Thesis Open Defense
Chapter 1 Introduction
Problem
Why eye tracking cannot be implemented today?
Although the best monitoring quality is to use every logging and tracking there
is, eye tracking cannot be implemented today.
Eye tracking is inconvenient:
Intrusive: see previous slide where an additional goggle is necessary.
Expensive: JPY 10,000 - 1,000,000.
Mouse tracking is convenient:
No additional hardware: in default available in every computer.
Non-intrusive: users will not be aware that mouse tracking is active.

8. 8/66
Doctoral Thesis Open Defense
Chapter 1 Introduction
Problem
Problem 2: mouse tracking applications mentioned in
academia does not suit public implementation.
Offline Mouse Tracking Application in Laboratories:
(1) Must be installed in each client and cursor position fixed to desktop.
(2) Data are stored in each client and must be retrieved individually.
Solution:
Online Mouse Tracking Client Application:
(1) Must be installed in each client but (2) data are sent to server.
Or
Online Mouse Tracking Server Application:
(1) Browser only / no additional installation and (2) data are sent to server.

9. 9/66
Doctoral Thesis Open Defense
Chapter 1 Introduction
Problem
Problem 3: Mouse Tracking Generates Big Data Rumors
Volume: a swipe from left to right can produced 1 MB of data.
Velocity: large data generated per second (data rate).
Veracity: transmission may result in data loss due to limited connectivity.
Variety: many types of mouse and other events.
Value: the data are valuable.
Rumors are not widely investigated: if mouse tracking generates huge
data, how huge?.

10. 10/66
Doctoral Thesis Open Defense
Chapter 1 Introduction
Objective
Objective: build application and implement.
Write an application that can capture mouse and
other browser human interfaces called mouse
tracking.
Make the mouse tracking application online that
can be implemented in public.
Investigate its data generation and resource usage.
Reduce its data generation and resource usage.

11. 11/66
Doctoral Thesis Open Defense
Chapter 1 Introduction
Objective
Objective: dig deeper into users’ online activities.

12. 12/66
Doctoral Thesis Open Defense
Chapter 2 Online Mouse Tracking
Chapter 2
New Online Mouse Tracking
Based on publication list, the publication that is related to this chapter is:
1* Implementation of real-time online mouse tracking on overseas quiz
session From server administrator point of view, Educational and
Information Technologies, Online First, Springer, March 2020.

13. 13/66
Doctoral Thesis Open Defense
Chapter 2 Online Mouse Tracking
System Overview

14. 14/66
Doctoral Thesis Open Defense
Chapter 2 Online Mouse Tracking
Mouse Tracking Desktop
Recording: clicks, movements, scrolls, zooms, keylogs, copies, and other events.

15. 15/66
Doctoral Thesis Open Defense
Chapter 2 Online Mouse Tracking
Touch Tracking Mobile
https://hicc.cs.kumamoto-u.ac.jp/ fajar/doctoral/mouse-tracking-keyboard-
mouse-scroll-zoom-demo-and-mobile.html

16. 16/66
Doctoral Thesis Open Defense
Chapter 2 Online Mouse Tracking
Online Mouse Tracking System

17. 17/66
Doctoral Thesis Open Defense
Chapter 2 Online Mouse Tracking
Privacy Policy

18. 18/66
Doctoral Thesis Open Defense
Chapter 2 Online Mouse Tracking
Implementation
Implementation

19. 19/66
Doctoral Thesis Open Defense
Chapter 2 Online Mouse Tracking
Implementation
Client Application: Tracks All Browser’s Activities
https://github.com/0fajarpurnama0/Real-Time-Online-Mouse-Tracking-

20. 20/66
Doctoral Thesis Open Defense
Chapter 2 Online Mouse Tracking
Implementation
Server Application: Tracks Only Website’s Activities
https://md.hicc.cs.kumamoto-u.ac.jp

21. 21/66
Doctoral Thesis Open Defense
Chapter 2 Online Mouse Tracking
Implementation
Moodle Plugin Install
https://github.com/0fajarpurnama0/Real-Time-Online-Mouse-Tracking-

22. 22/66
Doctoral Thesis Open Defense
Chapter 2 Online Mouse Tracking
Implementation
Mouse Tracking Quiz Session Between Mongolia and Japan

23. 23/66
Doctoral Thesis Open Defense
Chapter 2 Online Mouse Tracking
Investigation
Investigation

24. 24/66
Doctoral Thesis Open Defense
Chapter 2 Online Mouse Tracking
Investigation
Obtained Mouse Tracking Data
Data available: Mendeley repository titled ’Data for: Implementation of
Real-Time Online Mouse Tracking Case Study in a Small Online Quiz’.

25. 25/66
Doctoral Thesis Open Defense
Chapter 2 Online Mouse Tracking
Investigation
Rumor and Proof of Huge Data Generation
Log File Duration Students Size
Daily Pageview City Archive 2 month - 13 kB
Moodle Log and Grades 3h 30min 41 191 kB
Moodle Log Myanmmar 6 month 41 5 MB
Mouse Tracking Mongolia 3h 30min 41 122 MB
Mouse Tracking Afganistan 3h 42 114 MB
Nasa Server Log 1995 23 days - 153 MB
Open University Learning Analytics 1 year 32593 442 MB
HarvardX Person-Course 2013 1 year 301609 33.8 MB

26. 26/66
Doctoral Thesis Open Defense
Chapter 2 Online Mouse Tracking
Investigation
Students’ Events and Estimated Data Amount Generated

27. 27/66
Doctoral Thesis Open Defense
Chapter 3 Resource Saving Methods
Chapter 3
Resource Saving Methods
Based on publication list, the publication that are related to this chapter are:
1* Implementation of real-time online mouse tracking on overseas quiz
session From server administrator point of view, Educational and
Information Technologies, Online First, Springer, March 2020.
2* Using Realtime Online Mouse Tracking Based on Demand
Pre-processing for Lower Storage and Transmission Cost, Journal of Big
Data, Vol.7, no.27, pp.1-22, Springer, April, 2020.
3* Demonstration on Extending The Pageview Feature to Page Section
Based: Towards Identifying Reading Patterns of Users, ICSE, November,
2016.
.

28. 28/66
Doctoral Thesis Open Defense
Chapter 3 Resource Saving Methods
Existing Solutions
Redundant data reduction for example:
Long URL: https://www.publish0x.com/0fajarpurnama0/can-i-share-my-
work-after-copyright-transfer-xroyrle?a=4oeEw0Yb0B&tid=twitter
Into Short URL: https://t.co/Hl62T1llTV
Sampling rate (event / second) reduction for example delay capture
interval for 50 milliseconds = 20 max events per second.
Adaptive sampling: do not record during idle activities (eye tracking
record during idle activities).
Compression methods.

29. 29/66
Doctoral Thesis Open Defense
Chapter 3 Resource Saving Methods
Not Realtime vs Realtime in Mouse Tracking
Traditionally: accumulate > wait until session ends > send all. Realtime: send
events immediately on occurrence.

30. 30/66
Doctoral Thesis Open Defense
Chapter 3 Resource Saving Methods
Lossy Based on Demand Coordinate Compression
Default: full mouse tracking
log.
ROI: summarize coordinates
into areas.
Summarized: summarize
coordinates into amounts.
Events Summarized ROI Default
Amounts
Areas X
Coordinates X X

31. 31/66
Doctoral Thesis Open Defense
Chapter 3 Resource Saving Methods
Default Mouse Tracking Data Transmission
Send data immediately / real time on each coordinate passed:

32. 32/66
Doctoral Thesis Open Defense
Chapter 3 Resource Saving Methods
Region of Interest Mouse Tracking Data Transmission
Send data after leaving areas:

33. 33/66
Doctoral Thesis Open Defense
Chapter 3 Resource Saving Methods
Summarized Event Amounts of Mouse Tracking
Send data after closing webpage:

34. 34/66
Doctoral Thesis Open Defense
Chapter 3 Resource Saving Methods
Three Mouse Tracking Simulation

35. 35/66
Doctoral Thesis Open Defense
Chapter 3 Resource Saving Methods
Storage Reduction
Type Queries Data Size
Default Mouse Tracking 286510 ∼100 MB
ROI Mouse Tracking 1 28048 ∼7.7 MB
ROI Mouse Tracking 2 19061 ∼5.3 MB
ROI Mouse Tracking 3 17880 ∼4.9 MB
Summarized Event Amount 23 ∼16 kB

36. 36/66
Doctoral Thesis Open Defense
Chapter 3 Resource Saving Methods
Data Rate: Reduced > 50% storage and transmission

37. 37/66
Doctoral Thesis Open Defense
Chapter 3 Resource Saving Methods
CPU Usage: RP3 cannot handle the default MS

38. 38/66
Doctoral Thesis Open Defense
Chapter 3 Resource Saving Methods
RAM Usage: abnormally high for default mouse tracking

39. 39/66
Doctoral Thesis Open Defense
Chapter 4 The depth Levels of Logs Concept
Chapter 4
The depth Levels of Logs Concept
Based on form 3, the publication that are related to this chapter are:
1* Implementation of real-time online mouse tracking on overseas quiz
session From server administrator point of view, Educational and
Information Technologies, Online First, Springer, March 2020.
2* Using Realtime Online Mouse Tracking Based on Demand
Pre-processing for Lower Storage and Transmission Cost, Journal of Big
Data, Vol.7, no.27, pp.1-22, Springer, April, 2020.
3* Demonstration on Extending The Pageview Feature to Page Section
Based: Towards Identifying Reading Patterns of Users, ICSE, November,
2016.
.

40. 40/66
Doctoral Thesis Open Defense
Chapter 4 The depth Levels of Logs Concept
Conceptual Explaination: Log depth Level Concept

41. 41/66
Doctoral Thesis Open Defense
Chapter 4 The depth Levels of Logs Concept
Web Page / Course Content Level Log
Web Page / Course Content Level Log

42. 42/66
Doctoral Thesis Open Defense
Chapter 4 The depth Levels of Logs Concept
Web Page / Course Content Level Log
Mouse tracking is able to identify students leaving quiz
page (address problem 1: potential cheating).

43. 43/66
Doctoral Thesis Open Defense
Chapter 4 The depth Levels of Logs Concept
Web Page / Course Content Level Log
Online Examination Leave Detector (Possible Software)
https://hicc.cs.kumamoto-u.ac.jp/ fajar/doctoral/mouse-tracking/exam-leave-
detect-demo.html

44. 44/66
Doctoral Thesis Open Defense
Chapter 4 The depth Levels of Logs Concept
Area / Section Level Log
Area / Section Level Log

45. 45/66
Doctoral Thesis Open Defense
Chapter 4 The depth Levels of Logs Concept
Area / Section Level Log
Area Heatmaps are Able to Address Problem 1: Required
Reading Passage (Proof of Read)

46. 46/66
Doctoral Thesis Open Defense
Chapter 4 The depth Levels of Logs Concept
Area / Section Level Log
Area Heatmaps in Detail

47. 47/66
Doctoral Thesis Open Defense
Chapter 4 The depth Levels of Logs Concept
Area / Section Level Log
Terms of Services Read Detector (Possible Software)
https://hicc.cs.kumamoto-u.ac.jp/ fajar/doctoral/mouse-tracking/eproof-of-
read.html

48. 48/66
Doctoral Thesis Open Defense
Chapter 4 The depth Levels of Logs Concept
Coordinate / Point Level Log
Coordinate / point level log

49. 49/66
Doctoral Thesis Open Defense
Chapter 4 The depth Levels of Logs Concept
Coordinate / Point Level Log
Clicks, Left: Triangle, Middle: Square, Right: Pentagon.

50. 50/66
Doctoral Thesis Open Defense
conclusion and Future Work
Conclusion
Demonstrated mouse tracking as solution to problem 1 regarding to
monitoring demand for human-online interaction.

51. 51/66
Doctoral Thesis Open Defense
conclusion and Future Work
Conclusion
Answered problem 2 regarding public mouse tracking application by
building new real-time online mouse tracking application that does not require
additional application for the client and implemented.

52. 52/66
Doctoral Thesis Open Defense
conclusion and Future Work
Conclusion
Answered problem 3 regarding huge resource rumors by investigating
real-time online mouse tracking resource usage and reducing it over 50% by
summarizing coordinates into areas.

53. 53/66
Doctoral Thesis Open Defense
conclusion and Future Work
Thank you

54. 54/66
Doctoral Thesis Open Defense
Supplementary Materials
Almost All Online, Anywhere at Anytime, Face to Face

55. 55/66
Doctoral Thesis Open Defense
Supplementary Materials
Benefit and Significance
Mouse tracking is one of the missing keys of
anything fully online.
The code is open source that can freely
implemented and developed by other parties.
The data and resource usage investigation result
can help parties to plan in implementing mouse
tracking.
Reduced data and resource usage can help those
with limited connectivity.

56. 56/66
Doctoral Thesis Open Defense
Supplementary Materials
Obtained Moodle Log Data

57. 57/66
Doctoral Thesis Open Defense
Supplementary Materials
Obtained Grades

58. 58/66
Doctoral Thesis Open Defense
Supplementary Materials
Truth: Users Actually Have Full Control but Unaware

59. 59/66
Doctoral Thesis Open Defense
Supplementary Materials
Rumor and Proof of Huge Data Generation
Moodle Log Myanmar: (CP: monmon@hicc.cs.kumamoto-u.ac.jp)
6 months of entire course with 40 students.
21098 rows of table, 5MB.
Can a swipe generate a megabyte of data?
Yes, if the width of a webpage from left to right is 1000 pixels, then a
swipe from left to right can generate at large 1000 rows of table.
Mouse Tracking Session Mongolia:
2 hours of quiz session with 22 students each hour.
286511 rows of table, 100 MB.
Mouse Tracking Session Afganistan:
2 hours of quiz session with 23 students each hour.
282601 rows of table, 92 MB.
Problem: Huge Storage and Transmission Cost

60. 60/66
Doctoral Thesis Open Defense
Supplementary Materials
Server Storage Side: Query Amount and Data Size
Default Mouse Tracking: 100 MB
ID Name Date 2019/3/01 Duration Left Click Right Click Middle Click Mouse X Mouse Y Scroll X Scroll Y
1 Student 1 11:06:39 13.674 false false false 0 0 0 0
2 Student 1 11:06:39 0.002 false false false 1197 317 0 0
... ... ... ... ... ... ... ... ... ... ...
286511 Student 23 14:44:09 0.002 false false false 1009 179 0 0
ROI Mouse Tracking: 5.4 MB
ID Name Date 2019/3/01 Duration (second) Area (x1,x2,y1,y2) Left Clicks Right Clicks Middle Clicks Mouse Moves Scrolls
1 Student 1 11:06:39 14.148 {"header":[0,1920,0,64]} 0 0 0 1 1
2 Student 1 11:06:40 1.179 {"quiz1":[529,1900,291,570]} 0 0 0 86 0
... ... ... ... ... ... ... ... ... ...
19062 Student 23 14:44:09 0.002 {"title":[16,1904,150,270]} 0 0 0 1 0
Whole Page Summarized Mouse Tracking: 16 kB
Name Date 2019/3/01 Duration (seconds) Left Clicks Right Clicks Middle Clicks Mouse Moves Scrolls
Student 1 14:12:29 41 3 0 0 629 114
Student 2 14:44:09 90 7 0 0 1176 137
... ... ... ... ... ... ... ...
Student 22 11:55:14 2188 157 5 0 20912 6626
Student 23 11:57:37 2236 323 0 0 17982 6930

61. 61/66
Doctoral Thesis Open Defense
Supplementary Materials
Client Side: Browser Speed

62. 62/66
Doctoral Thesis Open Defense
Supplementary Materials
Amount of mouse event logs
Most analyzers only mined up to this level logs: page view, quiz, grades, and
other activity attempts. Even in this level, there are logs that most analyzers
are not aware of:
Amounts of events:
name duration leftclick rightclick middleclick mousemove scroll highlight
average 1368.16 175.78 0.29 7.98 11004.73 4158.12 0.61
total 56094.53 7207.00 12.00 327.00 451194.00 170483.00 25.00
minimum 41.26 3.00 0.00 0.00 633.00 116.00 0.00
maximum 2748.21 1179.00 3.00 179.00 20724.00 9521.00 4.00
stdev 728.82 295.33 0.72 39.05 4977.88 2291.82 1.16

63. 63/66
Doctoral Thesis Open Defense
Supplementary Materials
heatmap 10 minutes interval

64. 64/66
Doctoral Thesis Open Defense
Supplementary Materials
Grade heatmap

65. 65/66
Doctoral Thesis Open Defense
Supplementary Materials
Why RP3? To Facilitate Those With Limited Connectivity
Hand Carry Data Collecting Through Questionnaire and Quiz Alike Using
Mini-computer Raspberry Pi, IMLF Proceeding, June, 2017.

66. 66/66
Doctoral Thesis Open Defense
Supplementary Materials
Regular Synchronization is Necessary
From 6 Incremental Synchronization Implementation on Survey using Hand Carry Server Raspberry Pi,
IEICE Tech Report, vol.117, no.65, pp.21-24, June, 2017. and 7* Rsync and Rdiff implementation
on Moodle’s backup and restore feature for course synchronization over the network, (TENSYMP),
pp.24-29, IEEE, Mei, 2016.