This study evaluated a mobile diabetes management platform used by 8 pediatric patients over 2 months. The platform combined mobile tracking of health metrics, social networking, and gaming. Key findings from analysis of usage data, comments, and surveys include: 1) Users regularly tracked health data and engaged with social features, indicating the platform successfully encouraged engagement in self-management. 2) Usage patterns stabilized over time and aligned with best practices of pre- and post-meal tracking, though full alignment with clinical guidelines was not achieved. 3) Preliminary results suggest the platform has potential to effectively leverage mobile and social features to support pediatric self-management and provide useful insights.

1. An Evaluation Framework and A Pilot Study of a Mobile Platform for Diabetes Self-
Management: Insights from Pediatric Users
Rema Padman, Sravani Jaladi, Sean Kim, Saumitra Kumar, Philip Orbeta, Kate Rudolph, Tony Tran
The H. John Heinz III College, Carnegie Mellon University, Pittsburgh, PA, USA
Abstract
According to WHO, pediatric diabetes is a rising global public
health problem, with increasing impact on developing nations.
This study summarizes a multidimensional, scalable pilot
evaluation of a diabetes self-management platform combining
mobile technology with social networking to capture four key
metrics of Type 1 diabetes self-management, associated social
interactions, and gaming features providing targeted feedback
to 8 pediatric users. Based on their 2-month interaction with the
application, we analyze click-stream data from social
interactions, key health metrics, text comments, and usability
and satisfaction surveys to evaluate engagement with the
platform and effectiveness in controlling blood glucose using a
product-process-program framework. Our preliminary results
indicate that this framework was successful in demonstrating
the potential of the mobile health platform to effectively
leverage the growing use of mobile applications and social
media to present a unique benefit that engaged pediatric users
and provided useful insights for self-health management.
Keywords:
Pediatric diabetes, blood glucose self-management, mobile
applications, social media, gaming mechanics, product-process-
program evaluation.
Introduction
According to the World Health Organization, earlier onset of
Type 1 diabetes and increasing incidence of Type 2 diabetes in
the pediatric population worldwide require effective
interventions to be designed and implemented to counteract the
long term negative health outcomes and high costs associated
with the disease [1]. Market growth for mobile applications to
facilitate chronic disease management has resulted in
increasingly innovative solutions, including social media and
gaming components that are attractive for the pediatric
population, to increase compliance with best practices and
engagement with disease self-management [2]. Recognizing the
cost savings potential and opportunity for improvements in
health status for diabetes patients through wide spread adoption
and use of a diabetes self-management mobile application, a
mobile health start-up in Western Pennsylvania, named PHRQL
(Personal Health Record for Quality of Life), has developed a
unique application for smart phones that offers an innovative
platform for recording and capturing essential diabetes self-
management information while also continually engaging users
through social networking and gaming components. Eight
pediatric users were enrolled in a pilot study in Fall 2011 to
evaluate this platform along three key dimensions of product,
process, and program, to assess and understand the usage,
effectiveness and value of the platform for diabetes self
management.
Background
Due to the increased prevalence of diabetes and chronic
diseases as a whole, there is an ongoing shift toward a patient-
centered approach that enables patients to become more active
in their treatments and allows clinicians to tailor interventions
according to the patients’ needs and preferences [3, 4]. A
byproduct of patient-centered care is self-management of
chronic conditions. Type 1 diabetes self-management is an
approach to diabetes care where a patient monitors and
maintains their diabetes through blood glucose checks, insulin
therapy, a structured diet plan, and an activity regime. In
addition, self-management allows for greater peer support,
which has been linked to better health outcomes [5, 6]. For
pediatric Type 1 diabetes patients, this can be particularly
important as parents cannot always monitor their children’s
blood glucose levels.
Despite the growing trend towards diabetes self-management,
one of the biggest challenges is the lack of communication
between health care providers and patients in the outpatient
setting. Lack of customization of information, difficulty in
understanding it, and incomplete and inaccurate data
representation are broader challenges for all users [2, 3]. In
addition, traditional self-management tools do not easily fit into
the lifestyle of children and, as a result, compliance with
diabetic guidelines and consistent self-management suffer.
The need for flexible, easily accessible, and effective chronic
disease management support has given rise to rapid
development of diverse, mobile-based health applications.
According to recent reports, there are more than 250,000 mobile
health or mhealth applications available for the iPhone, more
than 30,000 for Android phones, and several thousand for
Blackberry phones [2]. The primary motivations for growth in
the mhealth applications market are the growth of mobile
phones and mobile internet access, greater patient participation,
quick and easy access to health care professionals, the
opportunity for better health outcomes and the cost-effective
nature of mobile health applications [2].
Technology, Study Details and Data
The app in this study is a mobile application currently designed
to help in the management of Type 1 diabetes in children and
young adults [7]. As shown in Figure 1, it allows users to record
key metrics per day such as blood glucose, food intake,
exercise/activity levels and insulin therapy. The Drag and
Drop™ feature makes it easier for children to quickly record
MEDINFO 2013
C.U. Lehmann et al. (Eds.)
© 2013 IMIA and IOS Press.
This article is published online with Open Access by IOS Press and distributed under the terms
of the Creative Commons Attribution Non-Commercial License.
doi:10.3233/978-1-61499-289-9-333
333

2. this information. The data is transferred to a cloud-based data
repository and the children can instantly view and track their
overall progress. To maintain high engagement among pediatric
users, the app has integrated game mechanics and social media.
Users can “friend” other diabetic users and together, they
compete to earn special badges to highlight their
accomplishments. A key feature is to enables a user to Share
and Compare™ his or her results. In addition, the user is able to
comment on other users’ profiles.
Figure 1 - Screenshots of the application: data entry (check-in),
gaming, social networking
The pilot study began in September 2011, concluded in
December 2011, and was limited to the Android platform. Eight
users and their parents were recruited, with informed consent
and disclosures about how the data will be used in analyzing
trends and patterns of usage, from the Type 1 pediatric diabetes
population around Western Pennsylvania, ranging in age from
10 years to 18 years. The mobile application was provided free
of charge, with limited instructions about its use in order to
observe the degree of intuitiveness of the application. User data
was stored on a cloud-based server and downloaded for analysis
on a weekly basis. In particular, three specific types of data
were used to evaluate the effectiveness of the application:
Health metrics data – “Check-in” data came from users keeping
track of various pieces of health information over time. This
was logged as one of four types: Activity, Carbs, Insulin and
Glucose. The data type and its corresponding value (with
appropriate units -mg/dL, g, or number of hours), the user’s ID
and the date and time of check-in were also recorded.
Comments data – The application had a feature that allowed
users to add other users to a private network through which a
user could keep track of the health progress of his or her
friends. As a supplement to this feature, a user could comment
on a friend’s check-in data, depicted in a plot of the different
data types over time, to potentially motivate the friend to
improve adherence to self-management guidelines. The
application tracked users’ interactions with each other, number
of different check-in data points on which different users
interacted, and text of the actual comments exchanged. Click-
stream data – Logs were maintained of how users interacted
with the application over time, recording the features users
tended to utilize, how much time users spent using specific
features, and the time of day when specific features were used.
Methods
Figure 2 - Evaluation framework
We applied a three-pronged evaluation framework, as shown in
Figure 2 which has been used successfully in other domains [8].
Product evaluation assessed the usability and functionality of
the application via benchmarking and technology assessment.
The evaluation also utilized surveys and conducted an analysis
of the user comments to assess the application’s usability and
functionality expressed via these comments. The benchmark
analysis focused on the features of the product and how well
they were aligned with the requirements for diabetes self-
management, particularly with the needs of children. These
needs include but are not limited to: easy, direct and simple data
capture, results reporting, usability and friendliness, and peer
support. A user satisfaction survey was created based on
usability principles and tailored to pediatric patients to gauge
usability and user friendliness of the application, but could not
be executed due to privacy and logistical challenges.
Process evaluation mapped the current and desired process
models underlying the use of PHRQL and measured impact of
the application on the self-management habits of a user, while
also developing a "best-practice" use model for future users.
The process evaluation then compared user behaviors with best
practice models. This evaluation approach had two distinct
segments. The first was the creation of a best practice process
model for diabetes self-management. Drawing on national
standards and direct interviews with diabetes educators, the best
practice process model shows the optimal self-management
pathway with the highest potential for improvement in diabetes
management and health outcomes. The second component
analyzed how users in the pilot study were interacting with and
using the application via several different approaches such as
visualizing these interactions and analysis using Markov Chain
model and statistical summaries. Results from the analyses were
compared to best practice model to identify useful feedback and
suggestions to improve patient self-management habits.
Program evaluation assessed the overall value of the
application in controlling diabetes along two different
dimensions - user engagement and effectiveness at lowering
blood glucose variability. User data was aggregated and
analyzed to understand usage patterns, engagement, and
adherence to best practices. We analyzed click-stream data of
user interactions with the application, their documentations of
key metrics and sharing of this information with peers via social
networking and gaming using descriptive statistics, and
multidimensional scaling and annealing methods.
Results
Product Evaluation
Our benchmarking analysis showed that the app has robust data
capture and user engagement features, aligning well with the
target market (i.e. diabetic children) [7]. Data capture and
results reporting are simple and almost instantaneous. The Drag
and Drop™ data entry made it easier for users to register their
health information. In addition, the Share and Compare™
feature enabled users to track other users’ progress. Together,
these two features generated greater opportunities for user
engagement because it was easier to enter accurate data, and
compare with peer users. The analysis also showed that
discussions about functionality were limited, but health
information was exchanged frequently, and the comments were,
in fact, facilitating user engagement through peer support. Most
of the users utilized the comments feature to notify their friends
on what they ate, blood glucose level, mood, and the activities
they participated in, as shown in Figure 3. Overall, these
comments were exchanged regularly; it was clear that the users
wanted others to know of their progress and receive feedback,
R. Padman et al. / An Evaluation Framework and a Pilot Study of a Mobile Platform for Diabetes Self-Management334

3. so users actively used the application because they were
anticipating a response from one of their friends. This evidence
of peer support is encouraging given that studies have shown
that peer supported interventions can “improve patients’ health
behavior, metabolic control, and quality of life [4, 5].”
Figure 3 - Sample of comments exchanged by users
Process Evaluation
The goal of the process evaluation was to provide insights into
the impact of the application on the self-management process
for users and to show a potential positive impact on user’s
compliance with diabetes self-management standards. The
management of Type 1 diabetes on a daily basis varies based on
the type of insulin administered. Insulin therapy is classified
based on the duration of action into rapid acting, short acting,
and long acting insulin [3, 4]. Daily management of the disease
varies based on the patient (newly diagnosed vs. established
patient). Research has shown that newly diagnosed patients
tend to have blood glucose levels above 250mg/dl and are
therefore instructed to take an additional dose of insulin in the
night before sleeping which a normal patient can forgo [3, 4].
The process models were developed to incorporate these
standards along with insights from diabetes educators in order
to create a best practice standard against which to compare
users' actual interactions with the application. Figure 4 shows
the best practice process models for diabetes self-management
for a patient on fast/rapid acting insulin. The nodes in the model
were color coded for each type of check-in. Blue blocks
represent blood glucose check-ins. Green represents an insulin
check-in. Yellow denotes a carb check-in and a red block
indicates that an activity check-in has occurred. Additional
process models developed for the other types of insulin therapy
are summarized in [7].
Figure 4- Fast/Rapid acting insulin process model
Visualization of the user interaction data (Figure 5) through
check-ins (blood glucose, activity, carbs, and insulin) was
developed on the same scale of the best practice model i.e.
management of the disease over a day. Visualization of user’s
interactions with the app showed that patients were developing
consistent interaction habits and patterns with the application
toward the end of our study period, suggesting that check-in
frequency had stabilized. It was encouraging to observe that the
app usage had not decreased or ceased, and instead had become
an important part of the user’s self-management routine.
Figure 5- Sample user interaction process model
However, our analysis also showed that usage was not always
in accordance with best practices but there was evidence that
users were capturing important aspects of the clinical
management guideline, specifically their pre and post prandial
blood glucose levels. The process model (Figure 6) shows the
check-in data for all eight users for days 1, 10, 20 and 40 that
have been stacked on top of each other to create a single,
unified visualization. The x-axis depicts the hour of day when
the check-in occurred and y-axis indicates each user's check-in
with his/her own color code. We observed that from midnight to
6AM, user 5 (brown) has a number of check-in activities, and a
few by users 1, 6 and 8 as well. Over the rest of the day, users 1,
5 and 6 are consistent in checking in their data on all key
measures around breakfast, lunch and dinner hours, but the
remaining users have sporadic and limited data entry. These
techniques thus provided opportunities for gleaning insights
into the habits of users, such as clustering of check-in data
around meal times, and increasing check-in compliance rates
over time.
Figure 6- Process model depicting users' interactions with the
application
In addition to the data visualization techniques, a Markov Chain
analysis of the check-in data assessed navigational pathways
and popular check-in patterns for the application across all users
[9]. Figure 7 shows the transition probability matrix on a graph.
Each circle represents a distinct check-in type and the size of
the circle is relative to the frequency of its use. The arcs
represent the direction of the transition probability. For example
the probability that an Activity check-in is followed by a
Glucose check-in is 85%, while the probability that a Glucose
check-in follows an Activity check-in is 2%, and the probability
of a Carbs check-in following a Glucose check-in is 47%. Bold
arcs represent probabilities that are greater than 50%.
Transitions from one state to the same state, for example a
Glucose check in followed by another Glucose check-in, can be
R. Padman et al. / An Evaluation Framework and a Pilot Study of a Mobile Platform for Diabetes Self-Management 335

4. calculated by subtracting all the other transitional probabilities
for the initial check-in from 1.
Key:
Figure 7 - Graph of transition probability matrix
The initial probability distribution of states (Glucose, Activity,
Carbs, Insulin) for the first check-in is shown in Table 1 with
Glucose having the highest probability of being the first check-
in, and all other types with a low probability of occurrence. The
Markov model converged in 6 steps, likely due to the small
number of check-in states and the high frequency of Glucose
check-in. It should also be noted that the probability of activity
check-in fell significantly as the number of check-ins increased.
This suggests that the Activity check-in will mostly likely be
used early on or it will not be used at all. Finally, once steady
state was reached, we observed that the probability of having a
Carbs or Insulin check-in is about the same, shown in Table 2.
Table 1 - Initial probability vector
Table 2 - Markov chain probabilities
The Markov Chain analysis was also used to identify high
frequency check-in patterns [7]. The most popular sequential
feature combination was Glucose, Carbs, Insulin, Glucose
(GCIG) pattern. This is not the optimal sequence identified in
the best practices process model which was GICG pattern, but it
does show that patients are capturing their pre- and post-
prandial Blood Glucose levels which are vital for successful
diabetes self-management.
Program Evaluation
The program evaluation is intended to measure how successful
the application is in enabling its users to better manage their
diabetes, by looking at both the engagement of its users and the
effectiveness of the application in improving health outcomes.
We measured engagement by investigating utilization patterns
of the different features over a period of time. Effectiveness was
measured by observing variability and trends in health metrics
checked in over a period of time, with special attention given to
users’ blood glucose levels. From the various analyses
conducted, it was evident that the application is very robust in
collecting different types of data ranging from health metrics to
social interactions and user activity.
Health Metrics
In order to get a better sense of user engagement with the
application, the health metrics data was analyzed. The study
observed the total number check-ins per day and per hour
within a day across all users to identify patterns in check-in
activity with blood glucose, insulin, carbohydrates, and activity
measurements. Adherence to best practice check-in times was
studied in the process evaluation. The frequency of check-ins
by users over the course of the study was explored to see
whether there was a significant increase or decrease in the user
engagement with the application. Figure 8 indicates steady use.
Figure 8-User Engagement over the study period
To analyze the effectiveness of the app, summary statistics with
variance in the four different types of check-ins measured
consistency in the values each user entered, implying that
greater variability suggested less successful diabetes
management. Time series modeling of users' blood glucose
levels over the span of the study tested correlation of blood
glucose variability with check-in data aggregated by day,
month, and time period. Given the study's limited duration and
number of participants, a statistically significant conclusion
could not be reached, but positive trends were observed.
Comments Data
The comments data indicated which users interacted with each
other and their usage frequency of the Share and Compare™
feature. A graphical depiction of the structure of the social
network used two methods: multi-dimensional scaling (MDS)
which shows users with similar features (i.e., the number of
connections) in closer proximity with each other, and annealing
which optimized the shortest path between all nodes and
ultimately put nodes with the most connections at the center [8].
Connections between the nodes were weighted based on the
number of interactions between each node to signify how
strongly each pair of nodes was related.
Figure 9-Social network using annealing at mid-point of study
Glucose Activity Carbs Insulin
R. Padman et al. / An Evaluation Framework and a Pilot Study of a Mobile Platform for Diabetes Self-Management336

5. Figure 9, with 2-4 connections per user node, all nodes within
reasonable proximity of each other, and users 37 and 42 as most
connected, illustrates the network based on user comments from
the initial half of the study period. Figure 10, with 3-5
connections per node and user 38 as the most connected at the
center by the end of the study, depicts the major changes in the
interactions amongst the users over a short duration. This
analysis captured which users were most connected to others as
well as users who were most active in utilizing the commenting
feature. Both of these criteria can be compared to health metrics
data in order to measure the effectiveness of social networking
in promoting adherence to the program. However, due to
limited data, this comparison did not produce useful insights.
Figure 10-Social network using annealing at end of the study
A qualitative analysis of user comments was conducted to
observe the number of different words, word frequency and
occurrence, sentence length, complexity, and readability of the
comment text. 180 different words that included 'good', 'high',
'low', and 'A1c' provide some indications of the content of the
exchanges. This qualitative analysis provided a better idea of
how effective different types of relationships and messages
were in encouraging users to adhere to the application.
Click-stream Data
Finally, click-stream data was used to tabulate how often the
different subsets of all application features were used in a given
time period (see Figure 11 for user 38). This type of analysis is
useful for identifying application features that correlate with
successful diabetes self-management. In addition, utilization
patterns of application features can help determine the features
that should be enhanced in future developments. Information
garnered from this analysis can also be used to understand if
users were adhering to best practices for entering health metrics
data and if features like commenting or gaming mechanisms
were contributing to increased user engagement.
Figure 11- Feature usage breakdown for user 38
Conclusions
This study evaluated a mobile health app along three
dimensions – product, process, and program. Product evaluation
examined the functionality and usability of the app and
compared it with other similar applications in the market. This
evaluation identified game mechanics and social media features
as key factors driving user engagement. Process evaluation
examined the impact of the app on users’ diabetes self-
management habits, detecting high variability in users'
interactions with the application and a lack of compliance with
best practices but a reassuring trend towards better self-
management habits over the duration of the study. Program
evaluation conducted data analytics on different types of log
data to understand the overall value of the app in controlling
diabetes and engaging users. Due to the limited number of study
participants, program evaluation could not conclusively
demonstrate that app usage decreased users' blood glucose
levels. However, positive trends were observed in user
engagement and blood glucose variability and increased
satisfaction with diabetes management. Ongoing studies with a
larger user population will use this framework to draw
actionable insights about the use of mobile health as an
intervention and self-management tool with pediatric as well as
adult users. Privacy and security issues, particularly important
in the pediatric setting, also need to be addressed using a
consistent and comprehensive approach.
Acknowledgements
We are grateful to the entire PHRQL team for the opportunity
to study this innovative platform.
References
[1] Aanstoot H-J, Anderson BJ, Daneman D, Danne T,
Donaghue K, Kaufman F, Réa RR, Uchigata Y. The Global
Burden of Youth Diabetes: Perspectives and Potential.
Pediatric Diabetes 2007: 8 (Suppl. 8): 1–44.
[2] Fox S. Mobile Health 2010. Pew Internet.
http://pewinternet.org/~/media//Files/Reports/2010/PIP_Mo
bile_Health_2010.pdf.
[3] Chomutare T, Fernandez-Luque, Arsand E, Hartvigsen G.
Features of mobile diabetes applications: review of the
literature and analysis of current applications compared
against evidence-based guidelines. J Med Internet Res.
2011; 13(3):e65.
[4] Sarkar U, Piette JD, Gonzales R, Lessler D, Chew LD,
Reilly B, Johnson J, Brunt M, Huang J, Regenstein M,
Schillinger D. Preferences for self-management support:
findings from a survey of diabetes patients in safety-net
health systems. Patient Educ Couns. 2008;70(1):102-110.
[5] Kyngas H, Rissanen M. Support as a crucial predictor of
good compliance of adolescents with a chronic disease. J
Clin Nurs. 2001;10(6):767-74.
[6] Fisher EB, Boothroyd RI, Coufal MM, Baumann LC,
Mbanya JC, Rotheram-Borus MJ, Sanquanprasit B,
Tanasugarn C. Peer support for self-management of
diabetes improved outcomes in international settings.
Health Affairs. 2012;31(1):130-9.
[7] Padman R, Tran T, Rudolph K, Jaladi S, Orbeta P, Kim S,
Kumar S. Diabetes Self-Management Mobile Application
Evaluation. 2011 Heinz College Working Paper, CMU.
[8] Long MH. Process and Product in ESL Program Evaluation.
TESOL Quarterly. 1984; 18(3): 409-425.
[9] Zheng K, Padman R, Johnson MP, Diamond HS. An
interface-driven analysis of user interactions with an
electronic health records system. J Am Med Inform Assoc.
2009, 16(2):228-37.
Address for correspondence:
Rema Padman, The H. John Heinz III College, Carnegie Mellon
University, Pittsburgh, PA 15213, USA, Email: rpadman@cmu.edu
R. Padman et al. / An Evaluation Framework and a Pilot Study of a Mobile Platform for Diabetes Self-Management 337