2. Current urban journey planners
UMAP’15 V. Codina, J. Mena, L. Oliva 2
Google Maps Moovit Citymapper
3. Transport mode
selection
Lack of preference learning from user's feedback
Only based on explicit transport preferences
Less transfersLess walking
But... limited personalization
UMAP’15 V. Codina, J. Mena, L. Oliva 3
Shorter routes
4. 2) Current context
To personalize suggested journey plans based on:
How: with context-aware recommendation strategies
Research goal
UMAP’15 V. Codina, J. Mena, L. Oliva 4
1) Past user ratings
5. Recommender engine decoupled from planner
Task: context-aware, user's satisfaction scoring
Role of our journey plan recommender
UMAP’15 V. Codina, J. Mena, L. Oliva 5
Journey plan
Target
context
Target
user preferences
Predicted score
UI
6. Journey plans are highly dynamic
Unique routes per user's request
Challenge: new item (early rater) problem
How to recommend items with no ratings at all?
Solution: Content-based Filtering
New item problem solved by exploiting item’s content
Journey plan recommendation domain
UMAP’15 V. Codina, J. Mena, L. Oliva 6
Collaborative Filtering not an option!
7. Idea: “show me more of the same kind of plans I liked”
Score predictions based on feature vector matching
Content-based approach to
journey plan recommendation
UMAP’15 V. Codina, J. Mena, L. Oliva 7
Item Profile
User profile
0.2 1 0.5 0 1
0.5 -0.7 0.2 0.8 -0.8
Journey plan
vector
Score =
User preference
vector
∈
∈
n = # attribute values
Derived via Stochastic Gradient Descent by solving
the regularized least squares problem over the training ratings
8. Originally we had these attributes per leg:
Transport mode (e.g. Walk, Tram, Bus, Taxi, Car, Bike, …)
Time (seconds)
Cost (€)
Physical effort (Kj)
Example:
Journey plan original data
UMAP’15 V. Codina, J. Mena, L. Oliva 8
Car Tram Walk
600s
3€
0kj
900s
1€
0kj
300s
0€
50kj
Journey plan Leg 1 Leg 2 Leg 3
9. Numeric attributes discretized into 5 equal intervals
E.g. Walk_Effort: [very low | low | medium| high | very high]
Discretization by soft cuts using a fuzzy-set approach
Journey plan feature vector representation
UMAP’15 V. Codina, J. Mena, L. Oliva 9
medium =0.5 high=0.5low= 0
150Kj
Interval values defined by mobility experts
225Kj 275Kj 425Kj
X = 350Kj
10. IDEA: user’s preferences can depend on context
Context matters for journey plan scoring
UMAP’15 V. Codina, J. Mena, L. Oliva 10
11. Context as additional dimension for prediction
3 main paradigms depending on how context is used
Classification of context-aware strategies
UMAP’15 V. Codina, J. Mena, L. Oliva 11
target
context
Prediction
model
in-context
training ratings
target
Item
Pre-filtering Post-filteringContextual modeling
Predicted score
12. Reduction-based approach that builds local user vectors
Using the ratings identified as relevant for the target context
The pre-filtering strategy:
Distributional Semantic Pre-Filtering (DSPF)
UMAP’15 V. Codina, J. Mena, L. Oliva 12
local
ratings
Ratings
filtering
Local user
modeling
≈
≠ Similarities based on the distributional
semantics of contextual conditions
in-context
training ratings
target context
local
user vectors
13. Method inspired on time-aware Matrix Factorization1
Content-Based linear model extended with:
Global contextual biases
e.g. “users tend to rate lower routes with long walks when hot”
Preference-specific contextual biases
e.g. “John rates higher bike routes when sunny”
The contextual modeling strategy:
Distributional Semantic Contextual Modeling (DSCM)
UMAP’15 V. Codina, J. Mena, L. Oliva 13
Distributional similarities
between conditions
[1] Y. Koren “Collaborative filtering with temporal dynamics” Commun. ACM, 2010.
14. Question: which context-aware strategy is better for
journey plan recommendation?
Two types of evaluation
Experiment 1: traditional offline evaluation
Experiment 2: user-centric online evaluation
Experimental Evaluation
UMAP’15 V. Codina, J. Mena, L. Oliva 14
Pre-filtering (DSPF)
Vs.
Contextual modeling (DSCM)
15. Offline evaluation
In-context journey plan rating acquisition
UMAP’15 V. Codina, J. Mena, L. Oliva 15
Suggested journey plans
Journey plan requests
Context settings panel:
• 2 user factors
• 8 urban factors
• ? = Unknown value
16. Contextually-tagged journey plan rating dataset in the
city of Barcelona (Spain)
Per-user data splitting
Error-based performance metric: RMSE
Offline evaluation: procedure
UMAP’15 V. Codina, J. Mena, L. Oliva 16
Users 68
Journey plans 1,628
Attribute values 69
Contextual Factors 10
Conditions 38
Ratings 3,256
17. Both context-aware strategies are better than baseline (Free)
But… differences between context-aware models are not significant (p-value = 0.25)
Offline evaluation: results
UMAP’15 V. Codina, J. Mena, L. Oliva 17
18. Participants used Android app under real contexts
Developed by several partners of the SUPERHUB project
User-centric online evaluation
UMAP’15 V. Codina, J. Mena, L. Oliva 18
Journey plan
request
User context
Top-4
recommendation
Rating
(used for model
fine-tuning)
19. A/B testing method and user study questions
User-centric online evaluation: setup
UMAP’15 V. Codina, J. Mena, L. Oliva 19
4 aspects evaluated:
• Top-n accuracy
• Ranking accuracy
• Context-awareness
• Overall satisfaction
Citizens of Barcelona
(Spain)
22. UMAP’15 V. Codina, J. Mena, L. Oliva 22
Context-Aware User Modeling Strategies
for Journey Plan Recommendation
QUESTIONS
Victor Codina
vcodina@bdigital.org
Hinweis der Redaktion
I'm going to present our paper ...
This is a joint work with Luis Oliva from Technical University of Catalonia and Jose Mena from Barcelona Digital tech center. My name is ...
Nowadays there are several free mobile apps that assist you during urban journey planning, helping you to find the shortest routes between two arbitrary points in a city by combining different transport modes, private and public ones.
Here I'm showing some popular examples that most of you probably know: Google Maps, Moovit, CityMapper.
A common strong point of theses apps is that they provide to users with real-time information about transit conditions and transport services.
However, a common limitation of these systems is their lack of user preference learning, which means they are not exploiting previous user interactions with the system to learn about their preferences and adapt journey plan suggestions to them
Commonly these apps allow users to personalize the generated routes by explicitly specifying a set of generic transport preferences for the current request.
These are for example the kind of options that one can choose using Moovit. I prefer routes with less walking. I want shorter routes. Or I want routes with less tranport mode interchanges.
This limitation of current journey planners is what motivated this work, whose research goal was to develop a journey planner able to generate journey plans recommendations based on user's feedback in the form of ratings, as shown in this screenshot, and adapted to the current context.
To accomplish this goal we decided to employ SoA context-aware user modeling and recommendation strategies, given their more than proved effectiveness in other domains, such Point Of Interest recommendation or movie and music domains.
Differently from other domains in our application the recommender engine is not the central component of the system. Instead, the recommender works as an independent module that is used by the journey planner to rank the set of generated candidate routes.
So, in our system the recommendation task was formulated as a journey plan scoring problem, or more generally as a rating prediction problem. So, given a candidate journey plan and target user and context, the recommender taks consisted of predicting the score given by the user to that plan under such contextual conditions.
A particularity of our recommendation domain was that journey plans, the items to recommend, where highly dynamic, in the sense that they were generated based on arbitrary start and destination points, which means they were mostly unique per request.
This implied that our recommender should be able to make predictions in new item conditions, where the target item to recommend has no ratings of other users associated.
So for this reason CF methods are not feasible in this particular task since they require ratings of different users on the same items to extract meaningful patterns from the data.
In such conditions only CB or Knowledge based strategies are feasible because they are able to exploit item's metadata in order to build user models and make recommendations.
So, in this work we used a content-based method as the basis of the context-aware strategies.
The main assumption of CB approaches is that users tend to like items with similar attributes to those he or she already liked in the past.
More formally, these approaches compute the suitability of an item for a user based on feature matching calculation.
To do such comparison it is necessary to encode user preferences and item descriptions in the same attribute space, commonly using a feature vector representation.
In the case of the item vector, positive values represent the set of attribute values that are relevant in the description of the item, and zero values the no presence of the attribute at all.
In the case of user vectors, values represents the user's interests, which can be positive, negative or neutral .
Once we have this feature representation of the target item and user, predictions can be generated by comparing the values of the two vectors. In this work we used the dot product, since it is one of the most popular methods.
User vectors can be derived from the past user rating by using several methods. Here we calculate them by solving the least squares problem using SGD.
So one can see that in CB recommendation the item vector construction is a very important step. Now I'm going to explain how we built the vectors from the original journey plan data.
To do so, we extracted several categorical attributes from the numeric data provided by the journey planner about each candidate journey plan, which included the following information:
- information about the time required to complete each leg composing the plan t
- information about the cost associated to each mode of transport mode used in the plan (in €)
– the physical effort required (if the plans include legs by foot or bike)
From these data we derived several discrete attributes describing the time and cost associated to each mode of transport separately, which we group by walk, bike, public and private and also global features like the total number of public interchanges. Each of these attributes was discretized into 5 equal intervals: from very low to very high values. For the value assignation we used a fuzzy-set method in order to accurately classify values close to the boundaries. The result of this process is weighted feature vector with values between [0,1] indicating the weight of the attribute representing the journey plan.
So far I have talked about how we generate context-free predictions, and now I’m going present the strategies we experimented with to incorporate context into the prediciton.
The main assumption of CARS is that user preferences can depend on context. For example…
CARS are commonly classified in three main paradigms depending on how they incorporate context: pre-filtering, post-filtering and contextual modeling. Pre-filtering methods use context only for selecting the rating data which is relevant to the target context, so they can be used in combination with any context-free recommendation method. This is also the case of post-filtering methods, but here context is exploited at the end of the process, to adjust context-free recommendations rather than data filtering. Finally, contextual modeling strategies are those approaches where context is incorporated into the estimation function as additional model parameters, giving rise to truly multi-dimensional prediction models
Although there are several context-aware recommendation approaches out there, in our evaluatio with only experimented with two state-of-the-art methods, one based on pre-filtering and another one based on contextual modeling. We selected them because of their superior rating prediction accuracy and scalability in domains with high context granularity (where contextual situations are defined by the conjunction of several contextual conditions) as in this case.
Now I will explain the main idea of each of the implemented methods:
The pre-filtering method consists of an adaptation of DSPF, a sophisticated reduction-based approach that builds a local prediction model for each target contextual situation based on the ratings identified as relevant for the given context. It is called distributional semantic because this method defines two context as similar if their composing condition have similar distributional semantics, in the sense of how they influence the user ratings.
This example illustrates how this strategy works:
1- Let's assume that our set of ratings is composed of ratings acquired in three different contexts, ratings provided while travelling with family, when sunny and in rainy conditions. Assuming that that the target context is sunny, and that family and sunny conditions have similar rating influencing patterns, the set of relevant ratings for that target context would be the ratings tagged with family and sunny.
2. Then, a local recommendation model is learned using the relevant subset of ratings. In particular, using our CB approach, a local user vector of preferences is learnt for each user, which will be used a posteriori to do the item-user matching in the given target context.
This approach consists of extending a linear CB prediction model by introducing global and user-specific parameters which model the influence of context to the user's preferences.
hat model the global effects of contextual conditions w.r.t. journey plan features
captures the global context variability of user preferences with respect to the item attributes. So for example, one of these parameters could represent that in hot conditions users tend to rate positively routes with short walking distances.
that model the context variability of user preferences
Our goal was to understand which stratregy is better in this particular recommendation task, so our evaluation consisted of a performance comparison between these two strategies: the pre-filtering and the contextual modeling strategy.
We did this comparison by means of two experiments: 1 using the traditional offline setting and 1 user-centric based on a A B testing method.
Once requested, the planner generated a set of alternative joruney plans using different tranport mode combinations
By clicking on each of the alternatives, users could see the plan details and vote the route under the active context .
In order to evaluate the strategies from a system-centric perspective we first needed a way to collect a set of in-context ratings for a variety of journey plans and contextual situations
To do so, we developed an ad-hoc web-based app in which users could request journey plans under imaginary contexts and rate them.
68 users participated in this experiment, which lasted one week. We collected more than 3000 ratings. As you can see the number of ratings collected doubles the number of rated journey plans. This is because, users were asked to rate the same plan in two different contextual situations. Next, I will explain this acquisition method in more detail.
We split the collected data set on training and test sets by using an all-but-n protocol, in which a portion of the ratings of each users are held for testing
and we measure the performance of the models in terms of their RMSE.
After one week, we collected around 3000 ratings and each user provided on average 50 ratings .
We split this data set on training and test in order to train the evaluated prediction models and measure their RMSE.
This bar char shows the results. DSPF corresponds to the results of the pre-filtering strategy, DSCM to the contextual modeling strategy and Free refers to the context-free method using a linear CB model.
As you can see both context-aware strategies clearly outperform the context-free baseline (the contextual modeling reduces RMSE by 14% and the prefiltering by 9%)
Comparing the context-aware strategies we can observe that the contextual modeling seems to have better performance although we cannot state that these differences are statistically significant. So this offline evaluation was not useful to identify a clear winner.
For this experiment users used the mobile app developed in the SUPERHUB project. Here you can see some screenshots of the UI trough which users could receive recommendations and rate them.
As I said previously, we also compared the performance of the proposed context-aware strategies using A/B testing. For this user-centric experiment we used a different set of users but of similar size, particularly 67 users completed the experiment.
This diagram shows the design of our experiment, in which users were randomly split in three groups of similar size, one group receiving recommendations from the pre-filtering strategy, another one using the contextual modeling and the other one using the context-free algorithm. Obviously users didn't know to which group were assigned to.
TODO: Improve this
Once completed the task, users answered 4 different questions on a five-point scale. Each of the questions was aimed to evaluate a different aspect of the recommendations: top-n accuracy, ranking accuracy, context-awareness, and the overall satisfaction with the system.
These bar charts show the mean score given by each group of users to the 4 questions. One can easily observe that the contextual modeling strategy, DSCM, is the one with higher perceived recommendation accuracy and context-awareness, which leads to a higher user's satisfaction according to the correlation analysis. And this case we found that the differences with respect the prefiltering strategy were statistically significant, validating thus its superior performance in this application.