Photo lineups play a significant role in the eyewitness identifica-tion process. This method is used to provide evidence in the prosecution and subsequent conviction of suspects. Unfortu-nately, there are many cases where lineups have led to the con-viction of an innocent suspect. One of the key factors affecting the incorrect identification of a suspect is the lack of lineup fair-ness, i.e. that the suspect differs significantly from all other candidates. Although the process of assembling fair lineup is both highly important and time-consuming, only a handful of tools are available to simplify the task. In these slides, we describe our work towards using recommend-er systems for the photo lineup assembling task. We propose and evaluate two complementary methods for item-based rec-ommendation: one based on the visual descriptors of the deep neural network, the other based on the content-based attrib-utes of persons. The initial evaluation made by forensic technicians shows that although results favored visual descriptors over attribute-based similarity, both approaches are functional and highly diverse in terms of recommended objects. Thus, future work should in-volve incorporating both approaches in a single prediction method, preference learning based on the feedback from forensic technicians and recommendation of assembled lineups instead of single candidates.

1. Towards Recommender Systems
for Police Photo Lineup
Ladislav Peškaa and Hana Trojanováb
a Department of Software Engineering,
b Department of Psychology,
Charles University, Prague,
Czech Republic

2. Ladislav Peska & Hana Trojanova: Towards
Recommender Systems for Police Photo Lineup
2
Recommender Systems
 Propose relevant items to the right persons at the right time
 Successful on many various domains
 Multimedia
 E-commerce
 POIs
 Events
 Recepies
 …
 However, some novel (relevant, important) tasks are yet to
discover
DLRS@RecSys2017, Como

3. Ladislav Peska & Hana Trojanova: Towards
Recommender Systems for Police Photo Lineup
3
Police Photo Lineup
 Eyewitness identification of the suspect / offender during
criminal proceedings
 Most common case: select suspect among other persons (fillers)
 Either „in natura“ or based on photographies
DLRS@RecSys2017, Como
- Suspect is positioned at
random
- 3-7 fillers are added, witness
should not know them
- Wittness is instructed that the
offender may or may not be
present

4. Ladislav Peska & Hana Trojanova: Towards
Recommender Systems for Police Photo Lineup
4
Errors in Photo Lineup
DLRS@RecSys2017, Como
1) Observe the event
2) Identify the offender
- among other candidates

5. Ladislav Peska & Hana Trojanova: Towards
Recommender Systems for Police Photo Lineup
5
Errors in Photo Lineup
DLRS@RecSys2017, Como
1) Observe the event.
- numerous sources of error (poor lighting,
large distance, short time, fear, fatigue…)
- affects memory and recognition processes
- decrease witness certainty and reliability
2) Identify the offender
- decreased certainty may lead
to incorrect identification
- and conviction of innocent
persons

6. Ladislav Peska & Hana Trojanova: Towards
Recommender Systems for Police Photo Lineup
6
Assembling Fair Photo Lineups
 System should be set to eliminate testimony of uncertain
witnesses
 Double-blind administration
 Fair (unbiased) assembling of lineups
 Out of the dataset of candidates, lineup administrator should select
fillers similar to the suspect
 Currently, only feature-based filtering systems are available
 Better automatization in content processing is necessary
 How to better approximate human-perceived similarity of
persons?
 Convolutional Neural Networks?
DLRS@RecSys2017, Como

7. Ladislav Peska & Hana Trojanova: Towards
Recommender Systems for Police Photo Lineup
7
Assembling Fair Photo Lineups
 First approach: item-based recommendations
 Propose top-k candidates for each suspect
 CB-RS, cosine similarity of candidates’ and suspect’s CB features
 Nationality, Age, Appearence features
 Baseline (mimic feature-based filters)
 Visual-RS, cosine similarity of candidates’ and suspect’s VGG-
Face probability layers
 Approximate the similarity of a candidate and the suspect through their
similarities with other persons
 VGG-Face1 network
 VGG network for facial recognition problem
 Trained as a multi-classification problem
 Dataset of 2,622 celebrities with 1,000 images each
1Parkhi, O. M., Vedaldi, A., & Zisserman, A. (2015). Deep Face Recognition. British Machine Vision Conference.
DLRS@RecSys2017, Como

8. Ladislav Peska & Hana Trojanova: Towards
Recommender Systems for Police Photo Lineup
8
Research Questions
 Is candidates recommendation (based on VGG-Face’s
features) suitable for the lineup assembling?
 How well is the human-perceived similarity approximated within top-k
candidates?
 What is the role of diversity in lineup assembling?
 Is there a space for personalized recommendations (i.e. personalized per
lineup administrator)?
 Can there be „too similar“ candidates?
DLRS@RecSys2017, Como

9. Ladislav Peska & Hana Trojanova: Towards
Recommender Systems for Police Photo Lineup
9
Evaluation – Protocol
 Dataset of missing and wanted persons in Czech Republic
 4423 males
 User study, 30 different lineups, 7 forensic technicians
 For each lineup, both CB-RS and Visual-RS proposed top-20 candidates
 Merged into one list, forensic technician should select relevant fillers for
particular suspect
 Follow-up questionaire and discussion
DLRS@RecSys2017, Como

10. Ladislav Peska & Hana Trojanova: Towards
Recommender Systems for Police Photo Lineup
10
Results
 In total 202 completed lineups, 800 selected candidates
 Recommendations were seemingly relevant in most cases
 Visual-RS considerably outperformed CB-RS
 The performance difference was much smaller for suspects from outside
of central Europe
 Effect of different ethnicity? (Known in the forensic psychology literature).
DLRS@RecSys2017, Como
Total lineups Selected candidates Level of agreement
All lineups
Visual-RS
202
466 (58%) 0.178
CB-RS 298 (37%) 0.138
Lineups with suspects from countries outside Central Europe
Visual-RS
53
105 (51%) 0.128
CB-RS 82 (40%) 0.205

11. Ladislav Peska & Hana Trojanova: Towards
Recommender Systems for Police Photo Lineup
11
Results II.
 Very low intersection (1.8%) of top-20 CB-RS and Visual-RS candidates
 Potential to apply some combination in the future work
 Average position of selected candidate
CB-RS: 8.9 (+/- 5.2); Visual-RS: 8.2 (+/- 5.7)
 Ordering seems relevant even in top-20, however can be improved
 Low level of agreement among participants on the selected candidates (0.178, 0.138)
 Limits of the global ordering?
 Potential for personalized recommendation?
 Less diversity is better
 Participants agreed on the need for providing homogeneous lineups
 Dynamical recommendation based on the selected candidates?
 No seemingly too similar candidates
 However, the dataset contained some duplicates
 Should be addressed in the future work
DLRS@RecSys2017, Como

12. Conclusions
 Recommending candidates based on CNN’s features seems
plausible in police photo lineup
 Item-based recommendations seems like a good first approach
 There is a potential for:
 Session-based personalized recommendations (need for uniformity of the final list)
 Long-term preference learning (due to the level of disaggreeement among lineup
administrators)
 Approaches combining CNN’s features with content-based attributes
Future work
 Move from recommended candidates to recommended lineups
 Evaluation through mock witnesses
 Deployable demo
DLRS@RecSys2017, Como Ladislav Peska & Hana Trojanova: Towards
Recommender Systems for Police Photo Lineup
12

13. DLRS@RecSys2017, Como Ladislav Peska & Hana Trojanova: Towards
Recommender Systems for Police Photo Lineup
13
Thank you!
Questions, comments?
Supplementary materials: http://github.com/lpeska/lineups, https://tinyurl.com/yat3rtr2
Slides: https://www.slideshare.net/LadislavPeska/towards-recommender-systems-for-police-photo-lineup