This document summarizes a presentation on semi-automatic tagging of images on Wikimedia Commons. It discusses student projects at Bern University of Applied Sciences that developed approaches using deep learning and custom models to tag images and link concepts to Wikidata while keeping a human in the loop. It also describes a current project between Wikimedia Sverige, Bern University of Applied Sciences, and SWITCH to implement a prototypical solution using the ISA Tool, Google Cloud Vision, and a new algorithm for tagging images on Wikimedia Commons semi-automatically based on metadata. The presentation reflects on lessons learned and outlines next steps to address issues, increase visibility, improve algorithms, and include structured data on Commons in SWITCH's Research Data Connectome.

1. Berner Fachhochschule | Haute école spécialisée bernoise | Bern University of Applied Sciences
Semi-automatic Tagging of Images on Wikimedia Commons
Wikidata Con 2023, 28-29 October 2023, Online & Taiwan on-site conference
Beat Estermann
▶ Bern University of Applied Sciences, Bern Academy of the Arts
Homepage image of the ISA Tool by Islahaddow, using File:2017 06 Ali- 00213.jpg by user:Alimdaihli, CC BY-SA 4.0 (Wikimedia Commons)
The text of this slide deck is made available under a CC BY 4.0 License.

2. Berner Fachhochschule | Haute école spécialisée bernoise | Bern University of Applied Sciences
▶ How to leverage the power of artificial intelligence for the tagging of images?
▶ Student Projects at Bern University of Applied Sciences in 2020 and 2021:
• Leverage deep learning approaches for the tagging of images.
• Link concepts to Wikidata.
• Develop custom models to escape the “black box” of proprietary algorithms.
• The way forward is not fully automatic tagging, but semi-automatic tagging with a human in the loop.
• Combine the CAT Tool and the ISA Tool on Wikimedia Commons for a start.
• In addition to image recognition, leverage also existing image metadata.
Background / Motivation

3. Berner Fachhochschule | Haute école spécialisée bernoise | Bern University of Applied Sciences
Purpose:
Building a FAIR Knowledge
Graph of Open Data for
Research & Education
Try the Beta:
https://opendatanavigator.switch.ch
Support:
Open Data Navigator &
API Project Memberships
Switch Connectome Project
© SWITCH

4. Berner Fachhochschule | Haute école spécialisée bernoise | Bern University of Applied Sciences
The SWITCH Foundation is
the Swiss national
infrastructure service
provider for higher
education and research.
The core aspects of the
foundation’s mission are to
enable, maintain and
promote a secure and
networked research and
education infrastructure in
Switzerland.
Who is SWITCH?
© SWITCH
https://www.switch.ch/
© SWITCH

5. Berner Fachhochschule | Haute école spécialisée bernoise | Bern University of Applied Sciences
▶ Project Partners:
• SWITCH
• Wikimedia Sverige / software developers & power user of the ISA Tool
• Bern University of Applied Sciences / Bern University of the Grisons
▶ Project Goal:
• Implement a prototypical solution for the semi-automatic tagging of images on Wikimedia
Commons, using:
• the ISA Tool
• Google Cloud Vision
• a newly developed algorithm for the extraction of entities based on metadata
Project Website: https://commons.wikimedia.org/wiki/Commons:ISA_Tool/Image_to_Concept
SWITCH InnoLab “Image to Concept” (2022-2023)

6. Berner Fachhochschule | Haute école spécialisée bernoise | Bern University of Applied Sciences
The ISA Tool (Live Demo…)

7. Berner Fachhochschule | Haute école spécialisée bernoise | Bern University of Applied Sciences
The Technical Solution
Link to File (Wikimedia Commons, User: Beat Estermann, CC BY-SA)

8. Berner Fachhochschule | Haute école spécialisée bernoise | Bern University of Applied Sciences

9. Berner Fachhochschule | Haute école spécialisée bernoise | Bern University of Applied Sciences
Current State of Implementation
▶ Test Version of the enhanced Tool deployed at: https://isa-dev.toolforge.org/
• Tag suggestions from the algorithms not always working.
• Saving tags not always working.
• The situation is currently worse than during user testing in winter 2022-2023.
A series of potential causes for the issues encountered have been identified;
bug fixing has been delayed due to a lack of resources.
▶ Production Version of the Tool available at: https://isa.toolforge.org/
(without the new features)
• This version was down between May and August 2023 due to prolonged maintenance issues.
• Up and running again since mid-August 2023.

10. Berner Fachhochschule | Haute école spécialisée bernoise | Bern University of Applied Sciences
▶ Resolve the remaining performance and reliability issues !!
▶ Increase the visibility and take-up of the tool among potential contributors.
▶ Assess and monitor the relevance of the ISA Tool in comparison to other tools (for adding
Structured Data on Commons).
▶ Engage in a dialogue with various stakeholders on what constitutes “good” tagging of images.
▶ Further improve/complement the algorithms used for semi-automatic tagging.
▶ Develop alternatives to the current requirement of uploading all media files to Wikimedia
Commons (free license requirement is too restrictive for many research use cases).
▶ Clarify roles and responsibilities with regard to deployment, operations, and maintenance
(SLAs).
Key Learnings

11. Berner Fachhochschule | Haute école spécialisée bernoise | Bern University of Applied Sciences
▶ For R&D projects, we need reliable partner organizations.
▶ The respective movement entities need to be empowered for this role.
▶ Maintenance and development of important tools & services need to be backed-up
by an organizational commitment.
▶ We should become experts in navigating between the “volunteer” and the “paid staff” worlds;
we need to develop a shared professional culture englobing both of them.
Reflections with regard to the wider Community Discussion
Movement Strategy

12. Berner Fachhochschule | Haute école spécialisée bernoise | Bern University of Applied Sciences
▶ Include Structured Data on Commons in the SWITCH Research Data Connectome
▶ Fix the remaining bugs of the enhanced version of the ISA Tool
▶ Further Develop the Metadata-to-Concept Algorithm
(students’ project at University of Virginia – School of Data Science)
Outlook

13. Berner Fachhochschule | Haute école spécialisée bernoise | Bern University of Applied Sciences
Beat Estermann
Bern Academy of the Arts
Representative Open Science Board &
Digital Humanities
beat.estermann@hkb.bfh.ch
Opendata.ch
Member of the Board
beat.estermann@opendata.ch
Contact