1. Knowledge-based fusion for
image tampering localization
Chyssanthi Iakovidou, Symeon Papadopoulos
and Yiannis Kompatsiaris
Multimedia Knowledge and Social Media Analytics Lab (MKLab, https://mklab.iti.gr/)
Information Technologies Institute (ITI), Centre for Research and Technology Hellas (CERTH)
June 5-7, 2020 @ AIAI 2020

2. Background
1 Image forensics
Image forensics deals with the problem of detecting
tampered images at:
• Image-level (Tampering detection)
two-class classification problem with the goal to
classify given image as authentic/forged
• Pixel-level (Tampering localization)
reporting and localizing tampered image regions
through heat-maps where tampered and non-
tampered pixels have distinct values
Tampering detection report;
Tampering localization report;
Heat-maps:
Label: “forged”
score: 70%

3. Background
2 Image forensics and current challenges
Challenges:
• Different tampering sessions leave different traces
SoA tools are designed to detect tampering based on only subsets of traces.
• In real scenarios a single forgery session includes multiple different manipulations
Analysis of different tools may be needed to get a robust forensic report
• Forged images shared over the Internet, undergo further transformations
Cropping, resizing, resaving disrupt tampering traces and hinder tools’ efficiency
• Few realistic tampering datasets available for experimenting
Realistic tampering is not easily achieved through automatic procedures

4. Proposed approach
3 Tamperinglocalization fusion framework
Goal
• Introduce extensible fusion framework for tampering localization and output refinement
Design strategy
• Select and categorize SoA approaches on a multi-criterion ranking and grouping process
• Integrate expert background knowledge on the behaviour of SoA approaches (types of images,
encoding, supported traces, known limitations, etc.)
• Employ a fusion mechanism based on local and cross-tool statistics to produce a single, refined fused
heat-map output for tampering localization
Benefits
• Leverage tools that are complementary to each other
• Present tampering localization results to end users in a manner that is easier to interpret

5. Proposed approach
3a Candidate selection process
• Detection of JPEG compression traces, in the transform domain.
• Detection of JPEG compression traces, in the spatial domain
• Noise-based detection

6. 3a
Proposed approach
Candidate selection process
• Evaluation on both realistic and synthetic
benchmark datasets
• Evaluation extended to include reports after
various post-processing operations have been
applied on the original datasets
• The ability to retrieve true positives of tampered images
at a low level of false positives (KS@0.05);
• The ability to achieve good localization of the tampered
region within the image (F1)
• The readability of the produced heat map, i.e., a high
distinction of assigned values for tampered and
untampered pixels expressed as the range of different
binarization thresholds that result in high F1 scores.

7. 3a
Proposed approach
Candidate selection process
a) performance: KS score, max F1 score; threshold binarization range,
b) average performance of methods based on normalized KS, F1 and threshold range per dataset;
c) rank aggregation results based on Borda count,Copeland and Kemeny-Young voting

8. 3a
Proposed approach
Candidate selection process
After evaluations of 14 established state-of-the-art methods [1] for splicing localization the
following were selected:
• ADQ1 [2] and DCT [3] are based on analysis of JPEG compression in the transform domain
• BLK [4] and CAGI [5] are based on analysis of JPEG compression in the spatial domain
• NOI3 [6] is a noise-based detector and is integrated as a complementary tool.
[1] M. Zampoglou, et al., "Large-scale evaluation of splicing localization algorithms for web images," MultimTools Appl., vol. 76, no. 4, p. 4801–4834, 2016.
[2] Z. Lin et al., "Fast, automatic and fine-grained tampered {JPEG} image detection via {DCT} coefficient analysis," Pattern Recognition, vol. 42, no. 11, 2009.
[3] S. Ye, et al, "Detecting digital image forgeries by measuring inconsistencies of blocking artifact," in IEEE Int. Conference on Multimedia and Expo, 2007.
[4] W. Li, et al., "Passive detection of doctored JPEG image via block artifact grid extraction," Signal Processing, vol. 89, no. 9, p. 1821:1829, 2009.
[5] Iakovidou, et al. (2018). Content-aware detection of JPEG grid inconsistencies for intuitive image forensics. J. of Visual Com. and Image Representation, 54, 155-170..
[6] D. Cozzolino, et al, "Splicebuster: A new blind image splicing detector," in IEEE International Workshop on Information Forensics and Security, 2015.

9. Proposed approach
3b Fusion strategy
Designing an extensible fusion and output refinement framework for tampering localization

10. Proposed approach
Normalization Unit
Output maps are normalized in the [0, 1]
range at image level.

11. Proposed approach
Binarization Unit
Automate map binarization by choosing
the binarization threshold as a value
belonging to the respective safe ranges
per method (empirically defined)

12. Proposed approach
Connected Component Unit
For every 8-connected region (blob) of
the binarized map we calculate centroid.
Next, we build a feature vector:
• # detected connected regions,
• location of centroids
• spatial standard deviation of the
pixels belonging to a region from their
respective centroid,
• image area of blob (bounding box)
containing the pattern of interest.

13. Proposed approach
Filtering Unit
Normalized maps and outputs from
Component Unit used for filtering.
First, we filter based on findings of each
method independently from one another:
• Blobs discarded if too big or too small
• Blobs whose bounding boxes overlap
by more than 90% are merged
• Blobs ranked and top 5 selected
based on i) centroids distance from
the overall map centroid, ii) density of
pixels in the blob, and iii) their size.

14. Proposed approach
Filtering Unit
Perform a content-aware filtering step
utilizing information extracted by
respective methods (CAGI and DCT) to
filter blobs that may occur as false
localizations, such as:
- Over/under exposed image areas
- Image areas of intense texture
- Image areas of consistent intensity
values

15. Proposed approach
Statistics Extraction Unit
Extraction of statistics to automate the
evaluation of the outputs’ usefulness
• Multilevel measurements of the
entropy of data and the Kolmogorov-
Smirnov (KS) statistic

16. Proposed approach
Fusion Unit
Leverages the intermediate calculations
to produce a single fused output.
• Interpretability of maps: Maps
ranked and assigned a confidence
score, Ci, based on difference of the
entropy before and after binarization.
• Compatibility between the traces
detected by different methods:
Confidence of a method is reinforced
if other methods detecting similar
traces also achieve high confidence.

17. Proposed approach
Fusion Unit
• Reliability of method: Score
assigned to each method during the
candidate selection process is used
to rank the methods to help define
their contribution to the final outcome
• Confidence in the presence of
identified tampered regions: The
blobs with the highest KS score of
the best ranking method serve as
baselines. The final refined map is
constructed through comparisons of
the baseline with blob mask of other
methods in a ranked, weighted order.

18. Experimental Evaluation
4 Experimental setup
We tested on two publicly available datasets:
• The First IFS-TC Image Forensics Challenge
training set that contains 450 user-submitted
forgeries and was designed to serve as a realistic
benchmark.
• The CASIA V2.0 dataset contains 5,123
realistically tampered color images of varying sizes
It includes uncompressed images and also JPEG
images with different quality factors.

19. Experimental Evaluation
4 Experimental setup
Evaluation metrics
Overall localization quality and readability is based on the pixel-wise agreement between the
reference mask (Ground Truth, GT) and the produced tampering localization heat map and it is
measured in terms of the achieved F-score (F1).
𝐹1 = 2𝑇𝑃/(2𝑇𝑃 + 𝐹𝑃 + 𝐹𝑁)
where (TP) number of true positive, (TN) number of true negative, (FP) number of false positive, and
(FN) number of false negative.
This evaluation methodology requires the output maps to be thresholded prior to any evaluation:
• normalize all maps in the [0, 1] range,
• successively shift the binarization threshold by 0.05 increments (step), and
• calculate the achieved F1 score for every step

20. Experimental Evaluation
4 Experimental results
F1 score curves on the (a) Challenge and (b) CASIA2 datasets for FUSED and five base methods.
Localization quality

21. Experimental Evaluation
4 Experimental results
Best mean F1 score and binarization range that allows F1 to remain high (> 70% of respective
maximum F1 score) and reported detections for F1 score >= 0.7 at each method's best binarization
threshold for Challenge and CASIA2 datasets.
Unique Localizations corresponds to the number of detections exclusively achieved by that method
Output interpretability

22. FUSIONINPUT GT CAGI BLK ADQ
1
NOI3 DCT

23. Discussion
5 Experimental findings and next steps
• In both datasets the fused output achieves high F1 scores over a wide range of
thresholds → increased localization ability and interpretability.
• In both datasets the fused method reports a high number of absolute localizations
while also contributing additional unique localizations through fusion and refinement of
the available individual outputs.
• Overall, we verified the importance of exploiting the available state-of-the-art methods
in a manner that improves the robustness and user-friendliness of the output.
• Next steps include introducing more methods to the framework and eliminating hard-
coded expert knowledge in the fusion criteria and rules moving towards introducing
fusion approaches based on supervised learning.

24. Thank you
Partially funded by the European Commission under contract num. H2020-825297 WeVerify and H2020-700024 TENSOR
Dr. Symeon Papadopoulos [papadop@iti.gr] [@sympap]
Media Verification team, http://mever.iti.gr/
Multimedia Knowledge and Social Media Analytics Lab,
https://mklab.iti.gr/