Models obtained by decision tree induction techniques excel in being interpretable. However, they can be prone to overfitting, which results in a low predictive performance. Ensemble techniques provide a solution to this problem, and are hence able to achieve higher accuracies. However, this comes at a cost of losing the excellent interpretability of the resulting model, making ensemble techniques impractical in applications where decision support, instead of decision making, is crucial. To bridge this gap, we present the GENESIM algorithm that transforms an ensemble of decision trees into a single decision tree with an enhanced predictive performance while maintaining interpretability by using a genetic algorithm. We compared GENESIM to prevalent decision tree induction algorithms, ensemble techniques and a similar technique, called ISM, using twelve publicly available data sets. The results show that GENESIM achieves better predictive performance on most of these data sets compared to decision tree induction techniques & ISM. The results also show that GENESIM's predictive performance is in the same order of magnitude as the ensemble techniques. However, the resulting model of GENESIM outperforms the ensemble techniques regarding interpretability as it has a very low complexity.

1. GENESIM: GENetic Extraction of a Single, Interpretable Model
Gilles Vandewiele, Kiani Lannoye, Olivier Janssens,
Femke Ongenae, Filip De Turck, Sofie Van Hoecke

2. 1.
Decision tree ensembles
Dietterich, Thomas G. "Ensemble methods in machine learning”

3. 3
An ensemble combines predictions of (uncorrelated) classifiers

4. 4
Advantages of decision tree ensembles

5. 5
Disadvantages of decision tree ensembles
Majority voting/mean
Stacking

6. 6
Interpretability is of major importance in critical domains!
Healthcare Finance Law

7. 7
GENESIM: best of both worlds

8. 2.
Merging decision trees
Andrzejak, Artur, Felix Langner, and Silvestre Zabala. "Interpretable models from distributed data via merging of decision trees."

9. 9
Decision tree Decision space
Converting a decision tree to a decision space

10. 10
Calculating the intersection of two spaces (computational geometry)

11. 11
Possible extension: pruning of the merged decision space

12. 12
How to convert the merged decision spaces back to decision trees?
Possible root nodes

13. 3.
GENESIM: genetic algorithms to the rescue!

14. 14
GENESIM

15. 15
Creating an initial population

16. 16
Iteratively merging decision trees in a genetic fashion

17. 17
Evaluating the fittest individual

18. 18
Mutations
8
Swap two random subtrees Change split threshold

19. 19
Other use-case: distributed learning
Partition 1
Partition n
.
.
.
.
.
.

20. 4.
Other similar techniques: inTrees & ISM
Deng, Houtao. "Interpreting tree ensembles with intrees.“
Van Assche, Anneleen, and Hendrik Blockeel. "Seeing the forest through the trees: Learning a comprehensible model from an ensemble."

21. 21
inTrees: from an ensemble to an ordered rule list

22. 22
Ordered rule list is less interpretable than decision tree!

23. 23
ISM: constructing new trees with split criteria calculated from the ensemble

24. 5.
Results

25. 25
Compared GENESIM to all prominent decision tree induction algorithms:
- C4.5
- CART
- QUEST
- GUIDE
Two (decision tree) ensemble methods:
- Random Forests
- eXtreme Gradient Boosting (XGBoost)
And one similar model extraction technique
- ISM (Interpretable, Single Model)
Hyper-parameters are tuned using grid
search or bayesian optimization (GPs)

26. 26
* 10 measurements with 3-fold cross-validation
→ Measure accuracy and model complexity (number of nodes)
* Construct Win-Tie-Loss matrix with bootstrap testing:
→ Algorithm A wins over algorithm B when its mean accuracy is higher
and the p-value of the bootstrap test <= 0.05
→ Tie between A and B when p-value > 0.05

27. 27
GENESIM outperforms all decision tree induction algorithms, except for C4.5 (but produces smaller trees!)
Complexity Accuracy

28. 6.
Conclusion and future work

29. 29
GENESIM extracts an interpretable model with enhanced predictive performance

30. 30
Reduce the computational runtime and perform optimizations to allow for hyper-parameter tuning
and running for more iterations!

31. 31
Try other meta-heuristics & increase initial population diversity
Particle Swarm
Optimization
Ant Colony
Optimization

32. 32
THANK YOU!
Acknowledgements:
-Reviewers & organizing committee
-My promotors: Filip De Turck & Femke Ongenae
gilles.vandewiele@ugent.be
@Gillesvdwiele
https://github.com/IBCNServices/GENESIM