This presentation by Deevid De Meyer outlines how Brainjar uses human-centric design and explainability to create machine learning systems that work together with humans to improve efficiency while reducing error rate.

1. Error-proof Machine Learning with explainability and human-centric design

2. WE ARE AN END-TO-END ARTIFICIAL
INTELLIGENCE SOFTWARE FACTORY
Human-in-the-loop
AI systems inevitably lose accuracy
over time. Brainjar builds systems that
are able to continually improve by
working together with their human
colleagues.
Technology agnostic
Because of our broad technology
expertise, we always choose the best
solution for problem. This can be AI
API, pre-trained model or custom
model and is always cloud agnostic.
End-to-end
Brainjar works end-to-end with the
customer to build and integrate the AI
solution. We start with a data analysis,
followed by labeling, model training,
front-end building and deployment.

3. WE ARE PART OF
THE RACCOONS GROUP
Software
prototyping
AI end-to-end
Chatbot framework
Blockchain services
Full stack Javascript
Quantum computing
Edge AI
Digital production house
AI Business
Strategy

4. AI is...

5. NOT new (but has changed)
Early AI (1950’s – 2000’s)
• Deterministic, human-made algorithms
• Centered around planning or abstract reasoning
• Scaled horribly
• Required maintenance of structured “knowledge
systems”
Modern AI (2000’s – Now)
• “Learning” algorithms -> Machine Learning
• Centered around human senses and prediction
• Can scale with data
• Can process unstructured data
Source: Stanford news

6. Probabilistic
(Modern) AI makes mistakes!
• ~95-99% accuracy is typical (depending on usecase)
• Requires strategy for 5% (human review)
• Algorithms can provide “confidence level”, allowing
humans to only review edge cases
Source: wikipedia

7. LESSON 1
Think about mistakes

8. NOT Self-Learning!
Machine Learning learns from VERIFIED data
• Many tasks require human verification for the
system to learn & improve
• Initial training can be done using historic data
• Human feedback still required during production
Source: Medium

9. NOT Self-Learning!
support question central support domain experts support question ML algorithm domain experts support question ML algorithm domain experts
Example: Mail Forwarding
(1) (2) (3)

10. LESSON 2
Learn from your users

12. Human-Centered Machine Learning
Cooperation benefits all parties
• AI eliminates tedium
• Humans focus on edge cases
• Two key principle builds trust with management
• High quality data is produced
• AI improves/adapts continuously
Source: wikipedia

14. LESSON 3
Learn your users

15. Our work

16. CASE 1
ORDER ADMINSTRATION
AT TVH
TVH Group is a Belgian company active in the market of forklift trucks, aerial work platforms, agricultural tractors and industrial vehicles.
The company has an order department that handles orders from customers via email.
TVH has a large order administration
department.
O R D E R A D M I N I S T R AT I O N
Customers can place orders by sending an
email with their request to TVH.
E M A I L O R D E R S
The rental administrators handle these
requests by reading the mails and
registering the order in the ERP system.
R E P E T I T I V E W O R K

17. A R T I F I C I A L I N T E L L I G E N C E
Determine if the email is a rental
order or not & extracts all entities
in the email
Present an intuitive verification
step for users with automatic
retraining
Structured order is placed in the
ERP system
C O N N E C T T O
M A I L B O X
SOLUTION
Connects with existing mailbox &
gets triggered when there is an
incoming mail
V E R I F I C A T I O N P L A C E O R D E R

18. Customers of TVH send orders via email to the order administration. A typical example is shown below.
CONNECT TO MAILBOX

19. V E R I F I C A T I O N P L A C E O R D E RC O N N E C T T O M A I L B O X
A R T I F I C I A L
I N T E L L I G E N C E
SOLUTION
Determine if the email is a rental
order or not & extracts all entities
in the email
Present an intuitive verification
step for users with automatic
retraining
Structured order is placed in the
ERP system
Connects with existing mailbox &
gets triggered when there is an
incoming mail

20. Automatically determines if the email is a rental order & extracts all entities in the email.
ARTIFICIAL INTELLIGENCE

21. P L A C E O R D E R
V E R I F I C AT I O N
SOLUTION
Determine if the email is a rental
order or not & extracts all entities
in the email
Present an intuitive verification
step for users with automatic
retraining
Structured order is placed in the
ERP system
Connects with existing mailbox &
gets triggered when there is an
incoming mail
C O N N E C T T O M A I L B O X A R T I F I C I A L I N T E L L I G E N C E

22. P r e s e n t a n i n t u i t i v e v e r i f i c a t i o n s t e p f o r u s e r s w i t h a u t o m a t i c r e t r a i n i n g .
VERIFICATION

23. P L A C E O R D E R
S t r u c t u r e d o r d e r i s p l a c e d
i n t h e E R P s y s t e m
P L A C E O R D E R
SOLUTION
Determine if the email is a rental
order or not & extracts all entities
in the email
Present an intuitive verification
step for users with automatic
retraining
Structured order is placed in the
ERP system
Connects with existing mailbox &
gets triggered when there is an
incoming mail
C O N N E C T T O M A I L B O X A R T I F I C I A L I N T E L L I G E N C E V E R I F I C A T I O N

24. BUSINESS CASE
The current order administration in each branch of TVH has 7 full time equivalents handling over 400 incoming orders via email per day.
5 MIN
T I M E P E R E M A I L
1 MIN
BEFORE:
AFTER:
80%
T I M E R E D U C T I O N

25. THE KLASSIF.AI SOLUTION
K l a s s i f . a i h a s i d e n t i f i e d f o u r k e y c o m p o n e n t s t h a t a l l o w o r g a n i s a t i o n s t o l e v e r a g e m a c h i n e l e a r n i n g t o
i m p r o v e t h e i r d o c u m e n t p r o c e s s i n g w o r k f l o w s . C o m b i n i n g t h e s e c o m p o n e n t s a l l o w s K l a s s i f . a i t o p r o c e s s
d o c u m e n t s a t c o m p u t e r s p e e d s w i t h h u m a n r e l i a b i l i t y .
For your experts, it’s business as usual.
Meanwhile, Klassif.ai is gathering training
data to learn the ins and outs of your
documents. The result is a training set that
represents the real world.
L A B E L I N G
Just like any mature software project
needs versioning and logging, so does a
machine learning model! Klassif.ai provides
a model history, along with statistics that
show you the model’s performance on
real, unseen data.
G O V E R N A N C E
Klassif.ai integrates seamlessly into your
current document processing workflow,
enriching your unstructured documents and
connecting them to your structured systems.
I N T E G R AT I O N
Every document type requires its own
approach. That’s why Klassif.ai classifies each
document before extracting the required
information. After all, an invoice needs to be
processed differently to an employee
contract.
C L A S S I F I C AT I O N &
E X T R A C T I O N

26. KLASSIF.AI
Platform features
OCR USER MANAGEMENT
AUTOMATIC
RETRAINING
SECURITY NOTIFICATIONS
ADMIN PANEL
MULTILINGUAL
SUPPORT
SCALABILITY
MODULAR AI MODELS ON-PREMISE OR IN THE CLOUD

27. PROJECT
APPROACH
K L A S S I F . A I
We map and visualise your business
processes
1 . I N TA K E
Your domain experts generate data while
doing their work as usual
4 . L A B E L
We customise our platform based on your
way of working
3 . C U S T O M I S E
We integrate our services with your
systems
2 . I N T E G R AT E
We train a machine learning algorithm
that meets your needs
5 . T R A I N M O D E L
We ready the platform for your experts
6 . D E P L O Y
P O C L E A D T I M E :
1 - 3 M O N T H S
P R O D U C T I O N R E A D Y:
2 - 5 M O N T H S

28. CASE 2
HEALTHCARE DATA EXTRACTION
AT RIZIV
RIZIV is the Belgian federal institute for health and disability insurance. The institute is, among others, responsible for handling files on
medical accidents.
Hospitals scan all case documents and
email them to RIZIV. An administrator at
RIZIV makes sure the case is complete.
C A S E D O C U M E N T S
A doctor from RIZIV goes over all case
documents and decides on refund of
medical costs.
R E V I E W B Y D O C T O R
An intern prints all case documents and
sorts these on date before scanning them
again.
M A N U A L W O R K

29. O C R A R T I F I C I A L I N T E L L I G E N C E V E R I F I C A T I O N
B Y D O C T O R
Convert scanned document to
computer readable text
Extract all entities such as, doctor
name, procedure date and
procedure type from document
Present an intuitive verification
step for doctors with automatic
retraining
U P L O A D
D O C U M E N T S
SOLUTION
Doctors can upload all case
documents into application

30. U P L O A D D O C U M E N T S
O C R
SOLUTION
Convert scanned document to
computer readable text
Extract all entities such as, doctor
name, procedure date and
procedure type from document
Present an intuitive verification
step for doctors with automatic
retraining
Doctors can upload all case
documents into application
A R T I F I C I A L I N T E L L I G E N C E V E R I F I C A T I O N
B Y D O C T O R

31. A R T I F I C I A L
I N T E L L I G E N C EU P L O A D D O C U M E N T S O C R
SOLUTION
Convert scanned document to
computer readable text
Extract all entities such as, doctor
name, procedure date and
procedure type from document
Present an intuitive verification
step for doctors with automatic
retraining
Doctors can upload all case
documents into application
V E R I F I C A T I O N
B Y D O C T O R

32. P L A C E O R D E R
S t r u c t u r e d o r d e r i s p l a c e d
i n t h e E R P s y s t e m
V E R I F I C AT I O N
B Y D O C T O RU P L O A D D O C U M E N T S O C R
SOLUTION
Convert scanned document to
computer readable text
Extract all entities such as, doctor
name, procedure date and
procedure type from document
Present an intuitive verification
step for doctors with automatic
retraining
Doctors can upload all case
documents into application
A R T I F I C I A L I N T E L L I G E N C E

33. Present an intuitive verification step for doctors with automatic retraining
VERIFICATION BY DOCTOR

34. CASE 3
DIGITALISING UNEMPLOYMENT BENEFITS
AT HVW
HVW, the Belgian relief fund for unemployment benefits, is a public institution for social security which pays unemployed citizens. Each
month HVW pays more than 120.000 benefit recipients their replacement income.
HVW has a large administration
department who processes all
unemployment benefits
B E N E F I T A D M I N I S T R AT I O N
The majority of unemployment forms are
still submitted on paper
PA P E R F O R M S
Due to human mistakes, often error are
introduced during the input step requiring
extra steps to fix mistakes
E R R O R P R O N E

35. H A N D W R I T I N G
R E C O G N I T I O N
A R T I F I C I A L I N T E L L I G E N C E V E R I F I C A T I O N
C o n v e r t h a n d w r i t t e n
c o n t a c t i n f o r m a t i o n t o
c o m p u t e r r e a d a b l e t e x t
E x t r a c t c a l e n d a r
i n f o r m a t i o n f r o m p h o t o
P r e s e n t a n i n t u i t i v e
v e r i f i c a t i o n s t e p f o r u s e r s
w i t h a u t o m a t i c r e t r a i n i n g
U P L O A D
P H O T O
U s e r s c a n u p l o a d a p h o t o o f
t h e i r u n e m p l o y m e n t c a r d
SOLUTION

36. U s e r s c a n u p l o a d a p h o t o o f t h e i r u n e m p l o y m e n t c a r d
UPLOAD PHOTO

37. U P L O A D P H O T O
U s e r s c a n u p l o a d a p h o t o
o f t h e i r u n e m p l o y m e n t
c a r d
H A N D W R I T I N G
R E C O G N I T I O N
C o n v e r t h a n d w r i t t e n c o n t a c t
i n f o r m a t i o n t o c o m p u t e r
r e a d a b l e t e x t
A R T I F I C I A L I N T E L L I G E N C E
E x t r a c t c a l e n d a r
i n f o r m a t i o n f r o m p h o t o
V E R I F I C A T I O N
P r e s e n t a n i n t u i t i v e
v e r i f i c a t i o n s t e p f o r u s e r s
w i t h a u t o m a t i c r e t r a i n i n g
SOLUTION

38. A R T I F I C I A L
I N T E L L I G E N C E
E x t r a c t c a l e n d a r i n f o r m a t i o n
f r o m p h o t o
U P L O A D P H O T O
U s e r s c a n u p l o a d a p h o t o
o f t h e i r u n e m p l o y m e n t
c a r d
H A N D W R I T I N G
R E C O G N I T I O N
C o n v e r t h a n d w r i t t e n
c o n t a c t i n f o r m a t i o n t o
c o m p u t e r r e a d a b l e t e x t
V E R I F I C A T I O N
P r e s e n t a n i n t u i t i v e
v e r i f i c a t i o n s t e p f o r u s e r s
w i t h a u t o m a t i c r e t r a i n i n g
SOLUTION

39. P L A C E O R D E R
S t r u c t u r e d o r d e r i s p l a c e d
i n t h e E R P s y s t e m
V E R I F I C AT I O N
P r e s e n t a n i n t u i t i v e
v e r i f i c a t i o n s t e p f o r u s e r s
w i t h a u t o m a t i c r e t r a i n i n g
U P L O A D P H O T O
U s e r s c a n u p l o a d a p h o t o
o f t h e i r u n e m p l o y m e n t
c a r d
H A N D W R I T I N G
R E C O G N I T I O N
C o n v e r t h a n d w r i t t e n
c o n t a c t i n f o r m a t i o n t o
c o m p u t e r r e a d a b l e t e x t
A R T I F I C I A L I N T E L L I G E N C E
E x t r a c t c a l e n d a r
i n f o r m a t i o n f r o m p h o t o
SOLUTION

40. P r e s e n t a n i n t u i t i v e v e r i f i c a t i o n s t e p f o r u s e r s w i t h a u t o m a t i c r e t r a i n i n g
VERIFICATION

41. CASE 4
AI PRESENTATION ANALIST
In an international organisation a lot of
effort is expended in keeping the
personnel up-to-date using training
videos.
T R A I N I N G
To help train their trainers, we built an AI
system that provides feedback on their
presentation style.
A U T O M AT E F E E D B A C K
Often trainers have extensive knowledge
on their subject but are not natural born
presenters.
T R A I N T H E T R A I N E R

42. A R C H I T E C T U R E A R T I F I C I A L I N T E L L I G E N C E U S E R I N T E R F A C E
Our clever architecture only
enables the expensive cloud
components on demand. Audio
and video are separated for
further processing.
Several AI components are
combined to analyse the audio
and video of the presentation.
Present an intuitive user interface
for presenters to view the
feedback.
SOLUTION
U P L O A D V I D E O
Users can upload their videos to
our application for processing

43. U s e r s c a n v i e w t h e f e e d b a c k a b o u t t h e i r p r e s e n t a t i o n
USER INTERFACE

44. U s e r s c a n v i e w t h e f e e d b a c k a b o u t t h e i r p r e s e n t a t i o n
USER INTERFACE

45. U s e r s c a n v i e w t h e f e e d b a c k a b o u t t h e i r p r e s e n t a t i o n
USER INTERFACE

46. And what about explainability?
Explainability is mainly relevant for decision making
AI (which we haven’t been asked to build yet)
• Still early days
• Be careful about feedback to external users!
➢ Micro loan approval experiment
Source: wikipedia

48. The choice among algorithm categories can partially be made based on the user persona's
ability to intervene at different parts of a machine learning pipeline.
If the user is allowed to modify the training data, then pre-processing can be used.
If the user is allowed to change the learning algorithm, then in-processing can be used.
If the user can only treat the learned model as a black box without any ability to modify the
training data or learning algorithm, then only post-processing can be used.
AIF360 recommends the earliest mediation category in the pipeline that the user has
permission to apply because it gives the most flexibility and opportunity to correct bias as
much as possible. If possible, all algorithms from all permissible categories should be tested
because the ultimate performance depends on dataset characteristics: there is no one best
algorithm independent of dataset

50. NEED FOR A HUMAN IN THE LINK?
LET’S GET STARTED!
CONTACT US STAY IN TOUCH
info@brainjar.ai
+32 (0) 473 36 15 55
Gaston Geenslaan 11, B4
3001 Leuven
https://brainjar.ai
https://twitter.com/brainjarai
https://nl.linkedin.com/company/brainjarbe