Developing and validating a document classifier: a real-life story - Marko Smiljanic
•Als PPTX, PDF herunterladen•
0 gefällt mir•216 views
A department, somewhere in EU, depends on having a steady input of 3000 new textual documents per day, 365 days a year. Documents come from 10 different sources and each document comes pre-classified into a single category of a large taxonomy. The department is unhappy: the accuracy of incoming document classifications seems to be low. Even after the department puts additional 800% FTE throughout the year to manually repair or discard wrongly classified documents, the accuracy still lags behind their targets. NIRI was hired to conduct a research and develop an accurate document classifier. The plan was to use NIRI’s classifier to replace the unreliable classes coming with documents, and thus solve the problem of low accuracy, as well as reduce the high cost of 800% FTE. In this talk we will share our experiences: classification approach used to meet the needs of our client, challenges in demonstrating progress during the project, and the approach used for the acceptance-validation of our classifier.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Andere mochten auch
Andere mochten auch (15)
Ähnlich wie Developing and validating a document classifier: a real-life story - Marko Smiljanic
Ähnlich wie Developing and validating a document classifier: a real-life story - Marko Smiljanic (20)
Mehr von Institute of Contemporary Sciences
Mehr von Institute of Contemporary Sciences (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Developing and validating a document classifier: a real-life story - Marko Smiljanic
- 1. Marko Smiljanić, NIRI Inteligent computing Ltd, CEO Developing and validating a document classifier: a real-life story
- 2. Developing and validating a document classifier: a real-life story Marko Smiljanić, CEO www.niri-ic.com
- 3. About us. NIRI: 10 years in Intelligent Computing Text Mining Knowledge Discovery and Management All about Data Science
- 5. Business Context The Challenge The Solution Effectiveness Laboratory measurements Impact estimation Reality Wrap up The flow
- 7. Business context Largest clients include Public Employment Services in EU, USA, and Asia Staffing companies in EU, USA
- 9. Business Context The Challenge The Solution Effectiveness Laboratory measurements Impact estimation Reality Wrap up The flow
- 11. Occupation Taxonomies ISCO (International Standard Classification of Occupations) ESCO O*NET and many more ISCO level 1 (10) ISCO level 2 (42) ISCO level 3 (124) ISCO level 4 (400) ESCO level 5 (5000) “Delivery service worker” Challenges (for humans) Knowing the taxonomy Ambiguous taxonomy Hybrid positions Vague vacancy
- 12. Client’s situation in 2014 Vacancy Aggregator and Classifier Correct Code? Publish Repair Code!NO 23% ОК 65% nohelp 14% OK 9% nocode 12% 2000-4000 per day (into >2000 taxonomy classes)
- 13. Business Context The Challenge The Solution Effectiveness Laboratory measurements Impact estimation Reality Wrap up The flow
- 14. The Solution: NIRI will build you a better classifier Vacancy Aggregator and Classifier NIRI Classifier Publish2000-4000 per day
- 15. Really? How accurate will it be? How will it fit our process? Reduce manual effort Increase volume Improve final accuracy Really. We will (try to):
- 16. But you need to give us training data > 1M vacancies No class 12% Not verified 14% Verified 74%
- 17. Long tail effect
- 18. Architecture of our solution Feature Extractor Negotiator Classifier 1 Classifier 2 Classifier N … Vacancy [Class, Confidence]+ Vacancy Classifier External Services
- 19. What to do with confidence? Vacancy, Code, Confidence Vacancy, Code, Confidence Vacancy, Code, Confidence Vacancy, Code, Confidence Vacancy, Code, Confidence Vacancy, Code, Confidence Vacancy, Code, Confidence Vacancy, Code, Confidence Vacancy, Code, Confidence Vacancy, Code, Confidence Vacancy, Code, Confidence … To check manualy Batch Processing CONFIDENCE High accuracy Low accuracy
- 20. Using confidence
- 21. Business Context The Challenge The Solution Effectiveness Laboratory measurements Impact estimation Reality Wrap up The flow
- 22. Measuring accuracy in the laboratory No class 12% Not verified 14% Verified 74% No class Incorrect Correct Test 20% Train 80% Train Test x 5 Vacancy Classifier
- 23. 74% 78% 80% 85% 14% 13% 12% 10% 12% 9% 8% 5% CORPUS CLASSIFIER CLASSIFIER 100 CLASSIFIER 1000 ONE OF MANY LABORATORY MEASUREMENTS Correct Incorrect No class Measuring accuracy in the laboratory
- 24. Measuring accuracy in the laboratory No class 12% Not verified 14% Verified 74% Vacancy Classifier No class 9% Incorrect 13% Correct 78% Original Classifier This is not relaity Biased train/test set Accuracy of test set unknown Inability to test against 26%
- 25. Business Context The Challenge The Solution Effectiveness Laboratory measurements Impact estimation Reality Wrap up The flow
- 27. This is what it actually looks like. Check Repair Reduce manual effort Increase volume Improve final accuracy We will
- 28. And we proposed this one. Bulk Accept Check Repair
- 29. Best/worst case analysis, some manual validation, careful assumptions: Bulk Accept Check Repair
- 30. Impact estimation showed that: Step 1 effort reduction 60% (due to bulk acceptance) Step 2 effort reduction 11% (due to bulk acceptance and top 5 offers) Significant published volume increase (almost to 100%) Accuracy slightly larger (+1%, to around 92%)
- 31. Business Context The Challenge The Solution Effectiveness Laboratory measurements Impact estimation Reality Wrap up The flow
- 32. No class 12% Not verified 14% Verified 74% How can we measure production accuracy? We can not, unless…
- 33. Golden Test Set
- 34. How was it built? Check & Repair 4 eye principle Vacancy Classifier Published Original Code & Top 5 VC codes Original Code & Top 5 VC codes Original Code & Top 5 VC codes Every single classification was marked as either Correct, Acceptable, or Wrong
- 35. Results 63.05% 73.91% 72.06% 74.38% 65.98% 77.56% 76.25% 78.69% CURRENT NIRI VC CURRENT (HQ SOURCE) NIRI VC (HQ SOURCE) GOLDEN TEST SET RESULTS Correct Acceptable Highest Quality Source (Training)
- 36. Business Context The Challenge The Solution Effectiveness Laboratory measurements Impact estimation Reality Wrap up The flow
- 37. Wrap up Clean semantic data, in real-life, can only be a myth. We are looking into data cleansing approaches. Measuring usefulness can be hard and expensive, but … … it can/must to be monitored after the system is deployed. It changes over time. Continuous learning, where possible is a great thing. 1) Implementing state-of-the-art machine learning algorithm is one thing. 2) Making it useful is another. 3) Explaining that to the end-user is the third. NIRI is a very cool company to work with! I hope you liked the story, and I thank you for your attention.
- 38. Developing and validating a document classifier: a real-life story Marko Smiljanić, CEO www.niri-ic.com
Hinweis der Redaktion
- FIXME Write about challenges.. Size ambiguiotz, semantics, multi faceted vacanc ies