Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie GitRecruit final 1
Ähnlich wie GitRecruit final 1 (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
GitRecruit final 1
- 1. GitRecruit Recruit the right tech talent using Github Insight Data Science Fellow: Yinghan Fu
- 2. Tech companies compete for talent • Recruiting is difficult and expensive. • Companies use Github for code repositories. • GitRecruit can automate talent search.
- 3. Algorithm-using README files to find similar repositories NLTK
- 4. Algorithm-using README files to find similar repositories NLTK -- the Natural Language Toolkit -- is a suite of open source Python modules, data sets and tutorials supporting research and development in Natural Language Processing. NLTK README
- 5. Algorithm-using README files to find similar repositories NLTK README tf-idf vector ݈ܽ݊݃݁݃ܽݑ ݄݊ݐݕ ݈݊ܽܽݎݑݐ ݐ݈ ݐ݈݅݇ݐ ⋮ 1.8 2.4 2.4 2 2 ⋮
- 6. Algorithm-using README files to find similar repositories NLTK README tf-idf vector Scikit-learn is a Python module for machine learning built on top of SciPy and distributed under the 3-Clause BSD license. The project was started in 2007 by David Cournapeau as a Google Summer of Matplotlib is a python 2D plotting library which produces publication quality figures in a variety of hardcopy formats and interactive environments across platforms. matplotlib can be used in python ~110,000 repository README files ~70% pull requests NumPy is the fundamental package needed for scientific computing with Python. This package contains: a powerful N- dimensional array object . sophisticated (broadcasting) functions ݈ܽ݊݃݁݃ܽݑ ݄݊ݐݕ ݈݊ܽܽݎݑݐ ݐ݈ ݐ݈݅݇ݐ ⋮ 1.8 2.4 2.4 2 2 ⋮
- 7. Algorithm-using README files to find similar repositories NLTK README tf-idf vector ~110,000 repository README files ~70% pull requests ݈ܽ݊݃݁݃ܽݑ ݄݊ݐݕ ݈݊ܽܽݎݑݐ ݐ݈ ݂ܿ݅݅ݐ݊݁݅ܿݏ ⋮ 0 0 0 2.5 2.3 2.8 0 0 0 0 3.1 0 5.4 4.3 6.7 ⋮ ⋮ ⋮ ݈ܽ݊݃݁݃ܽݑ ݄݊ݐݕ ݈݊ܽܽݎݑݐ ݐ݈ ݐ݈݅݇ݐ ⋮ 1.8 2.4 2.4 2 2 ⋮
- 8. Algorithm-using README files to find similar repositories ܰ ܭܶܮ 0.5 0.3 0.8 Cosine similarity
- 9. Algorithm-using README files to find similar repositories ܰ ܭܶܮ 0.5 0.3 0.8 Cosine similarity
- 10. Optimization and benchmark show the search engine is effective Database Driver Audio Text Processing Training 133 repository readme files Optimize mean average precision Testing 123 repository readme files Mean average precision 0.39 (random 0.09, p-value<0.00001) …………… …………… …………… …………… …………… …………… …………… …………… …………… …………… …………… …………… …………… …………… …………… …………… …………… …………… …………… …………… …………… …………… …………… …………… …………… …………… …………… …………… …………… …………… …………… …………… …………… …………… …………… …………… ....
- 11. About me, Yinghan Fu
- 12. About me, Yinghan Fu • GitRecruit – MySQL, Python, NLTK, scikit-learn etc. • Machine learning – C++, Python • Dynamic programming algorithm – C++ • User-user/item-item collaborative filtering – Python, mrjob, MongoDB
- 13. The search engine can concentrate python packages from the same category
- 14. Github activities are highly concentrated 16 mil repos 6 mil users 2.7 mil pull requests merged 0.4 mil repos have any pull requests merged
- 15. tf-idf vectorization ܶܨ = 1 + log (݂ݐ௧,ௗ) ܨܦܫ = log ܰ ݂݀௧
- 16. Parameters optimized • Number of words included • Minimum document frequency • Maximum document frequency • Sublinear term frequency • Cosine similarity • Maximum n-gram