There are many batch and stream scenarios in Alibaba, and many data analysts are non-technical, like to use GUI or script tool to deal with data to help business decisions. We’d like to share our experiences on developing algorithms on Apache Flink and build web UI and client to help people easily use algorithms on data analysis, training and inferencing with machine learning model.

1. Alibaba’s Common Algorithm Platform on Flink
Xu Yang
yangxu@alibaba-inc.com

2. Agenda
• Background
• Why based on Flink/Blink
• Pla7orm Introduc;on
• Demos

3. Background
• Alibaba Group
• Alibaba Compu;ng Pla7orm
• PAI ( Pla7orm of AI )

4. Why based on Flink/Blink
• More requirements on stream processing
• Advanced Flink architecture
• User
– Low learning curve
– Less coding
– More func;ons

5. • Code Name: Alink
- Common part of related words
- Alibaba, Algorithm, AI, Flink, Blink
• Current supported algorithms
- Sta;s;cs, Machine Learning, Recommenda;on, Outlier
Alibaba’s Common Algorithm Platform on Flink

6. Run)me
Distributed Streaming Dataflow
CEP
Event Processing
DataStream API
Stream Processing
DataSet API
Batch Processing
Local
Single JVM
Cloud
GCE EC2
Cluster
Standalone YARN
Table
Rela;onal
Table
Rela;onal
Flink ML
Maching Learning
Gelly
Graph Processing
Alink ML
For Streaming
Alink ML
For Batch
Alink Stat
For Streaming
Alink Stat
For Batch
Stream Operator
Stream Processing
Batch Operator
Batch Processing
Alink SDK & Web UI & Client & Visualiza)on
Processing for Structural Data
Alink ......
For Streaming
Alink ......
For Batch
Common Libs
Alink Architecture

7. Alink UI
• Web UI
– Drag-drop, easy to build workflow
• Client
– Local run
– Edit and run script
• Console
– Client without GUI

17. Alink UI
• Web UI
– Drag-drop, easy to build workflow
• Client
– Local run
– Edit and run script
• Console
– Client without GUI

19. Local Run! Cluster Run!

20. Alink UI
• Web UI
– Drag-drop, easy to build workflow
• Client
– Local run
– Edit and run script
• Console
– Client without GUI

21. Alink Functions (Part 1 of 3)
• Sta;s;cs and Visualiza;on
- Current and History
- Basic Sta;s;c
• Mean, Variance, StdVar, CV, StdErr, Moment, Central Moment, Skewness, Kurtosis
• Histogram, TopK, Bo[omK, Frequency, Percen;le, Quan;le, Median, Mode
• Covariance, Coef of Correla;on, Cross Table, Ranking List
- Sta;s;cal Analysis
• PCA, Correspondence Analysis, Mul;-collinearity
• T-Test, Chi2-Test, KS-Test, AD-Test

22. Demo for Statistics and Visualization
• IJCAI-17 Dataset
- h[ps://;anchi.aliyun.com/datalab/index.htm
- Trading amounts and loca;ons of Alipay users
- 19.6 million users, 67 million trades

23. Stat Demo: Current and History
• AllStat for History, stat from start to now
• WindowStat for Current, stat over last 3 seconds
• Trading amounts
• Frequency of shop_level

24. Data Frequent & Count Demo

25. Stat Demo: Distribution
• Get 2 stream data: shop_level=‘low’, shop_level=‘high’
• Consider 2 Features : comment_cnt and pay
• Probabilis;c Distribu;on

26. Stat Demo: Distribution

27. Stat Demo: Relationship of Features
• Numerical Features: pay, comment_cnt and shop_level_int
- Mul;collinearity, Coef of Correla;on
• Categorical Features: province and shop_level
- Correspondence Analysis, Cross Table

28. Stat Demo: Relationship of Features

29. Stat Demo: Ranking List
• provinces for user counts
• provinces for sum of pay, showed in map
• catalogs for sum of pay

30. Stat Demo: Ranking List

31. Alink Functions (Part 2 of 3)
• Data Preprocessing
– Normalize
– Standard
– FillMissingValue
– Split
– StringIndexer
– Stra;fiedSample
• Recommenda)on
– Alterna;ng Least Squares (ALS)
– Factoriza;on Machine (FM)
– ItemCF
– Simrank
• Outlier
– Stochas;c Outlier Selec;on (SOS)
– A[ribute Value Frequency (AVF)
– KSigma
– BoxPlot
– OneClassSVM
– Isola;onForest!
• Nature Language Processing
– Segment
– Tokenize
– FilterStopWords
– FeatureOnWordMap
– TFIDF
– WordRecogni;on
• Clustering
– KMeans
– Power Itera;on Clustering (PIC)
!
!

32. Alink Functions (Part 3 of 3)
• Classifica)on
– Logis;c Regression
– Linear SVM
– Perceptron
– Mul;-Class Logis;c Regression
– Mul;-Class Linear SVM
– Mul;-Class Perceptron
– Random Forest
– ID3
– C45
– CART
– Naïve Bayes
– KNN
• OnlineLearning
– FTRL
– Perceptron
– Passive Aggressive (PA)
– PA-I
– PA-II
• Regression
– Linear Regression
– Lasso Regression
– Ridge Regression
– Linear SVR
– Linear Regression Stepwise
• Others
– One Hot Encoding
– EvalClassifica;on
– EvalRegression
– MLModelPredic;on

33. Demo: Text Classification
• Dataset
- h[p://jmcauley.ucsd.edu/data/amazon/
- 142.8 million product reviews with ra;ng value (1 ~ 5)
- Task: From review content, predict ra;ng value

34. Demo: Text Classification

35. Demo: Text Classification

37. • Dataset： MNist
Demo: Handwriting Recognition

38. !
!
!
!
!
!
!
!
!
Logistic Regression!
Using Machine Learning Method
def createModel(kvColName, labelColName, predResultColName):
tensorColName = kvColName
return PipelineModel([
KvToTensorPipeOp(AlinkParameter()
.put("kvColName", kvColName)
.put("tensorColName", tensorColName)
.put("valDelimiter", ":")
.put("colDelimiter", " ")),
Mul;ClassLogistRegPipeOp(AlinkParameter()
.put("nClass", 10)
.put("tensorColName", tensorColName)
.put("labelColName", labelColName)
.put("isSparse", True)
.put("sparseFeatureDim", 28 * 28)
.put("predResultColName", predResultColName))
])
db = DerbyDB("alink_demo_db_derby_2")
trainData = DBSourceBatchOp(db, "MNIST")
testData = DBSourceBatchOp(db, "MNIST_TEST")
pm = createModel("CONTENT", "LABEL", "PRED")
pm.train(trainData)
pm.predBatch(testData)
.link(EvalClassifica;onBatchOp("LABEL", "PRED"))
.print()

39. !
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
Deep Learning Settings!
!
!
!
!
!
!
!
!
!
!
Tensor Processing!
Using Deep Learning Method (1/2)
def createModel(kvColName, labelColName, model, predColName):
tensorColName = kvColName
tempLabelColName = "TEMP_" + labelColName
tempPredColName = "TEMP_" + predColName
return PipelineModel([
KvToTensorPipeOp(AlinkParameter()
.put("kvColName", kvColName)
.put("tensorColName", tensorColName)
.put("valDelimiter", ":")
.put("colDelimiter", " ")),
TensorReshapePipeOp(AlinkParameter()
.put("selectedColName", tensorColName)
.put("outputColName", tensorColName)
.put("size", [28, 28])),
TensorStandardPipeOp(AlinkParameter()
.put("selectedColName", tensorColName)
.put("outputColName", tensorColName)
.put("mean", 0.13066047740239506)
.put("stdvar", 0.3081078)),
SelectPipeOp("(" + labelColName + ".CAST(INT) + 1).CAST(STRING) AS "
+ tempLabelColName + ", CONTENT, " + labelColName),
DLPipeOp(model, AlinkParameter()
.put("featureTensorColName", tensorColName)
.put("labelTensorColName", tempLabelColName)
.put("predResultColName", tempPredColName)
.put("batchSize", 12)
.put("maxEpoch", 2)),
TensorFunc;onPipeOp(AlinkParameter()
.put("selectedColName", tempPredColName)
.put("outputColName", predColName)
.put("funcName", "argMax")
.put("keepColNames", [labelColName]))
])

40. !
!
!
!
!
!
!
!
!
!
!
!
!
!
!
Neural Network Definition!
Using Deep Learning Method (2/2)
nnet = NN.Sequen;al()
.add(NN.Reshape([1, 28, 28]))
.add(NN.Spa;alConvolu;on(1, 6, 5, 5).setName("conv1_5x5"))
.add(NN.Tanh())
.add(NN.Spa;alMaxPooling(2, 2, 2, 2))
.add(NN.Tanh())
.add(NN.Spa;alConvolu;on(6, 12, 5, 5).setName("conv2_5x5"))
.add(NN.Spa;alMaxPooling(2, 2, 2, 2))
.add(NN.Reshape([12 * 4 * 4]))
.add(NN.Linear(12 * 4 * 4, 100).setName("fc1"))
.add(NN.Tanh())
.add(NN.Linear(100, 10).setName("fc2"))
.add(NN.LogSovMax())
db = DerbyDB("alink_demo_db_derby")
trainData = DBSourceBatchOp(db, "MNIST")
testData = DBSourceBatchOp(db, "MNIST_TEST")
pm = createModel("CONTENT", "LABEL", nnet, "PRED")
pm.train(trainData)
pm.predBatch(testData)
.link(EvalClassifica;onBatchOp("LABEL", "PRED"))
.print()

41. Demo: Recommendation of Movies
• Dataset:
– MovieLens 20M Dataset
– 20 million ra;ngs; 27,000 movies; 138,000 users
– Ra;ng value range: 0.5 ~ 5
• Task
– Recommend movies for each user
• Algorithm:
– ALS (Alterna;ng Least Square)

42. Batch and Stream Recommendation

43. Rated Movies of User(id=100)

44. Recommend for User(id=100)

45. Rated Movies of User(id=100041)

46. Recommend for User(id=100041)

47. Top 10 Recommended Movies

48. Demo: Domain Name Outlier
• Dataset
- h[ps://github.com/
andrewaeva/DGA

49. Demo: Domain Name Outlier
• Using AGD Detec;on Algorithm

50. hgy2.mp4