The document provides information about Microsoft's Cognitive Toolkit (CNTK), including benchmark performance comparisons with other deep learning frameworks and examples of using CNTK for common neural network architectures and natural language processing tasks. It shows that CNTK achieves state-of-the-art performance and scales nearly linearly with multiple GPUs. The document also provides code examples for defining common neural network components and training models with CNTK.

13.  https://github.com/Microsoft/CNTK

16. Caffe Cognitive Toolkit MxNet TensorFlow Torch
FCN5 (1024) 55.329ms 51.038ms 60.448ms 62.044ms 52.154ms
AlexNet (256) 36.815ms 27.215ms 28.994ms 103.960ms 37.462ms
ResNet (32) 143.987ms 81.470ms 84.545ms 181.404ms 90.935ms
LSTM (256)
(v7 benchmark)
- 43.581ms
(44.917ms)
288.142ms
(284.898ms)
-
(223.547ms)
1130.606ms
(906.958ms)
http://dlbench.comp.hkbu.edu.hk/
Benchmarking by HKBU, Version 8
Single Tesla K80 GPU, CUDA: 8.0 CUDNN: v5.1
Caffe: 1.0rc5(39f28e4)
CNTK: 2.0 Beta10(1ae666d)
MXNet: 0.93(32dc3a2)
TensorFlow: 1.0(4ac9c09)
Torch: 7(748f5e3)

17. 2 only supports 1 GPU
Achieved with 1-bit gradient quantization
algorithm
0
10000
20000
30000
40000
50000
60000
70000
80000
1 2 3 4 5
speed comparison (samples/second), higher = better
[note: December 2015]
Series1 Series2 Series3

18. MICROSOFT COGNITIVE TOOLKIT
First Deep Learning Framework Fully Optimized for Pascal
78
2,400
3,500
7,600
13,000
0
2,000
4,000
6,000
8,000
10,000
12,000
14,000
1 2 3 4 5
Toolkit Delivering Near-Linear Multi-GPU Scaling
AlexNet Performance
images/sec
AlexNet training batch size 128, Grad Bit = 32, Dual socket E5-2699v4 CPUs (total 44 cores)
CNTK 2.0b3 (to be released) includes cuDNN 5.1.8, NCCL 1.6.1, NVLink enabled
170x Faster
v. CPU Server

21. $ pip install <url>
https://docs.microsoft.com/en-us/cognitive-toolkit/Setup-CNTK-on-your-machine

23. https://notebooks.azure.com/cntk/libraries/tutorials

28. Example: 2-hidden layer feed-forward NN
h1 = s(W1 x + b1) h1 = sigmoid (x @ W1 + b1)
h2 = s(W2 h1 + b2) h2 = sigmoid (h1 @ W2 + b2)
P = softmax(Wout h2 + bout) P = softmax (h2 @ Wout + bout)
with input x  RM
and one-hot label L  RM
and cross-entropy training criterion
ce = LT
log P ce = cross_entropy (L, P)
Scorpusce = max

29. Example: 2-hidden layer feed-forward NN
h1 = s(W1 x + b1) h1 = sigmoid (x @ W1 + b1)
h2 = s(W2 h1 + b2) h2 = sigmoid (h1 @ W2 + b2)
P = softmax(Wout h2 + bout) P = softmax (h2 @ Wout + bout)
with input x  RM
and one-hot label y  RJ
and cross-entropy training criterion
ce = yT
log P ce = cross_entropy (L, P)
Scorpusce = max

30. example: 2-hidden layer feed-forward NN
h1 = s(W1 x + b1) h1 = sigmoid (x @ W1 + b1)
h2 = s(W2 h1 + b2) h2 = sigmoid (h1 @ W2 + b2)
P = softmax(Wout h2 + bout) P = softmax (h2 @ Wout + bout)
with input x  RM
and one-hot label y  RJ
and cross-entropy training criterion
ce = yT
log P ce = cross_entropy (P, y)
Scorpusce = max

31. h1 = sigmoid (x @ W1 + b1)
h2 = sigmoid (h1 @ W2 + b2)
P = softmax (h2 @ Wout + bout)
ce = cross_entropy (P, y)

32. •
+
s
•
+
s
•
+
softmax
W1
b1
W2
b2
Wout
bout
cross_entropy
h1
h2
P
x y
h1 = sigmoid (x @ W1 + b1)
h2 = sigmoid (h1 @ W2 + b2)
P = softmax (h2 @ Wout + bout)
ce = cross_entropy (P, y)
ce

33. •
+
s
•
+
s
•
+
softmax
W1
b1
W2
b2
Wout
bout
cross_entropy
h1
h2
P
x y
ce
LEGO-like composability allows CNTK to support
wide range of networks & applications

34. Script configure and executes through CNTK Python APIs…
trainer
• SGD
(momentum,
Adam, …)
• minibatching
reader
• minibatch source
• task-specific
deserializer
• automatic
randomization
• distributed
reading
corpus model
network
• model function
• criterion function
• CPU/GPU
execution engine
• packing, padding

35. from cntk import *
# reader
def create_reader(path, is_training):
...
# network
def create_model_function():
...
def create_criterion_function(model):
...
# trainer (and evaluator)
def train(reader, model):
...
def evaluate(reader, model):
...
# main function
model = create_model_function()
reader = create_reader(..., is_training=True)
train(reader, model)
reader = create_reader(..., is_training=False)
evaluate(reader, model)

37. def create_reader(map_file, mean_file, is_training):
# image preprocessing pipeline
transforms = [
ImageDeserializer.crop(crop_type='Random', ratio=0.8, jitter_type='uniRatio')
ImageDeserializer.scale(width=image_width, height=image_height,
channels=num_channels,
interpolations='linear'),
ImageDeserializer.mean(mean_file)
]
# deserializer
return MinibatchSource(ImageDeserializer(map_file, StreamDefs(
features = StreamDef(field='image', transforms=transforms), '
labels = StreamDef(field='label', shape=num_classes)
)), randomize=is_training, epoch_size = INFINITELY_REPEAT if is_training else
FULL_DATA_SWEEP)

38. def create_reader(map_file, mean_file, is_training):
# image preprocessing pipeline
transforms = [
ImageDeserializer.crop(crop_type='Random', ratio=0.8, jitter_type='uniRatio')
ImageDeserializer.scale(width=image_width, height=image_height,
channels=num_channels,
interpolations='linear'),
ImageDeserializer.mean(mean_file)
]
# deserializer
return MinibatchSource(ImageDeserializer(map_file, StreamDefs(
features = StreamDef(field='image', transforms=transforms), '
labels = StreamDef(field='label', shape=num_classes)
)), randomize=is_training, epoch_size = INFINITELY_REPEAT if is_training else
FULL_DATA_SWEEP)

39. Model
z = model(x):
h1 = Dense(400, act = relu)(x)
h2 = Dense(200, act = relu)(h1)
r = Dense(10, act = None)(h2)
return r
Loss
cross_entropy_with_softmax(z,Y)

40. 28 pix
28pix
Model
z = model(x):
h = Convolution2D((5,5),filt=8, …)(x)
h = MaxPooling(…)(h)
h = Convolution2D ((5,5),filt=16, …)((h)
h = MaxPooling(…)(h)
r = Dense(output_classes, act= None)(h)
return r

41. Problem: Tagging entities in Air Traffic Controller (ATIS) data
Rec
show
o
Rec
burbank
From_city
Rec
to
o
Rec
seattle
To_city
Rec
flights
o
Rec
tomorrow
Date
http://karpathy.github.io/2015/05/21/rnn-effectiveness/

42. 0 0 1 0
Ԧ𝑥(t)
E
i = 943
O= 150
L
i = 150
O= 300
D
i = 300
O= 129
a =
sigmoid
ℎ(t-1) ℎ(t)
Ԧ𝑦(t)
Ԧ𝑥(t)
0 0 1 0
Text token
L
E
Ԧ𝑦(t)Class label
D
1 x 943
z = model():
return
Sequential([
Embedding(emb_dim=150),
Recurrence(LSTM(hidden_dim=300),
go_backwards=False),
Dense(num_labels = 129)
])

43. lr_schedule = C.learning_rate_schedule([0.05]*3 + [0.025]*2 + [0.0125],
C.UnitType.minibatch, epoch_size=100)
sgd_learner = C.sgd(z.parameters, lr_schedule)

44. ATIS
Train
96samples
(mini-batch)
.
.
.
.
#1
#2
#3
#96
Input feature ( 96 x Ԧ𝑥(t))
z = model():
return
Sequential([
Embedding(emb_dim=150),
Recurrence(LSTM(hidden_dim=300),
go_backwards=False),
Dense(num_labels = 129)
])
Loss cross_entropy_with_softmax(z,Y)
Trainer(model, (loss, error), learner)
Trainer.train_minibatch({X, Y})
Error classification_error(z,Y)
Choose a learner
(SGD, Adam, adagrad etc.)
One-hot
encoded
Label
(Y: 96 x
129/sample
Or word in
sequence)
t23t1
t1
t1
t1
t15
t9
t12

47. Function z = CNTKLib.Times(weightParam, input) + biasParam;
Function loss = CNTKLib.CrossEntropyWithSoftmax(z, labelVariable);
Function conv = CNTKLib.Pooling(CNTKLib.Convolution(convParam, input),
PoolingType.Average, poolingWindowShape);
Function resNetNode = CNTKLib.ReLU(CNTKLib.Plus(conv, input));

48. var parameterLearners = new List<Learner>() {
Learner.AdamLearner(classifierOutput.Parameters(), learningRate, momentum) };
var trainer = Trainer.CreateTrainer(classifierOutput, trainingLoss, prediction,
parameterLearners);

49. Function conv = CNTKLib.ReLU(CNTKLib.Convolution(convParams, features, strides ));
Function pooling = CNTKLib.Pooling(conv, PoolingType.Max, poolingWindow, stride, padding);
Function classifier = TestHelper.Dense(pooling, numClasses, device, Activation.None);
var minibatchSource =
MinibatchSource.TextFormatMinibatchSource("Train_cntk_text.txt"),
streamConfigurations, MinibatchSource.InfinitelyRepeat);

50. var minibatchData = minibatchSource.GetNextMinibatch(minibatchSize, device);
var arguments = new Dictionary<Variable, MinibatchData> {
{ input, minibatchData[featureStreamInfo] },
{ labels, minibatchData[labelStreamInfo] }
};
trainer.TrainMinibatch(arguments, device);

51. https://github.com/Microsoft/CNTK/tree/master/Examples/TrainingCShar
p

54. Accelerating adoption of AI by developers
(consuming models)
Rise of hybrid training and scoring scenarios
Push scoring/inference to the event (edge,
cloud, on-prem)
Some developers moving into deep learning as
non-traditional path to DS / AI dev
Growth of diverse hardware arms race across all
form factors (CPU / GPU / FPGA / ASIC /
device)
Data prep
Model deployment &
management
Model lineage & auditing
Explain-ability
D ATA S C I E N C E & A I
C H A L L E N G E SK E Y T R E N D S

55. Challenge
• Traditional power line inspection services are
costly
• Demand for low cost image scoring and support
for multiple concurrent customers
• Needed powerful AI to execute on a drone
solution
Solution
• Deep learning to analyze multiple streaming data
feeds
• Azure GPUs support Single Shot multibox
detectors
• Reliable, consistent, and highly elastic scalability
with Azure Batch Shipyards
Drone-based electric grid
inspector powered by
deep learning

56. snow
leopard?
Deep neural network Spark ML classifier
Decision tree or logistic
regression
Image featuresImage
Class 1 Class 1
Gap
Identifying Snow Leopards
Computer vision and classification on Spark

57. Apps + insights
Social
LOB
Graph
IoT
Image
CRM INGEST STORE PREP & TRAIN MODEL & SERVE
Data orchestration
and monitoring
Data lake
and storage
Hadoop/Spark/SQL
and ML
.
IoT
Azure Machine Learning
T H E A I D E V E L O P M E N T L I F E C Y C L E

58. Azure Machine Learning Studio
Platform for data scientists to graphically
build and deploy experiments
• Rapid experiment composition
• > 100 easily configured modules for
data prep, training, evaluation
• Extensibility through R & Python
• Serverless training and deployment
Some numbers:
• 100’s of thousands of deployed models
serving billions of requests

60. Begin building now with the
tools and platforms you know
Build, deploy, and
manage models at
scale
Boost productivity with
agile development

61. Notebooks
IDEs
Azure Machine Learning
Workbench
VS Code Tools for AI
N E W C A PA B I L I T I E S
Experimentation and
Model Management
Services
AZURE MACHINE
LEARNING SERVICES
Spark
SQL Server
Virtual
machines
GPUs
Container
services
SQL Server
Machine Learning
Server
ON-PREMISES
EDGE
Azure IoT Edge
TRAIN & DEPLOY
OPTIONS
AZURE

66. Local machine
Scale up to DSVM
Scale out with Spark on HDInsight
Azure Batch AI (Coming Soon)
ML Server
Experiment Everywhere
A ZURE ML
EXPERIMENTATION
Command line tools
IDEs
Notebooks in Workbench
VS Code Tools for AI

67. Manage project dependencies
Manage training jobs locally, scaled-up or
scaled-out
Git based checkpointing and version control
Service side capture of run metrics, output logs
and models
Use your favorite IDE, and any framework
Experimentation service
U S E T H E M O S T P O P U L A R I N N O VAT I O N S
U S E A N Y TO O L
U S E A N Y F R A M E W O R K O R L I B R A R Y

68. DOCKER
Single node deployment
(cloud/on-prem)
Azure Container Service
Azure IoT Edge
Microsoft ML Server
Spark clusters
SQL Server
Deploy Everywhere
A ZURE ML
MODEL MANAGEMENT

69. Deployment and management of models as HTTP
services
Container-based hosting of real time and batch
processing
Management and monitoring through Azure
Application Insights
First class support for SparkML, Python, Cognitive
Toolkit, TF, R, extensible to support others (Caffe,
MXnet)
Service authoring in Python
Manage models

70. AI Powered Spreadsheets

71. VS Code extension with deep integration to Azure
ML
End to end development environment, from new
project through training
Support for remote training
Job management
On top of all of the goodness of VS Code
(Python, Jupyter, Git, etc)
VS Code Tools for AI

72. Windows and Mac based
companion for AI development
Full environment set up (Python,
Jupyter, etc)
Embedded notebooks
Run History and Comparison
experience
New data wrangling tools
Azure Machine Learning
Workbench - What Is It?

73. AI Powered Data Wrangling
Rapidly sample, understand, and
prep data
Leverage PROSE and more for
intelligent, data prep by example
Extend/customize transforms and
featurization through Python
Generate Python and Pyspark for
execution at scale

75. Machine Learning & AI Portfolio
When to use what?
What engine(s) do you want
to use?
Deployment target
Which experience do you
want?
Build your own or consume pre-
trained models?
Microsoft
ML & AI
products
Build your
own
Azure Machine Learning
Code first
(On-prem)
ML Server
On-
prem
Hadoop
SQL
Server
(cloud)
AML services (Preview)
SQL
Server
Spark Hadoop Azure
Batch
DSVM Azure
Container
Service
Visual tooling
(cloud)
AML Studio
Consume
Cognitive services, bots

77. http://aka.ms/aml_deep_dive

78. https://aischool.microsoft.com/learning-paths/SPYpcLhRMyEAa2maw6YoU
https://channel9.msdn.com/events/Ignite/Microsoft-Ignite-Orlando-2017/BRK4033
https://channel9.msdn.com/events/Ignite/Microsoft-Ignite-Orlando-2017/BRK2270
https://www.microsoft.com/en-us/cognitive-toolkit/
https://azure.microsoft.com/services/machine-learning-services/
https://azure.microsoft.com/services/virtual-machines/data-science-virtual-machines/