微軟大數據整合解決方案 cortana intelligence suite 概述

2. 關於我…
• 建國中學、輔仁大學應用數學系、交通大學工業工程研究所。1999年於南加大取得 M.S.
Degree in Computer Science 後，留美由一間小新創公司的軟體工程師開始職涯。後歷任 US.
Interactive Inc. 資深工程師、Sierra Systems 技術主管。
• 2005 返台並加入台灣微軟，曾任技術中心 (MTC) 資深架構師、開發體驗暨平台推廣事業部
(DX) 資深協理，現任大中華區 Azure CAT 高級項目經理，提供企業公有雲實作最佳解決方案
及微軟產品規劃。
• 具多年協助大型企業及新創公司導入新一代技術平台、及技術行銷之經驗，除 Azure 公有雲解
決方案外，於企業搜尋解決方案、企業單一入口及 Windows Apps、Office Apps 等亦有涉獵。

5. Platform Services
Infrastructure Services
Web
Apps
Mobile
Apps
API
Apps
Notification
Hubs
Hybrid
Cloud
Backup
StorSimple
Azure Site
Recovery
Import/Export
SQL
Database DocumentDB
Redis
Cache
Azure
Search
Storage
Tables
SQL Data
Warehouse
Azure AD
Health Monitoring
AD Privileged
Identity
Management
Operational
Analytics
Cloud
Services
Batch
RemoteApp
Service
Fabric
Visual Studio
Application
Insights
VS Team Services
Domain Services
HDInsight Machine
Learning Stream Analytics
Data
Factory
Event
Hubs
Data Lake
Analytics Service
IoT Hub
Data
Catalog
Security &
Management
Azure Active
Directory
Multi-Factor
Authentication
Automation
Portal
Key Vault
Store/
Marketplace
VM Image Gallery
& VM Depot
Azure AD
B2C
Scheduler
Xamarin
HockeyApp
Power BI
Embedded
SQL Server
Stretch Database
Mobile
Engagement
Functions
Cognitive Services Bot Framework Cortana
Security Center
Container
Service
VM
Scale Sets
Data Lake Store
BizTalk
Services
Service Bus
Logic
Apps
API
Management
Content
Delivery
Network
Media
Services
Media
Analytics

6. 主題：將資料轉換為智能及決策
Intelligence
Dashboards &
Visualizations
Information
Management
Big Data Stores Machine Learning
and Analytics
CortanaEvent Hubs
HDInsight
(Hadoop and
Spark)
Stream
Analytics
Data Intelligence Action
People
Automated
Systems
Apps
Web
Mobile
Bots
Bot
Framework
SQL Data
WarehouseData Catalog
Data Lake
Analytics
Data Factory
Machine
Learning
Data Lake Store
Cognitive
Services
Power BI
Data
Sources
Apps
Sensors
and
devices
Data

7. Vehicle Telemetry Analytics 車輛遙測收集及分析
https://channel9.msdn.com/Shows/Azure-Friday/Cortana-Analytics-Vehicle-Telemetry-Analytics-Solution-Template

8. Information Management 資訊管理
Data
Sources
Apps
Sensors
and devices
Data
Information
Management
Event Hubs
Data Catalog
Data Factory

9. Compose and orchestrate data services at scale
大數據收集之協調整合
INGEST
SQL
<>
SQL
DATA SOURCES
{ }
SQL
• 建立、排程、組織及管理資料管線
• 顯現資料歷程
• 連接到企業內部原生資料庫或雲端資料來源
• 監視資料管線健全狀態
• 自動化雲端資源管理
• 轉換及分析：如半結構化資料加入 Hadoop 處理轉換為 Hive, Pig,
或自訂程式碼
Information
Management
Event Hubs
Data Catalog
Data Factory

10. Get more value from your enterprise data assets
資料目錄
Information
Management
Event Hubs
Data Catalog
Data Factory
• 註冊企業資料資產
• 快速探索資料資產，縮短找尋資料的時間
• 讓資料存留在預期的位置；選擇您的連接工具
• 控制可探索註冊之資料資產的使用者
• 利用開放的 REST API 融入現有工具和程序
• 降低在資料生態系統使用者之間的屏障，弭平 IT 和業務之間的間隙

11. Ingest events from websites, apps and devices at cloud scale
事件中樞- 從網站、應用程式和裝置擷取雲端等級的遙測數據
• 每秒，即時，記錄數百萬個事件 (AMQP & HTTPS)
• 靈活的授權與節流 (throttling) 來連接裝置
• 使用以時間為基礎的事件緩衝 (time-based event buffering)
• 彈性化的規模 scale up and down
• 使用原生用戶端程式庫，以連接最廣泛的平台 (.NET, Java,
Rest)
• 支援多種雲端服務隨插即用的配接器 (ex: NiFi, ELK stack, etc.)
Azure
API
Management
Backend Services
Data
Information
Management
Event Hubs
Data Catalog
Data Factory
Data
sources
Apps
Sensors
and
devices
Event Hubs
SQL Database Machine Learning
HDInsightStorage
Power BIStream Analytics

12. Big Data Stores 存放海量資料，並視需求擴充
Big Data Stores
SQL Data
Warehouse
Data Lake Store
Data
Sources
Apps
Sensors
and devices
Data
Information
Management
Event Hubs
Data Catalog
Data Factory

13. A hyper-scale repository for big data analytics workloads
支援智慧行動的無限制 Data Lake
• 支援使用開放 Apache Hadoop 分散式檔案系統 (HDFS) 標準的任何應
用程式
• 不限資料大小，可以儲存數兆個檔案，單一檔案的大小可大於 1 PB，
這比其他雲端存放區大 200 倍
• 適合用來儲存任何類型的資料，如高解析度視訊等大規模資料集、基因
和地震資料集、醫學資料等
• 調整輸送量以支援大規模平行處理分析
• 永遠加密、角色型安全性與稽核：內建單一登入 (SSO)、多
重要素驗證及流暢管理數百萬個身分識別等功能。使用角色
型存取控制 (RBAC)，將 POSIX 型 ACL 授權給使用者和群組
LOB
Applications
SocialDevices
Clickstream
Sensors
Video
Web
Relational
HDInsight
ADL Analytics
Machine Learning
Spark
R
ADL Store
Big Data Stores
SQL Data
Warehouse
Data Lake Store

14. SQL Data Warehouse
SQL 資料倉儲
• 結合 SQL Server 關聯式資料庫與 Azure 雲端相應放大功能。
• 在數秒內增加或減少存儲空間、暫停或繼續計算
• 實證經驗，在 6 億個使用者帳戶間支援 13 億筆每日AAD驗證
• 內建 Microsoft PolyBase 技術，可簡化並使用分散式分析，讓
您透過熟悉的工具在多個資料來源間執行單一 T-SQL 查詢
• 與 SQL Server Integration Services、Azure Analysis Services、
Azure 串流分析、Azure Machine Learning、Azure Data
Factory 及 Azure 儲存體的無縫整合
Power BI
App Service
SQL Database
SQL Data Warehouse
Machine Learning
Hadoop
Intelligent App
Big Data Stores
SQL Data
Warehouse
Data Lake Store

15. Machine Learning and Analytics
預測結果及制定決策
Big Data Stores
SQL Data
Warehouse
Data Lake Store
Data
Sources
Apps
Sensors
and devices
Data Intelligence
Information
Management
Event Hubs
Data Catalog
Data Factory
Machine Learning
and Analytics
HDInsight
(Hadoop and
Spark)
Stream
Analytics
Data Lake
Analytics
Machine
Learning

16. Easily build, deploy, and share predictive analytics solutions
無論是否資料科學家，快速建立、佈署並分享預測分析方案
• 提供多種強固學習演算法模組、訓練資料及視覺化檢驗
• 支援 R & Python 及數百個 CRAN 套件
• 預測成果以 Web Service 佈署於雲端，享受可擴充性及穩定性，支援單次或批次呼叫
Machine Learning
and Analytics
HDInsight
(Hadoop and
Spark)
Stream
Analytics
Data Lake
Analytics
Machine
Learning

17. 如果搭上了鐵達尼號，會生還嗎?
http://demos.datasciencedojo.com/demo/titanic/

18. 發佈為 Web Service https://docs.microsoft.com/zh-tw/azure/machine-learning/machine-learning-excel-add-in-for-
web-services

19. 機器學習演算法小祕技 https://docs.microsoft.com/zh-tw/azure/machine-learning/machine-learning-algorithm-cheat-sheet

20. Data Lake Analytics
以 U-SQL、R、Python 和 .Net 輕鬆開發及執行大規模平行轉換和處理資料的程
式，來處理 PB 規模的資料
• 基於 YARN 架構的公有雲平台服務
• 隨選服務，幾秒內即可開始處理巨量資料作業
• 使用您熟悉的技術：U-SQL, Spark, Hive, HBase and Storm
• 處理 PB 規模的資料。無需管理基礎結構，您可以視需要處理資料、立即調
整規模，並只按各項作業付費
• 能夠使用 Azure SQL Database 和 Azure SQL 資料倉儲等關聯式來源的最佳
化資料視覺效果
• 提供 99.9% SLA 的支援
Data Lake Analytics
SQL DW SQL DB Storage BlobsData Lake Store SQL DB in a VM
Machine Learning
and Analytics
HDInsight
(Hadoop and
Spark)
Stream
Analytics
Data Lake
Analytics
Machine
Learning

21. Comprehensive set of managed Apache big data projects
企業用雲端 Spark 和 Hadoop 服務
• 唯一受完整管理的雲端 Hadoop 產品，為 Spark、Hive、
MapReduce、HBase、Storm、Kafka 和 R 伺服器提供最佳化開放
原始碼分析叢集
• 支援 Visual Studio、Eclipse 和 IntelliJ 等開發環境
• 透過整合兩個最熱門的筆記本 Jupyter 和 Zeppelin 來解析資料
• 支援 Linux 及 Windows，數秒之內完成佈署，不需要修補作業系統或更新
Hadoop 版本，Azure 會為您代勞
• R 伺服器中的多執行緒數學程式庫與透明的並行作業，最多可處理比開放
始碼 R 多 1000 倍的資料，且速度快 50 倍
• 透過低耦合計算及儲存體，相應增加或減少工作負載以符合成本效益
Core Engine
Batch
Map Reduce
Script
Pig
SQL
Hive
NoSQL
HBase
Streaming
Storm
In-Memory
Spark
Machine Learning
and Analytics
HDInsight
(Hadoop and
Spark)
Stream
Analytics
Data Lake
Analytics
Machine
Learning

22. Machine Learning
and Analytics
HDInsight
(Hadoop and
Spark)
Stream
Analytics
Data Lake
Analytics
Machine
Learning
Real-time stream processing in the cloud
串流分析
• 即時分析資料流，例如 IoT 方案
• 每秒串流處理數百萬個事件
• 串流分析以高輸送量處理資料，產生可預測的結果，而且不會遺失
資料
• 為來自裝置與應用程式的資料，建立即時儀表板與警示
• 相互關聯多個資料串流
• 使用熟悉的 SQL 語言，加快開發的速度
Event Hubs
Blob Storage
Stream
Analytics
SQL Database
Event Hubs
Power BI
Blob Storage
Table Storage

24. Demo: Cortana Intelligence Gallery https://gallery.cortanaintelligence.com

25. Intelligence
智能
Intelligence
Cortana
Bot
Framework
Cognitive
Services
Big Data Stores
SQL Data
Warehouse
Data Lake Store
Data
Sources
Apps
Sensors
and devices
Data
Information
Management
Event Hubs
Data Catalog
Data Factory
Machine Learning
and Analytics
HDInsight
(Hadoop and
Spark)
Stream
Analytics
Data Lake
Analytics
Machine
Learning

30. microsoft.com/cognitive

31. DEMO
microsoft.com/cognitive

32. Roll your own with REST APIs
Simple to add: just a few lines of
code required
Integrate into the language and
platform of your choice
Breadth of offerings helps you find the
right API for your app
Built by experts in their field from
Microsoft Research, Bing, and Azure
Machine Learning
Quality documentation, sample
code, and community support
Easy Flexible Tested
GET A
KEY

33. https://www.youtube.com/watch?v=aEBi4OpXU4Q

34. 2015 System
Human Error Rate 4%
Speech Recognition could reach human parity in the next 3 years

37. microsoft.com/cognitive

38. Your bots – wherever your users converse
Intelligence
Cortana
Bot
Framework
Cognitive
Services
• Bot Connector Service: A service to register your bot, configure channels and publish to the Bot Directory. Connect your bot(s) seamlessly to
text/sms, Office 365 mail, Skype, Slack, Twitter, and more.
• Bot Builder SDK: An open source SDK hosted on GitHub. Everything you need to build great dialogs within your Node.js or C# bot
• Bot Directory: A public directory of bots registered through the Bot Connector Service. Discover, try, and add bots to conversation experiences
https://dev.botframework.com/

40. Microsoft Bot Builder (SDKs)
Microsoft Bot Directory
Knowledge & Intelligence Services
VisionWeb Search
Language Speech Knowledge
…ML
Public APIs (Cognitive Services)
Dialog Manager
Knowledge &
Action Graph Entity
Private APIs
Microsoft Bot Connector 4
3
2
1
Add smarts to your bot
Build a great bot
Make your bot discoverable
Connect your bot to channels
ConversationasaPlatform
Conversation as a Platform

42. https://qnamaker.ai

44. Dashboards & Visualizations
將資料轉換成易於了解的、互動式的視覺圖像
Dashboards &
Visualizations
Power BI
IntelligenceInformation
Management
Big Data Stores Machine Learning
and Analytics
CortanaEvent Hubs
HDInsight
(Hadoop and
Spark)
Stream
Analytics
Data Intelligence
Bot
Framework
SQL Data
WarehouseData Catalog
Data Lake
Analytics
Data Factory
Machine
Learning
Data Lake Store
Cognitive
Services
Data
Sources
Apps
Sensors
and
devices
Data

45. Keep a pulse on your business with live, interactive dashboards
將資料轉換為互動、即時的圖表，隨時隨地觀看
Event Hubs
Stream Analytics
Machine Learning
Storage
SQL database
HDInsight
Power BI
Power BI
• Power BI 儀表板為商務使用者設計，無需IT背景
• 隨時隨地透過行動裝置即時存取
• 提供多種 2D/3D 圖表，互動、即時
• 亦可將報表內嵌到您既有網頁中
• 整合來自Excel 試算表、內部資料、Hadoop 資料集、資料
流和雲端服務的資料源
Power BI
Dashboards &
Visualizations
Power BI

46. 主題：將資料轉換為智能及決策
Intelligence
Dashboards &
Visualizations
Information
Management
Big Data Stores Machine Learning
and Analytics
CortanaEvent Hubs
HDInsight
(Hadoop and
Spark)
Stream
Analytics
Data Intelligence Action
People
Automated
Systems
Apps
Web
Mobile
Bots
Bot
Framework
SQL Data
WarehouseData Catalog
Data Lake
Analytics
Data Factory
Machine
Learning
Data Lake Store
Cognitive
Services
Power BI
Data
Sources
Apps
Sensors
and
devices
Data

48. DATACENTERS (CLOUD)
PC / DEVICE
DIGITAL TAPE
DVD / BLU-RAY
CD

49. CLOUD / IoT
PC / MOBILE

50. CLOUD
MOBILE

57. Microsoft Ventures

58. 主題：將資料轉換為智能及決策
Intelligence
Dashboards &
Visualizations
Information
Management
Big Data Stores Machine Learning
and Analytics
CortanaEvent Hubs
HDInsight
(Hadoop and
Spark)
Stream
Analytics
Data Intelligence Action
People
Automated
Systems
Apps
Web
Mobile
Bots
Bot
Framework
SQL Data
WarehouseData Catalog
Data Lake
Analytics
Data Factory
Machine
Learning
Data Lake Store
Cognitive
Services
Power BI
Data
Sources
Apps
Sensors
and
devices
Data

59. © 2017 Microsoft Corporation. All rights reserved.