--session donnée lors du SQL Saturday Lisbon 2015-- http://www.sqlsaturday.com/369/EventHome.aspx Data Management Gateway (and also AS Connector) is what make modern Microsoft BI stack hybrid. Power BI and Azure Data Factory use that component to interact with On-Prem Data assets. That session is a Deep dive into the DMG and the hybrid architecture involved by Power BI and ADF. How does it work ? Security, Firewall, Certificates, Multiple gateways, Admin delegation, Scale out, Disaster Recovery…. All that topics will be covered during that technical session.

1. Data Management Gateway
Deep Dive

2. Our Main Sponsors:

3. Jean-Pierre Riehl
Practice Manager Data & BI
@djeepy1 - http://blog.djeepy1.net
Who am I ?
Pure-Player Microsoft
 Self-Service BI experts
 « Agile BI » projects
 Data Strategy
 Hybrid-BI
MVP SQL Server
French PASS
Chapter Leader

4. An hybrid story…

5. Microsoft Cloud-First strategy…

6. But your data remains On-Prem…

7. Here it comes DMG

8. DMG-aware tools
Power BI for O365
PowerBI.com
Azure Data Factory

9. Power BI
Excel
Power Query Power Pivot
Power View Power Map
Power BI Tenant SQL
Cloud On-Prem
Oracle …
Power BI
Admin Center
Data
Catalog
Q&A
Power BI Sites
Data Refresh
Indexing / Search
OData
Access

10. PowerBI.com
Live Power BI
reports &
dashboards
Live
Query
Scheduled
Refresh
Cloud
On-premises

11. Azure Data Factory

12. Hybrid features
 Data Refresh
 Data Movement
 Live Query
 Indexing Sources
 OData Access (Data Virtualization)

13. One DMG to rule them all
DMZ
SQL
Cloud On-Prem
Oracle
…
Power BI for O365
Data
Management
Gateway
OnPrem Data Sources
PowerBI.com
Azure Data Factory
Excel

14. DMG DETAILS

15. Versions of DMG
 There are multiple versions of DMG
Data Management Gateway
Power BI
ADF
1.4.5549.1 24/3/2015
Analysis Services Connector Power BI v2 1.4.5576.2 17/4/2015

16. What’s that
 A Windows Service
 DIAHostService
 A configuration application
 Managed by the cloud
 Power BI Admin Center
 Azure Portal (new version)

18. FIRST LOOK AT DMG

19. Data Sources Supported
Source Authentification
SQL Server / Oracle
Sybase, Teradata, DB2, MySQL
Windows, Basic
Files (CSV, XML, Text, Excel, Access)
Folders
Windows
SharePoint List (Online)
SharePoint List (On-prem)
Anonymous
Anonymous, Windows
OData Feed
Anonymous, Windows, Basic
Key (Data Market)
Azure Marketplace Key (Data Market)
Azure HDInsight
Azure Blob Storage
Azure Table Storage
Key (Azure Account)
SQL Azure
PostgreSQL
Basic
Teradata, Sybase, DB2, MySQL Windows, Basic
SSAS Tabular Windows
+SSIS Data
Feed

20. Data Sources supported
 Details on data sources supported
 https://support.office.com/en-us/article/Supported-Data-Sources-
and-Data-Types
 Prequisites
 Beware of types
 Drivers installed
 https://support.office.com/en-us/article/Data-source-prerequisites-
Power-Query

21. HOW DOES IT WORK

22. Cloud On-Prem
Data Refresh explained
Power BI
OnPrem
Sources
DMG
Azure Service
Bus
0
Register DMG
Register Data Sources
1
Ask for Data Refresh
Check permissions
Find Gateway
4 Execute Query
5 Get Data
6 Compress, Chunk
Queue
request2
3 Read
request
Gateway
Cloud Service
7 Update Model

23. Cloud On-Prem
OData access explained
Power BI
OnPrem
Sources
DMG
0 Index Sources
2
Resolve Data Source
Redirect to local Gateway
4 Execute Query
3 Request Data
1 Search Source
Excel
Power Query
Hybrid Proxy

24. DMG IN ACTION

25. Data Source Discovery
 Data Sources are declared on « Admin
Center »
 They are associated to a Gateway
 Gateway Cloud Service resolves Data
Source and find right Gateway
 Resolution = Provider + Server + Catalog
 Warning: Case sensitive

26. DATA SOURCE DISCOVERY

27. SECURITY

28. Excel
Power Query
Power BI
DMG
Data sources
• SQL (1433)
• FileShare
• Etc.
+DNS resolution
Azure Service Bus (outbound)
• 9350-9354
Power BI Admin Center (outbound)
• 80/443
Direct HTTP
(inbound)
• HTTPS (8050)
• HTTP (8051)
HTTPS
(443)
Cloud
On-Prem
Network Security
Credential
Manager

29. Identity Management
2 flavors :
 Impersonation by DMG
 Impersonation by Data Source
How is query authentified on Data Source ?

30. Impersonation by DMG
 Connection identity is impersonated by DMG
 Peer-to-peer connection
 Credentials are stored by DMG for each
source
 Credentials can be stored encrypted locally
 Credentials can be stored encrypted on Cloud
 Credentials are set up locally
 Credential Manager is a Click Once application
 Credential Manager discuss with DMG

31. CREDENTIAL MANAGER

32. Impersonation by Data Source
 Used by Live Query
 Keep identity of original user
 Use the EffectiveUsername method
 DMG authenticates to SSAS
 DMG passes EffectiveUsername property
 SSAS executes the query as if it was original user
 DMG must be SSAS Admin

33. EffectiveUsername

34. HIGH AVAILABILITY &
DISASTER RECOVERY

35. As easy as scale-out
DMGs
Cloud
On-Prem
Power BI
…
Instance 1 Instance 2
Server A Server B

36. HA & DR
 Gateways and Data sources are managed by
Cloud
 Credentials are encrypted by MDG
 Use of Certificate, self-generated or yours
 Need Certificate when registering a new instance
 Warning when stored localy
 Load Balancing

37. HA & DR

38. Going further
 Express Route
 Mashup Container (aka Power Query)
 Monitoring & Logging

39. Resources
 Getting Started : https://support.office.com/en-us/article/Power-BI-for-Office-365-Admin-Center-Help-
5e391ecb-500c-47a3-bd0f-a6173b541044
 Data Factory Reference : https://msdn.microsoft.com/en-us/library/azure/dn834987.aspx
 Supported Data sources : https://support.office.com/en-us/article/Supported-Data-Sources-and-Data-
Types
 DMG Prerequisites : https://support.office.com/en-us/article/Data-source-prerequisites-Power-Query
 SSIS Data Feed : http://www.microsoft.com/en-us/download/details.aspx?id=39931
 How does DMG work - sequence diagram :
 https://support.office.com/en-us/article/An-Inside-look-at-Microsoft-Data-Management-Gateway-
078D306D-2D28-409C-962B-D80B69A1D0B2
 Azure IP adresses : http://www.microsoft.com/en-us/download/details.aspx?id=41653
 Store encrypted credentials in the cloud for business continuity : https://support.office.com/en-
us/article/Business-Continuity-for-a-Data-Management-Gateway-00106c58-a0a1-4cea-8886-
da51b8cadc5b
 Matt Masson Session at SPC : http://channel9.msdn.com/Events/SharePoint-
Conference/2014/SPC397
 Mini Nay & Luming Han Session at PASS Summit :
http://www.sqlpass.org/summit/2014/Sessions.aspx#sid7231

40. Obrigado!
Thank you!
Merci !
@djeepy1 - http://blog.djeepy1.net
Any questions ?