Bi and AI updates in the Microsoft Data Platform stack

•

0 gefällt mir•647 views

This document provides an overview and agenda for a presentation on new features in the Microsoft data platform universe including SQL Server, Azure, and Power BI, with a focus on business intelligence (BI) and artificial intelligence (AI). The presentation covers new features in SQL Server 2019 for BI like calculation groups and dynamic formatting in Analysis Services tabular mode. It also highlights important updates in Azure including new serverless and hyperscale options for Azure SQL Database, updates to data ingestion and storage with Azure Data Factory v2 and Azure Data Lake Gen 2, and changes to data modeling, preparation, and serving capabilities in Azure SQL Data Warehouse. The document concludes by discussing options for incorporating AI and machine learning into BI solutions using either

Technologie

Powered by
BI and AI in the
Microsoft data
platform
universe
(with a dash of cloud)
Ivan Donev

Agenda
• What’s new in SQL 2019 for BI
• Important updates on Azure for Data platform
• AI and ML for the masses

What is new in 2019 for BI
• SQL Server AS
• MD nothing
• Tabular – Calc groups, M:M relationships, dynamic formatting
• SQL Server IS
• Nothing
• SQL Server RS
• Nothing
• SQL Server MDM and DQS
• Almost nothing

DEMO with AS 2019
Calculation groups
Dynamic formatting
M:M in tabular

Updates in Azure SQL DB
• Azure SQL DB – Hyperscale• Azure SQL DB – Serverless

Azure SQL DB – Serverless
• Single DB Serverless compute tier
• Billed on compute used per SECOND
• Used only in the vCore model
• Parametrize the min/max vCores
• Scenarios
• Intermittent usage
• Frequently rescaled DBs
• New deployments prior historical usage data

Azure SQL DB – Hyperscale
• Up to 100TB
• Fast backups (filesystem snapshots)
• Up to a minute restores
• Faster throughput
• Fast scale out and scale up
• Distributed architecture

Modern DWH – important updates in Ingest
• Azure Data Factory v2
• Integration runtimes to run SSIS
as-is
• Storing SSIS catalogue in SQL DB
• Mapping workflow
• Wrangling workflow

Modern DWH – important updates in Store
• Azure Data Lake Gen 2
• Hierarchical file system
• Security
• Performance
• Much easier to integrate with
other services

Modern DWH – important updates in Prep and
Train
• Azure Databricks Delta
• Spark engine with RDBMS features

Databricks Delta
• ACID transactions
• Versioned PARQUET files
• Streaming writes to a table (i.e.
Kafka)
• Batch upserts
• High performance reads
• Schema enforcement

Modern DWH – important updates in Model and
Serve
• Changes in Azure DWH
• Concurrency increased to 128
• Adaptive caching (NVMe !!!)
• Unlimited Columnstore storage
capacity
• Workload classification and
importance improvements
• Changes in PowerBI

PowerBI Updates worth noting
• PowerBI Dataflows
• Self-service data transformation
• Shared and certified datasets
(preview)
• Paginated Reports (SSRS)
• Premium
• XMLA Endpoints
• Premium
• Auto ML
• Premium
• CDS integration

The AI in BI
• Options to use in DWH and BI

The ML in BI
Not scalableSelf-service AI
•Prototyping
•Do not need additional configuration or tuning
•Options are
•Microsoft Cognitive services with PowerBI (demo)
•AutoML in Dataflows in PowerBI Premium
•R/Python visuals
Scalable, configurable, needs specialized staffEnterprise AI
•Mandatory to run ML and store it in the Store/Serve model
•Options are
•Databricks
•Azure ML
•R/Python in Dataflow as data sources

How to choose?
• The aim?
• Prototype/Test/Verify
• Production/O16N
• The knowledge
• R/Python/Scala/Java/…
• The task
• Image processing/Text analytics/Prediction/Classification
• The post-production support
• Can you support the solution afterwards?

Demo
• PowerBI Dataflows
• Using External Cognitive Services APIs

THANK YOU
All my demos will be described and uploaded on our blog:
http://sqlmasteracademy.com/techblog/

Weitere ähnliche Inhalte

Was ist angesagt?

Modern ETL: Azure Data Factory, Data Lake, and SQL Database

Eric Bragas

Customer migration to Azure SQL database, December 2019

George Walters

AliCloud Object Storage Service (OSS) Core Features

Alibaba Cloud

The challenge with today’s “data explosion” is finding the most appropriate answer to the question, “So where do I put my data?” while avoiding the longer-term problem: data warehouses, data lakes, cloud storage, NoSQL databases, … are often the places where “big” data goes to die. Enter Physics 101, and my corollary to Newton’s First Law of Motion: Data in motion tends to stay in motion until it comes rest on disk. Similarly, if data is at rest, it will remain at rest until an external “force” puts it in motion again. Data inevitably comes to rest at some point. Without “external forces”, data often gets lost or becomes stale where it lands. “Modern” architectures tend to involve data pipelines where downstream consumers of data make use of data generated upstream, often with built-for-purpose repositories at each stage. This session will explore how data that has come to rest can be put in motion again; how Kafka can keep it in motion longer; and how pipelined architectures might be created to make use of that data.

Data in Motion: Building Stream-Based Architectures with Qlik Replicate & Kaf...

HostedbyConfluent

Netflix's Big Leap from Oracle to Cassandra

Roopa Tangirala

Kubernetes has quickly become the ad-hoc standard for managing software deployments. The Scylla team recently released the beta version of the Scylla Operator for Kubernetes. Join us to learn how Kubernetes can be used to automate the deployment, scaling, and various operations of Scylla NoSQL. In this webinar we will: - Present the mapping between Scylla and Kubernetes entities, and explain their reasoning - Give insight into the Operator Open Source project, and how you can get involved - Explain and demo common procedures with Scylla Operator - Discuss the challenges of running a high performance, persistent application on Kubernetes, and the trade offs we considered - Share our plans for Scylla on Kubernetes

Running Scylla on Kubernetes with Scylla Operator

ScyllaDB

Introducing NoSQL and MongoDB to complement Relational Databases (AMIS SIG 14...

Lucas Jellema

See webinar video recording of this presentation at https://resource.alibabacloud.com/webinar/detail.htm?webinarId=37 As the third installment of the Alibaba Cloud Big Data Quickstart Series, this webinar presentation introduces the basic concepts and architecture of the offline processing engine MaxCompute and online integrated development environment DataWorks. This includes an explanation and demonstration of how to use the Data Integration component of DataWorks to integrate unstructured data stored in OSS and structured data stored with ApsaraDB for RDS (MySQL) to MaxCompute.

Big Data Quickstart Series 3: Perform Data Integration

Alibaba Cloud

Logging infrastructure for Microservices using StreamSets Data Collector

Cask Data

Beyond Relational

Lynn Langit

Cloud-based Linked Data Management for Self-service Application Development

Peter Haase

Organizational compliance and security SQL 2012-2019 by George Walters

George Walters

Microsoft ignite 2018 SQL Server 2019 big data clusters - intro session

Travis Wright

Building a data warehouse with AWS Redshift, Matillion and Yellowfin

Lynn Langit

Benchmarking Aerospike on the Google Cloud - NoSQL Speed with Ease

Lynn Langit

Sql connections germany - migration considerations when migrating your on pre...

Charley Hanania

In the online gaming industry we receive a vast amount of transactions that need to be handled in real time. Our customers get to choose from hundreds or even thousand options, and providing a seamless experience is crucial in our industry. Recommendation systems can be the answer in such cases but require handling loads of data and need to utilize large amounts of processing power. Towards this goal, in the last two years we have taken down the road of machine learning and AI in order to transform our customer’s daily experience and upgrade our internal services.

Personalization Journey: From Single Node to Cloud Streaming

Databricks

We as data engineers are aware of trade off’s between development speed, metadata governance and schema evolution (or restriction) in rapidly evolving organization. Our day to day activities involve adding/removing/updating tables, protecting PII Information, curating and exposing data to our consumers. While our data lake keeps growing exponentially, there is equal increase in our downstream consumers. Struggle is to maintain balance between quickly promoting metadata changes with robust validation for downstream systems stability. In relational world DDL, DML changes can be managed through numerous options available for every kind of database from the vendor or 3rd party. As engineers we developed a tool which uses centralized git managed repository of data schemas in yml structure with ci/cd capabilities which maintains stability of our data lake and downstream systems. In this presentation Northwestern Mutual Engineers, will discuss how they designed and developed new end-to-end ci/cd driven metadata management tool to make introduction of new tables/views, managing access requests etc in a more robust, maintainable and scalable way, all with only checking in yml files. This tool can be used by people who have no or minimal knowledge of spark. Key focus will be: Need for metadata management tool in a data lake Architecture and Design of the tool Maintaining information on databases/tables/views like schema, owner, PII, description etc in simple to understand yml structure Live demo of creating a new table with CI/CD promotion to production

Automated Metadata Management in Data Lake – A CI/CD Driven Approach

Databricks

Implementing tools, let alone an entire Unified Data Platform, like Databricks, can be quite the undertaking. Implementing a tool which you have not yet learned all the ins and outs of can be even more frustrating. Have you ever wished that you could take some of that uncertainty away? Four years ago, Western Governors University (WGU) took on the task of rewriting all of our ETL pipelines in Scala/Python, as well as migrating our Enterprise Data Warehouse into Delta, all on the Databricks platform. Starting with 4 users and rapidly growing to over 120 users across 8 business units, our Databricks environment turned into an entire unified platform, being used by individuals of all skill levels, data requirements, and internal security requirements. Through this process, our team has had the chance and opportunity to learn while making a lot of mistakes. Taking a look back at those mistakes, there are a lot of things we wish we had known before opening the platform to our enterprise. We would like to share with you 10 things we wish we had known before WGU started operating in our Databricks environment. Covering topics surrounding user management from both an AWS and Databricks perspective, understanding and managing costs, creating custom pipelines for efficient code management, learning about new Apache Spark snippets that helped save us a fortune, and more. We would like to provide our recommendations on how one can overcome these pitfalls to help new, current and prospective users to make their environments easier, safer, and more reliable to work in.

10 Things Learned Releasing Databricks Enterprise Wide

Databricks

Vitalii Bondarenko "Machine Learning on Fast Data"

DataConf

Was ist angesagt? (20)