Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Hands-On with U-SQL and Azure Data Lake Analytics (ADLA)

7.019 Aufrufe

Veröffentlicht am

U-SQL is the query language for big data analytics on the Azure Data Lake platform. This session will explore the unification of SQL and C# in this new query language, examples of combining data from external sources such as Azure SQL Database and Blob storage with Azure Data Lake store, creating and referencing assemblies, job submission and tools. The ADL platform will also be compared and contrasted to the HDInsight/Hadoop platform.

Veröffentlicht in: Daten & Analysen
  • Login to see the comments

Hands-On with U-SQL and Azure Data Lake Analytics (ADLA)

  1. 1. Hands-On with U-SQL and Azure Data Lake Analytics (ADLA) A first look at U-SQL on Azure Data Lake Public Preview Jason Brugger (@JasonLBrugger) MCSE: Data Platform, MCSE: Business Intelligence July 16, 2016
  2. 2. This presentation has been modified from its original format. Animations have been removed and it has been reformatted for publication on Slideshare.net.
  3. 3. Assumptions • You are familiar with the differences between a traditional RDBMS and a Big Data solution. • You are familiar with both T-SQL and C#.
  4. 4. What is a data lake? What are Azure Data Lake Store and Azure Data Lake Analytics? • A data lake is a storage repository that holds a vast amount of raw data in its native format until it is needed. – Margaret Rouse (on AWS) • Pentaho CTO James Dixon has generally been credited with coining the term “data lake”. He describes a data mart (a subset of a data warehouse) as akin to a bottle of water…” cleansed, packaged and structured for easy consumption” while a data lake is more like a body of water in its natural state. – Chris Campbell, Blue Granite • Data Lake Analytics is an Azure Big Data computation service that lets you use data to drive your business using the insights gained from your data in the cloud, regardless of where it is and regardless of its size. – Ed Macauley, Microsoft Data Lake Store Data Lake Analytics
  5. 5. ADLA vs. HDInsight (e.g. Hadoop) • HDInsight (Cluster as a Service) • Provision cluster of n nodes • Run your queries • Delete cluster • (Repeat) • ADLA (Query as a service) • Don’t provision anything • Specify node count (parallelism) at job submission time • Pay per query
  6. 6. Getting Started – What’s Needed? • Azure subscription • Sign-up for ADL preview • Visual Studio 2015 + Azure Data Lake Tools for Visual Studio • Microsoft Azure PowerShell (1.0+ via WPI) • Not to be confused with the version of PowerShell, e.g. 5.0. • Microsoft Azure SDK for .NET (Optional)
  7. 7. Tools and Navigation • Azure Portal
  8. 8. Tools and Navigation • Azure Portal • Visual Studio • Server Explorer
  9. 9. Tools and Navigation • Azure Portal • Visual Studio • Server Explorer • Project Templates • PowerShell • SDKs • C# • Node.js
  10. 10. Getting data into ADL • Portal
  11. 11. Getting data into ADL • Portal • PowerShell • Login-AzureRmAccount • Import-AzureRmDataLakeStoreItem • Connecting to External Data (Demo #2) • SSIS • ADF
  12. 12. The Data (NOAA Weather observations) Station Datekey Element Value Mflag Qflag Sflag TimeKey US1FLSL0019 20150101 PRCP 173 N US1TXTV0133 20150101 PRCP 119 N USC00178998 20150101 TMAX -33 700 USC00178998 20150101 TMIN -167 700 USC00178998 20150101 TOBS -67 700 USC00178998 20150101 PRCP 0 700 USC00178998 20150101 SNOW 0 USC00178998 20150101 SNWD 0 USR0000CSNR 20150101 TMAX 194 H D U Notice sparse data with many null values
  13. 13. The Data (NOAA Weather observations) Station Datekey Element Value Mflag Qflag Sflag TimeKey US1FLSL0019 20150101 PRCP 173 N US1TXTV0133 20150101 PRCP 119 N USC00178998 20150101 TMAX -33 700 USC00178998 20150101 TMIN -167 700 USC00178998 20150101 TOBS -67 700 USC00178998 20150101 PRCP 0 700 USC00178998 20150101 SNOW 0 USC00178998 20150101 SNWD 0 USR0000CSNR 20150101 TMAX 194 H D U Multiple observation types, per site, per day
  14. 14. The Data (NOAA Weather observations) Station Datekey Element Value Mflag Qflag Sflag TimeKey US1FLSL0019 20150101 PRCP 173 N US1TXTV0133 20150101 PRCP 119 N USC00178998 20150101 TMAX -33 700 USC00178998 20150101 TMIN -167 700 USC00178998 20150101 TOBS -67 700 USC00178998 20150101 PRCP 0 700 USC00178998 20150101 SNOW 0 USC00178998 20150101 SNWD 0 USR0000CSNR 20150101 TMAX 194 H D U Tenths of degree C
  15. 15. The Data (NOAA Weather observations) Station Datekey Element Value Mflag Qflag Sflag TimeKey US1FLSL0019 20150101 PRCP 173 N US1TXTV0133 20150101 PRCP 119 N USC00178998 20150101 TMAX -33 700 USC00178998 20150101 TMIN -167 700 USC00178998 20150101 TOBS -67 700 USC00178998 20150101 PRCP 0 700 USC00178998 20150101 SNOW 0 USC00178998 20150101 SNWD 0 USR0000CSNR 20150101 TMAX 194 H D U Correlates to external data uploaded to Azure SQL Database
  16. 16. Basic U-SQL query Load .csv file from Data Lake Store using built-in Extractor Schematize using C# data types, note nullability Output schematized rows to a table variable SELECT using familiar SQL-like queryOutput query result to Data Lake Store using built-in Outputter
  17. 17. Key Takeaways & ‘gotchas’ • SQL statements MUST be uppercase • Header rows not currently supported by Extractor • e.g skipFirstNRows:1 not currently supported • Be mindful of nullability in C# types • Built-in operators include support for .Csv(), .Tsv(), & .Text() • Various options such as delimiter • Build custom extractors by inheriting IExtractor
  18. 18. DEMO #1 • Demo local execution • Simple aggregation of 10,000 rows down to 43, by element type
  19. 19. Persisted schema with meta data object model Familiar CREATE DATABASE statement Familiar CREATE VIEW statement; View maintains extractor and schema definitions so from now on, we can just select from the view. Note data kept in its native compressed (.gz) format. Extractor handles decompression in this case Wildcard {*} yields file- set of all matching files
  20. 20. Combining with external data • Create catalog secret using PowerShell (specifies remote Host & credentials & ADLA catalog) • New-AzureRmDataLakeAnalyticsCatalogSecret • Create credential (in turn, references catalog secret) • Create data source (in turn, references credential & specifies data source type (e.g. Azure SQL Db) & specifies remote catalog) CREATE EXTERNAL TABLE denotes underlying table resides remotely Schema using C# types Remote table name
  21. 21. External data with federated query SELECT FROM EXTERNAL data source EXECUTE Embedded query executes remotely at data source. This is T-SQL, not U-SQL
  22. 22. External data with federated query Embedded query executes remotely at data source. This is T-SQL, not U-SQL Table variable contains only rows returned id name US1FLHB0090 TAMPA 10.2 NNW US1FLHB0048 GREATER NORTHDALE 0.4 ENE USW00012810 MACDILL AFB USC00088890 TEMPLE TERRACES US1FLHB0007 TAMPA 8.4 NW US1FLHB0025 CARROLLWOOD 1.7 SE US1FLHB0040 UNIVERSITY WEST 2.0 WNW USC00088786 TAMPA US1FLHB0028 WEST PARK 0.4 S US1FLHB0055 TAMPA 5.0 NNE US1FLHB0012 CARROLLWOOD 0.5 WNW US1FLHB0096 TAMPA 5.4 SSW US1FLHB0005 CITRUS PARK 1.3 ENE USC00080520 BAY LAKE US1FLHB0093 TEMPLE TERRACE 1.5 SE US1FLHB0039 CARROLLWOOD 2.0 SSE US1FLHB0087 TAMPA 7.9 N US1FLHB0071 TAMPA 6.1 N USW00012842 TAMPA INTL AP US1FLHB0010 TAMPA 5.1 S US1FLHB0036 TAMPA 4.4 SSW US1FLHB0029 TAMPA 6.5 NNE US1FLHB0064 TAMPA 4.7 NW US1FLHB0051 LUTZ 2.2 SSE
  23. 23. Data relationship exhibit Azure SQL Db Federate d Query dbo.station dbo.calend ar Azure Data Lake Analytics @station_t pa dbo.calend ar dbo.observ -ation Result Azure Data Lake Store
  24. 24. Complex types in U-SQL & EXPLODE • SQL.ARRAY • Like a List or Array in C# • Can be used in conjunction with String.Split() • SQL.MAP • Key-Value pairs • Like a Dictionary (Hash table) in C# • EXPLODE • Expands to rows ID (int) Data (SQL.MAP) 1 ((“A”,25),(“B”,35),(“C”,45)) 2 ((“A”,27),(“B”,38),(“C”,42)) ID 1 A 25 1 B 35 1 C 45 2 A 27 2 B 38 2 C 42 EXPLODE
  25. 25. Tying it all together with U-SQL Familiar JOIN syntax; Note double equals “==“, the only supported JOIN operatorFamiliar WHERE and GROUP BY syntax
  26. 26. Tying it all together with U-SQL CROSS APPLY the value (recall this was in tenths of degrees C)
  27. 27. Tying it all together with U-SQL CROSS APPLY the value (recall this was in tenths of degrees C) Declare a new SQL.MAP with conversion factors by C, F, and K
  28. 28. Tying it all together with U-SQL CROSS APPLY the value (recall this was in tenths of degrees C) Declare a new SQL.MAP with conversion factors by C, F, and KEXPLODE the SQL.MAP into rows and new columns scale, temp
  29. 29. Tying it all together with U-SQL Familiar aggregation AVG using exploded column temp
  30. 30. Tying it all together with U-SQL Using String.Concat .NET method to build description of derived column e.g. “AVG_TMAX_F” AVG_TMAX_C AVG_TMAX_K AVG_TMIN_F AVG_TMIN_C AVG_TMIN_K
  31. 31. Tying it all together with U-SQL CREATE TABLE AS SELECT (CTAS) – Conceptually similar to select into
  32. 32. Tying it all together with U-SQL CREATE TABLE AS SELECT (CTAS) – Conceptually similar to select into No HEAPs in ADLA; Clustered index required
  33. 33. Tying it all together with U-SQL CREATE TABLE AS SELECT (CTAS) – Conceptually similar to select into No HEAPs in ADLA; Clustered index required Partitioned by Round Robin distributes data evenly.
  34. 34. Tying it all together with U-SQL CREATE TABLE AS SELECT (CTAS) – Conceptually similar to select into No HEAPs in ADLA; Clustered index required Partitioned by Round Robin distributes data evenly. No update or merge support
  35. 35. DEMO #2 • Data Lake data set consists of daily readings from 98,035 stations over 5 years • ~32,720,048 rows per file • About 164M rows total • 24 Tampa stations • Filtering and aggregating it down to 5 years x 12 months x 2 elements x 3 temperature scales, or 360 rows • Monitor job execution status • Streams, Vertices, Display avg execution time (heat map), Diagnostics, History, Script*
  36. 36. Working with Assemblies & Libraries • Code-behind file • Convenient, simple • Assembly created and referenced automatically • No support for NuGet, but manually add references…OR: • Class library • Right-click, register assembly to ADLA • Option to automatically copy to DLS • NuGet supported normally
  37. 37. Example: Simple Linear Regression to predict temp .dlls for statistics library copied to Data Lake Store CREATE ASSEMBLY from file; (Can also create from binary) Custom C# class method signature Noaa.Predict.Regress( int, SqlMap<int, decimal?>) : decimal?
  38. 38. Example: Simple Linear Regression to predict temp .dlls for statistics library copied to Data Lake Store CREATE ASSEMBLY from file; (Can also create from binary) Custom C# class method signature Noaa.Predict.Regress( int, SqlMap<int, decimal?>) : decimal? MAP_AGG() function returns SQL.MAP – like a reverse EXPLODE, which we pass as function parameter
  39. 39. MAP_AGG() Exhibit Month Year Avg Temp 1 2011 34 1 2012 36 1 2013 33 1 2014 35 1 2015 37 2 2011 41 2 2012 39 … 12 2015 26
  40. 40. .CS code-behind file Year to predict, e.g. 2016 Referenced Library namespace
  41. 41. .CS code-behind file Return predicte d Temp Year to predict, e.g. 2016 SqlMap contains series against which regression is performed Referenced Library namespace
  42. 42. DEMO #3 • Pivoting our existing averages on Month & aggregating Year & Temp into Key-Value pairs (SQL.MAP) which we pass as parameter to custom function • Passing predictive year (2016) as a parameter • Limit our selection to just AVG_TMAX_F • Result adds another 12 rows of predicted temps to our existing 360 row result table
  43. 43. Additional homework subjects, not covered • Extractor (UDO) by inheriting IExtractor • IOutputter • IProcessor – transform single row, read one, output one • IReducer – read n rows, output 1 row • ICombiner – like a user-defined Join • IApplier – input one row, output n rows • User-defined Aggregators (IAggregate) – AGG keyword • ARRAY_AGG() • Blob as External storage • No Primary Keys • No columnstore (yet) • Table Value Functions - YES, but not with cross apply • No support for R, but leverage .NET libraries as demo’d • User-defined Types • Partitioning by Hash, Direct Hash, Range • http://www.slideshare.net/MichaelRys/usql- partitioned-data-and-tables-sqlbits-2016
  44. 44. Reference • Code • https://github.com/SQL-Jason/NOAA_USQL_Demo.git • Data • http://www1.ncdc.noaa.gov/pub/data/ghcn/daily/by_year/ • Blog • http://jasonbrugger.wordpress.com
  45. 45. Attribution The Accord.NET Framework Copyright (c) 2009-2014, César Roberto de Souza <cesarsouza@gmail.com> This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version. The copyright holders provide no reassurances that the source code provided does not infringe any patent, copyright, or any other intellectual property rights of third parties. The copyright holders disclaim any liability to any recipient for claims brought against recipient by any third party for infringement of that parties intellectual property rights. This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. National Oceanic and Atmospheric Administration (NOAA) README FILE FOR DAILY GLOBAL HISTORICAL CLIMATOLOGY NETWORK (GHCN-DAILY) Version 3.22 How to cite: Note that the GHCN-Daily dataset itself now has a DOI (Digital Object Identifier)so it may be relevant to cite both the methods/overview journal article as well as the specific version of the dataset used. The journal article describing GHCN-Daily is:Menne, M.J., I. Durre, R.S. Vose, B.E. Gleason, and T.G. Houston, 2012: An overview of the Global Historical Climatology Network-Daily Database. Journal of Atmospheric and Oceanic Technology, 29, 897-910, doi:10.1175/JTECH-D-11- 00103.1. To acknowledge the specific version of the dataset used, please cite:Menne, M.J., I. Durre, B. Korzeniewski, S. McNeal, K. Thomas, X. Yin, S. Anthony, R. Ray, R.S. Vose, B.E.Gleason, and T.G. Houston, 2012: Global Historical Climatology Network - Daily (GHCN-Daily), Version 3. [indicate subset used following decimal, e.g. Version 3.12]. NOAA National Climatic Data Center. http://doi.org/10.7289/V5D21VHZ [access date].
  46. 46. Bibliography • Campbell, C. “Top Five Differences between Data Lakes and Data Warehouses.” Business Insights. Blue Granite, 26 Jan 2015. Web. https://www.blue- granite.com/blog/bid/402596/Top-Five-Differences-between-Data-Lakes-and-Data-Warehouses • Gopalan, R. (21 Jun 2016). U-SQL Part 4: Use custom code to extend U-SQL [Webinar]. PASS Big Data Virtual Chapter. • Macauley, E. “Overview of Microsoft Azure Data Lake Analytics.” Microsoft Azure. Microsoft, 16 May 2016. Web. https://azure.microsoft.com/en- us/documentation/articles/data-lake-analytics-overview/ • Reddy, S. (31 May 2016). Introduction to Azure Data Lake [Webinar]. PASS Big Data Virtual Chapter. • Rossello, Justin. “Querying Azure SQL Database from an Azure Data Lake Analytics U-SQL Script.” eat{Code}live. 21 Nov 2015. Web. http://eatcodelive.com/2015/11/21/querying-azure-sql-database-from-an-azure-data-lake-analytics-u-sql-script/ • Rouse, M. “Definition Data Lake.” SearchAws. TechTarget, May 2015. Web. http://searchaws.techtarget.com/definition/data-lake • Rys, M. (8 Mar 2016). Introducing U-SQL; Part 2 of 2: Scaling U-SQL and doing SQL in U-SQL [Webinar]. PASS Big Data Virtual Chapter. Retrieved from http://www.youtube.com/channel/UCkOKmMW_LEsACOqE8C1RWdw • Rys, M. (16 Feb 2016). Introducing U-SQL; Part 1 of 2: Introduction and C# extensibility [Webinar]. PASS Big Data Virtual Chapter. Retrieved from http://www.youtube.com/channel/UCkOKmMW_LEsACOqE8C1RWdw • Rys, M., et. al. Azure/usql, (2016), GitHub repository, https://github.com/Azure/usql • “U-SQL Language Reference.” Microsoft Azure. Microsoft, 28 Oct 2015. Web. https://msdn.microsoft.com/en- US/library/azure/mt591959(Azure.100).aspx

×