This 400 level presentation explains the U-SQL Query Execution in Azure Data Lake and provides several Performance Tuning tips: What tools are available and some best practices.
7. U-SQL C#
user code
C++
system code
Algebra
other files
(system files, deployed resources)
managed dll
Unmanaged dll
Input
script
Compilation output (in job folder)
Compiler & Optimizer
Files
Meta
Data
Service
Deployed to vertices
8. Some fixed amount of work
Each square is called a “vertex”
Each vertex represents a fraction of the work
17.
data may be
distributed such that
all rows that match a
certain key go to a
single vertex
imbalanced
execution, vertex time
out.
0
5,000,000
10,000,000
15,000,000
20,000,000
25,000,000
30,000,000
35,000,000
40,000,000
California
NewYork
Illinois
Ohio
Michigan
NewJersey
Washington
Arizona
Tennessee
Maryland
Minnesota
Alabama
Louisiana
Oregon
Connecticut
Mississippi
Kansas
Nevada
Nebraska
Idaho
Maine
RhodeIsland
Delaware
Alaska
DistrictofColumbia
Wyoming
Population by State
18.
@rows =
SELECT Gender, AGG<MyAgg>(…) AS Result
FROM @HugeInput
GROUP BY Gender;
Gender==Male Gender==Female
@HugeInput
Vertex 0 Vertex 1
21. U-SQL Partitioning
Data Skew – Recursive Reducer
// Metrics per domain
@Metric =
REDUCE @Impressions ON UrlDomain
USING new Bing.TopNReducer(count:10)
;
// …
Inherent Data Skew
[SqlUserDefinedReducer(IsRecursive = true)]
public class TopNReducer : IReducer
{
public override IEnumerable<IRow>
Reduce(IRowset input, IUpdatableRow output)
{
// Compute TOP(N) per group
// …
}
}
Recursive
• Allow multi-stage aggregation trees
• Requires same schema (input => output)
• Requires associativity:
• R(x, y) = R( R(x), R(y) )
• Default = non-recursive
• User code has to honor recursive semantics
www.bing.com
brought to a single vertex
22. U-SQL Partitioning during Processing
Partitioning – Combiner Modes
// Existing Watson hits
@DeDup =
COMBINE @Exceptions AS L WITH @WatsonBuckets AS R
ON L.AppId WITH R.AppId
USING new Windows.WatsonDedupCombiner()
;
// …
[SqlUserDefinedCombiner(Mode = CombinerMode.Right)]
public class WatsonDedupCombiner : ICombiner
{
public override IEnumerable<IRow>
Combine(IRowset left, IRowset right, IUpdatableRow output)
{
// DeDup against existing Call Stacks
// …
}
}
CombinerMode
• Allow parallelism even within a partition
public enum CombinerMode
{
Inner, /// Inner join - both row level
Left, /// Left Group Join - left row level
Right, /// Right Group Join - right row level
Full /// Full Group Join - none row level
• Default = Full Group Join
• User code has to honor row level semantics
Row Level Combiner
X
IDother columns
X
Q
Q
F
Q
X
X
Z
F
F
X
LEFT
X
ID other columns
Q
Q
F
Q
X
Z
F
F
X
RIGHT
F
Z
M1
M2
M3
M4
Enables Broadcast JOIN
24.
Partition Scheme When to use?
HASH(keys)
DIRECT HASH Exact control of hash bucket
RANGE(keys) Keeps ranges together
ROUND ROBIN To get equal distribution (if others give skew)
25.
26. // Unstructured Files (24 hours daily log impressions)
@Impressions =
EXTRACT ClientId int, Market string, OS string, ...
FROM @"wasb://ads@wcentralus/2015/10/30/{*}.nif"
FROM @"wasb://ads@wcentralus/2015/10/30/{Market:*}_{*}.nif"
;
// …
// Filter to by Market
@US =
SELECT * FROM @Impressions
WHERE Market == "en"
;
U-SQL Optimizations
Partition Elimination – Unstructured Files
Partition Elimination
• Even with unstructured files!
• Leverage Virtual Columns (Named)
• Avoid unnamed {*}
• WHERE predicates on named virtual columns
• That binds the PE range during compilation time
• Named virtual columns without predicate = error
• Design directories/files with PE in mind
• Design for elimination early in the tree, not in the leaves
Extracts all files in the folder
Post filter = pay I/O cost to drop most data
PE pushes this predicate to the EXTRACT
EXTRACT now only reads “en” files!
en_10.0.nif
en_8.1.nif
de_10.0.nif
jp_7.0.nif
de_8.1.nif
../2015/10/30/
…
27. // TABLE(s) - Structured Files (24 hours daily log impressions)
CREATE TABLE Impressions (Day DateTime, Market string, ClientId int, ...
INDEX IX CLUSTERED(Market, ClientId)
PARTITIONED BY
BUCKETS (Day)
HASH(Market, ClientId) INTO 100
);
DECLARE @today DateTime = DateTime.Parse("2015/10/30");
// Market = Vertical Partitioning
ALTER TABLE Impressions ADD PARTITION (@today);
// …
// Daily INSERT(s)
INSERT INTO Impressions(Market, ClientId)
PARTITION(@today)
SELECT * FROM @Q
;
// …
// Both levels are elimination (H+V)
@Impressions =
SELECT * FROM dbo.Impressions
WHERE
Market == "en"
AND Day == @today
;
U-SQL Optimizations
Partition Elimination – TABLE(s)
Partition Elimination
• Horizontal and vertical partitioning
• Horizontal is traditional within file (range, hash, robin)
• Vertical is across files (bucketing)
• Immutable file system
• Design according to your access patterns
Enumerate all partitions filtering for today
30.ss
30.1.ss
29.ss
28.ss
29.1.ss
Impressions
…
deen
jp
de
PE across files + within each file
28. @Inpressions =
SELECT * FROM
searchDM.SML.PageView(@start, @end) AS PageView
OPTION(LOWDISTINCTNESS=Query)
;
// Q1(A,B)
@Sessions =
SELECT
ClientId,
Query,
SUM(PageClicks) AS Clicks
FROM
@Impressions
GROUP BY
Query, ClientId
;
// Q2(B)
@Display =
SELECT * FROM @Sessions
INNER JOIN @Campaigns
ON @Sessions.Query == @Campaigns.Query
;
U-SQL Optimizations
Partitioning – Minimize (re)partitions
Input must be partitioned on:
(Query)
Input must be partitioned on:
(Query) or (ClientId) or (Query, ClientId)
Optimizer wants to partition only once
But Query could be skewed
Data Partitioning
• Re-Partitioning is very expensive
• Many U-SQL operators can handle multiple partitioning choices
• Optimizer bases decision upon estimations
Wrong statistics may result in worse query performance
29. // Unstructured (24 hours daily log impressions)
@Huge = EXTRACT ClientId int, ...
FROM
@"wasb://ads@wcentralus/2015/10/30/{*}.nif"
;
// Small subset (ie: ForgetMe opt out)
@Small = SELECT * FROM @Huge
WHERE Bing.ForgetMe(x,y,z)
OPTION(ROWCOUNT=500)
;
// Result (not enough info to determine simple Broadcast
join)
@Remove = SELECT * FROM Bing.Sessions
INNER JOIN @Small ON Sessions.Client ==
@Small.Client
;
U-SQL Optimizations
Partitioning - Cardinality
Broadcast JOIN right?
Broadcast is now a candidate.
Wrong statistics may result in worse query performance
Optimizer has no stats this is small...
30.
31. // Bing impressions
@Impressions = SELECT * FROM
searchDM.SML.PageView(@start, @end) AS PageView
;
// Compute sessions
@Sessions =
REDUCE @Impressions ON Client, Market
READONLY Market
USING new Bing.SessionReducer(range : 30)
;
// Users metrics
@Metrics =
SELECT * FROM @Sessions
WHERE
Market == "en-us"
;
// …
Microsoft Confidential
U-SQL Optimizations
Predicate pushing – UDO pass-through columns
32. // Bing impressions
@Impressions = SELECT * FROM
searchDM.SML.PageView(@start, @end) AS PageView
;
// Compute page views
@Impressions =
PROCESS @Impressions
READONLY Market
PRODUCE Client, Market, Header string
USING new Bing.HtmlProcessor()
;
@Sessions =
REDUCE @Impressions ON Client, Market
READONLY Market
USING new Bing.SessionReducer(range : 30)
;
// Users metrics
@Metrics =
SELECT * FROM @Sessions
WHERE
Market == "en-us"
;
Microsoft Confidential
U-SQL Optimizations
Predicate pushing – UDO row level processors
public abstract class IProcessor : IUserDefinedOperator
{
/// <summary/>
public abstract IRow Process(IRow input, IUpdatableRow output);
}
public abstract class IReducer : IUserDefinedOperator
{
/// <summary/>
public abstract IEnumerable<IRow> Reduce(IRowset input, IUpdatableRow output);
}
33. // Bing impressions
@Impressions = SELECT Client, Market, Html FROM
searchDM.SML.PageView(@start, @end) AS PageView
;
// Compute page views
@Impressions =
PROCESS @Impressions
PRODUCE Client, Market, Header string
USING new Bing.HtmlProcessor()
;
// Users metrics
@Metrics =
SELECT * FROM @Sessions
WHERE
Market == "en-us"
&& Header.Contains("microsoft.com")
AND Header.Contains("microsoft.com")
;
U-SQL Optimizations
Predicate pushing – relational vs. C# semantics
34.
35. // Bing impressions
@Impressions = SELECT * FROM
searchDM.SML.PageView(@start, @end) AS PageView
;
// Compute page views
@Impressions =
PROCESS @Impressions
PRODUCE *
REQUIRED ClientId, HtmlContent(Header, Footer)
USING new Bing.HtmlProcessor()
;
// Users metrics
@Metrics =
SELECT ClientId, Market, Header FROM @Sessions
WHERE
Market == "en-us"
;
U-SQL Optimizations
Column Pruning and dependencies
C H M
C H M
C H M
Column Pruning
• Minimize I/O (data shuffling)
• Minimize CPU (complex processing, html)
• Requires dependency knowledge:
• R(D*) = Input ( Output )
• Default no pruning
• User code has to honor reduced columns
A B C D E F G J KH I … M … 1000
36. • Use SELECT with UDFs instead of PROCESS
• Use User-defined Aggregators instead of REDUCE
• Hint Cardinality if you use CROSS APPLY and it does chose the
wrong plan
• Avoid ORDER BY unless needed (OUTPUT, “Top N Rows”)
• Learn to use Windowing Functions (OVER expression)
• Use SQL.MAP and SQL.ARRAY instead of C# Dictionary and array
37. Multiple INSERTs into same table
• Generates separate file per insert in physical
storage:
• Can lead to performance degradation
• Recommendations:
• Try to avoid small inserts
• Rebuild table after frequent insertions with:
ALTER TABLE T REBUILD;