Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Biml Tips and Tricks: Not Just for SSIS Packages! (SQLBits 2019)

2.539 Aufrufe

Veröffentlicht am

Biml Tips and Tricks: Not Just for SSIS Packages! (Presented at SQLBits on March 2nd, 2019)

Veröffentlicht in: Daten & Analysen
  • Als Erste(r) kommentieren

  • Gehören Sie zu den Ersten, denen das gefällt!

Biml Tips and Tricks: Not Just for SSIS Packages! (SQLBits 2019)

  1. 1. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Session Description "Wait, what? Biml is not just for generating SSIS packages?" Absolutely not! Come and see how you can use Biml (Business Intelligence Markup Language) to save time and speed up other Data Warehouse development tasks. You can generate complex T-SQL statements with Biml instead of using dynamic SQL, create test data, and even populate static dimensions. Don't Repeat Yourself, start automating those boring, manual tasks today!
  2. 2. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) @cathrinew cathrinew.net
  3. 3. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
  4. 4. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) …the next 50 minutes… T-SQL Test Data Dimensions
  5. 5. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Wait… Can't I use <something else>?
  6. 6. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Of course! But Biml works well with source metadata
  7. 7. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) And… Biml is fun!
  8. 8. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) But... Biml? How?
  9. 9. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Traditional Biml
  10. 10. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Crazy Fun Biml
  11. 11. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) What do you need?
  12. 12. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Preview Pane The Power is in the...
  13. 13. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Traditional BimlScript
  14. 14. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Traditional BimlScript <# foreach (var table in RootNode.Tables) { #> <Package Name="Load_<#=table.Name#>" /> <# } #> <Package Name="Load_Customer" /> <Package Name="Load_Product" /> <Package Name="Load_Sales" />
  15. 15. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) BimlScript to Biml BimlExpress Preview Pane
  16. 16. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) BimlScript to... ??? BimlExpress Preview Pane
  17. 17. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Crazy Fun BimlScript <# foreach (var table in RootNode.Tables) { #> Yay, a package! <#=table.Name#> :) <# } #> Yay, a package! Load_Customer :) Yay, a package! Load_Product :) Yay, a package! Load_Sales :)
  18. 18. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
  19. 19. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) T-SQL from Biml
  20. 20. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) GetColumnList() GetColumnAssignmentList() GetColumnComparisonList()
  21. 21. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml Column Methods Return code fragments Use as building blocks in custom T-SQL Customize output by passing parameters Filter columns by using lambda expressions
  22. 22. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Column Methods Return code fragments Use as building blocks in custom T-SQL Customize output by passing parameters Filter columns by using lambda expressions
  23. 23. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Lambda Expressions "A lambda expression is an anonymous function that you can use to create delegates or expression tree types"
  24. 24. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Lambda Expressions "A lambda expression is an anonymous function that you can use to create delegates or expression tree types"
  25. 25. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Lambda Expressions column => column.Name == "ColumnID"
  26. 26. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Lambda Expressions column => column.Name == "ColumnID" The arrow is the lambda operator
  27. 27. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Lambda Expressions column => column.Name == "ColumnID" Input parameter is on the left side
  28. 28. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Lambda Expressions column => column.Name == "ColumnID" Expression is on the right side
  29. 29. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Lambda Expressions column => column.Name == "ColumnID"
  30. 30. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) GetColumnList() Get columns for SELECT:
  31. 31. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) GetColumnList() [Col1], [Col2], [Col3]
  32. 32. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) GetColumnList("src") [src].[Col1], [src].[Col2], [src].[Col3]
  33. 33. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) GetColumnComparisonList() Get columns for JOIN … ON:
  34. 34. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) GetColumnComparisonList() [l].[Col1] = [r].[Col1] AND [l].[Col2] = [r].[Col2] AND [l].[Col3] = [r].[Col3]
  35. 35. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) GetColumnComparisonList("!=") [l].[Col1] != [r].[Col1] AND [l].[Col2] != [r].[Col2] AND [l].[Col3] != [r].[Col3]
  36. 36. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) GetColumnComparisonList("src", "dst") [src].[Col1] = [dst].[Col1] AND [src].[Col2] = [dst].[Col2] AND [src].[Col3] = [dst].[Col3]
  37. 37. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) GetColumnComparisonList ("!=", "src", "dst") [src].[Col1] != [dst].[Col1] AND [src].[Col2] != [dst].[Col2] AND [src].[Col3] != [dst].[Col3]
  38. 38. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) GetColumnComparisonList ("!=", "src", "dst", " OR ",) [src].[Col1] != [dst].[Col1] OR [src].[Col2] != [dst].[Col2] OR [src].[Col3] != [dst].[Col3]
  39. 39. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) GetColumnComparisonList GetColumnComparisonList() GetColumnComparisonList("!=") GetColumnComparisonList("dst", "src") GetColumnComparisonList(c => c.IsUsedInPrimaryKey) GetColumnComparisonList(c => c.IsUsedInPrimaryKey, "dst", "src") GetColumnComparisonList(c => c.IsUsedInPrimaryKey, "!=", "dst", "src")
  40. 40. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) GetColumnAssignmentList() Get columns for UPDATE … SET:
  41. 41. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) GetColumnAssignmentList() [l].[Col1] = [r].[Col1] ,[l].[Col2] = [r].[Col2] ,[l].[Col3] = [r].[Col3]
  42. 42. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) GetColumnAssignmentList("src", "dst") [src].[Col1] = [dst].[Col1] ,[src].[Col2] = [dst].[Col2] ,[src].[Col3] = [dst].[Col3]
  43. 43. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) DEMO T-SQL from Biml
  44. 44. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Test Data
  45. 45. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml and Bogus Create Random and Specific Test Data
  46. 46. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Bogus Project Simple and sane fake data generator for .NET https://github.com/bchavez/Bogus Advanced functionality for custom objects …or simple functionality for Biml hacks :)
  47. 47. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Address Commerce Company Database Date Finance Internet Lorem Name Person Phone System Bogus API Supports all data types plus built-in test data, including:
  48. 48. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Bogus Methods <#@ assembly name="Bogus.dll" #> <#@ import namespace="Bogus" #> <# var f = new Faker(); #> <# Person p = new Person(); #> INSERT INTO dbo.Employee(FirstName, LastName, Birthday) VALUES ( '<#=p.FirstName#>', '<#=p.LastName#>', '<#=p.DateOfBirth.ToString("yyyyMMdd")#>' )
  49. 49. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) BOgus Installation Install via Nuget: Install-Package Bogus Or download latest release .zip: https://github.com/bchavez/Bogus/releases
  50. 50. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) DEMO Test Data with Bogus
  51. 51. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Static Dimensions
  52. 52. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Static Dimensions SCD Type 0: Not changing • Unknown dimension members • Date dimension • Code tables
  53. 53. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Static Dimension Creation 1. Create table definition 2. Add static source rows 3. Generate INSERT statements
  54. 54. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml Static Sources Part of <Table> objects Defines rows to be inserted when table is created
  55. 55. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml Static Source <Table Name="Table" SchemaName="Database.Schema"> <Columns> <Column Name="Column1" DataType="Int32" /> <Column Name="Column2" DataType="String" Length="10" /> </Columns> <Sources> <StaticSource Name="TableRows"> ... </StaticSource> </Sources> </Table>
  56. 56. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml Static Source <StaticSource Name="TableRows"> <Rows> <Row> <ColumnValues> <ColumnValue ColumnName="Col1" Value="-1" /> <ColumnValue ColumnName="Col2" Value="'N/A'" /> </ColumnValues> </Row> </Rows> </StaticSource>
  57. 57. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml Static Source <# if (table.Sources.Any()) { #> <#=TableToPackageLowerer.GetStaticSourceInsertStatements( table.SchemaQualifiedName, table.HasIdentity, (AstTableStaticSourceNode)table.Sources.FirstOrDefault() )#> <# } #>
  58. 58. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) DEMO Static Dimensions
  59. 59. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) …the Past 50 minutes… T-SQL Test Data Dimensions
  60. 60. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Is this useful?
  61. 61. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) What other ideas do you have?
  62. 62. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Have fun!
  63. 63. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) @cathrinew cathrinew.net hi@cathrinew.net Biml resources and demo files: cathrinew.net/biml

×