BenchmarkDotNet - Powerful .NET library for benchmarking

1. BENCHMARKDOTNET -
POWERFUL .NET LIBRARY FOR
BENCHMARKING
Larry Nung

2. AGENDA
Introduction
Getting started
Jobs
Columns
Diagnosers
Exporters
Params
Setup
Baseline
Reference
Q & A 2

3. INTRODUCTION
3

4. INTRODUCTION
 A powerful .NET library for benchmarking
 Standard benchmarking routine: generating an isolated
project per each benchmark method; auto-selection of
iteration amount; warmup; overhead evaluation; statistics
calculation; and so on.
 Supported runtimes: Full .NET Framework, .NET Core (RTM),
Mono
 Supported languages: C#, F#, and Visual Basic
 Supported OS: Windows, Linux, MacOS
 Easy way to compare different environments
(x86 vs x64, LegacyJit vs RyuJit, and so on; see: Jobs)
 Reports: markdown, csv, html, plain text, png plots.
 Advanced features: Baseline, Params
 Powerful diagnostics based on ETW events
(see BenchmarkDotNet.Diagnostics.Windows) 4

5. GETTING STARTED
5

6. INSTALLATION
 Install-Package BenchmarkDotNet
6

7. INSTALLATION
7

8. INSTALLATION
8

9. INSTALLATION
9

10. INSTALLATION
10

11. WRITE CODE TO BENCHMARK
using BenchmarkDotNet.Attributes;
…
public class ProgramBenchmarker {
protected Program m_Program { get; set; } = new
Program();
[Benchmark] public void Test() {
m_Program.Test();
}
}
…
11

12. RUN THE BENCHMARK
using BenchmarkDotNet.Running;
…
public class Program {
static void Main(string[] args) {
var summary =
BenchmarkRunner.Run<ProgramBenchmarker>();
}
public void Test() {… }
}
…
}
12

13. VIEW RESULTS
13

14. ANALYZE RESULTS
14

15. JOBS
15

16. JOBS
 Describes how to run your benchmark
 Avaliable Jobs
 DryJob
 ClrJob
 CoreJob
 MonoJob
 LegacyJitX86Job
 LegacyJitX64
 RyuJitX64Job
 SimpleJob
 LongRunJob
 MediumRunJob
 ShortRunJob
 VeryLongRunJob 16

17. JOBS
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Attributes.Jobs;
...
[ShortRunJob]
public class ProgramBenchmarker {
...
}
17

18. JOBS
18

19. COLUMNS
19

20. COLUMNS
 A column in the summary table
 Available Columns
 NamespaceColumn
 MedianColumn
 MinColumn
 MaxColumn
 RankColumn
 e.t.c
20

21. COLUMNS
using BenchmarkDotNet.Attributes.Columns;
…
[NamespaceColumn]
[MedianColumn]
[MinColumn]
[MaxColumn]
[RankColumn]
[RankColumn(NumeralSystem.Roman)]
[OrderProvider(SummaryOrderPolicy.FastestToSlowest)]
public class ProgramBenchmarker {
…
} 21

22. COLUMNS
22

23. COLUMNS
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Columns;
using BenchmarkDotNet.Configs;
…
[Config(typeof(Config))]
public class ProgramBenchmarker {
private class Config : ManualConfig {
public Config() {
Add(new TagColumn("HashCode", item =>
item.GetHashCode().ToString()));
}
}
…
}
23

24. COLUMNS
24

25. COLUMNS
using BenchmarkDotNet.Columns;
using BenchmarkDotNet.Reports;
using BenchmarkDotNet.Running;
…
public class HashCodeColumn : IColumn {
public string ColumnName { get; } = "HashCode";
public HashCodeColumn() { }
public bool IsDefault(Summary summary, Benchmark benchmark) =>
false;
public string GetValue(Summary summary, Benchmark benchmark) =>
benchmark.Target.Method.Name.GetHashCode().ToString();
public bool IsAvailable(Summary summary) => true;
public bool AlwaysShow => true;
public ColumnCategory Category => ColumnCategory.Custom;
public string Id { get; } = "1";
public int PriorityInCategory { get; } = 0;
public override string ToString() => ColumnName;
}
25

26. COLUMNS
…
[Config(typeof(Config))]
public class ProgramBenchmarker {
private class Config : ManualConfig {
public Config() {
Add(new HashCodeColumn()));
}
}
…
} 26

27. DIAGNOSERS
27

28. DIAGNOSERS
 Can attach to your benchmark and get some useful
info
 Available Diagnosers
 MemoryDiagnoser
28

29. DIAGNOSERS
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Diagnosers;
...
[Config(typeof(Config))]
public class ProgramBenchmarker {
private class Config : ManualConfig {
public Config() {
Add(MemoryDiagnoser.Default);
}
}
...
} 29

30. DIAGNOSERS
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Diagnosers;
...
[MemoryDiagnoser]
public class ProgramBenchmarker {
...
}
30

31. DIAGNOSERS
31

32. EXPORTERS
32

33. EXPORTERS
 Allows you to export results of your benchmark in
different formats.By default, files with results will be
located
in .BenchmarkDotNet.Artifactsresults directory.
 Available Exporters
 HtmlExporter
 CsvExporter
 MarkdownExporter
 AsciiDocExporter
 CsvMeasurementsExporter
 PlainExporter
 JsonExporter
33

34. CONFIG
using BenchmarkDotNet.Attributes.Exporters;
…
[AsciiDocExporter]
[CsvMeasurementsExporter]
[PlainExporter]
[JsonExporter]
public class ProgramBenchmarker {
…
}
34

35. CONFIG
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Exporters;
using BenchmarkDotNet.Exporters.Csv;
using BenchmarkDotNet.Exporters.Json;
…
[Config(typeof(Config))]
public class ProgramBenchmarker {
private class Config : ManualConfig {
public Config() {
Add(AsciiDocExporter.Default);
Add(CsvMeasurementsExporter.Default);
Add(PlainExporter.Default);
Add(JsonExporter.Default);
}
}
…
}
35

36. HTMLEXPORTER
36

37. MARKDOWNEXPORTER
37

38. CSVEXPORTER
38

39. ASCIIDOCEXPORTER
39

40. CSVMEASUREMENTSEXPORTER
40

41. PLAINEXPORTER
41

42. JSONEXPORTER
42

43. PARAMS
43

44. PARAMS
using BenchmarkDotNet.Attributes;
...
public class ProgramBenchmarker {
[Params(100, 200)]
public int Parameter { get; set; }
protected Program m_Program { get; set; } = new
Program();
[Benchmark] public void Test() {
m_Program.Test();
}
}
44

45. PARAMS
45

46. SETUP
46

47. SETUP
using BenchmarkDotNet.Attributes;
...
public class ProgramBenchmarker {
protected Program m_Program { get; set; }
[Setup]
public void Setup() {
m_Program = new Program();
}
[Benchmark]
public void Test() {
m_Program.Test();
}
} 47

48. BASELINE
48

49. BASELINE
using System.Threading;
using BenchmarkDotNet.Attributes;
...
public class ProgramBenchmarker {
protected Program m_Program { get; set; } = new Program();
[Benchmark(Baseline = true)]
public void Test1() {
m_Program.Test();
Thread.Sleep(10);
}
[Benchmark]
public void Test2() {
m_Program.Test();
Thread.Sleep(20);
}
} 49

50. BASELINE
50

51. REFERENCE
51

52. REFERENCE
 NuGet Gallery | BenchmarkDotNet 0.10.3
 https://www.nuget.org/packages/BenchmarkDotNet
 dotnet/BenchmarkDotNet: Powerful .NET library for
benchmarking
 https://github.com/dotnet/BenchmarkDotNet
 Home - BenchmarkDotNet Documentation
 http://benchmarkdotnet.org/
52

53. Q&A
53

54. QUESTION & ANSWER
54