Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
Research Institutes of Sweden
ENABLING VISUAL
ANALYTICS WITH
- Exploring Regression Test Results
in ASIC Verification
Mark...
2
Automation in ASIC verification generates large
amounts of test results
Visual analytics through a game engine helps
foc...
Development engineer, ABB, Malmö, Sweden 2007-2010
 Editor and compiler development
 Process automation
PhD student, Lun...
 MSc thesis student Andreas Brytting
 ASIC verification tool vendor
 Research project
4
Credits
5
WP3 Test effectiveness
Task 2 Test result overload
 ASIC design vs. software engineering
 Challenge and solution proposal
 Tool demo
 Evaluation plans
6
Outline
7
ASIC development
ASIC
Verification
ASIC
Design
output from design phase
register-transfer level
(coding)
functional corr...
 Abstraction
 hardware description languages (VHDL,
Verilog, SystemVerilog, … )
 Version control systems
 Test automat...
 What to verify?
 Functional verification
 Timing verification
 Performance verification
 How to verify?
 Simulation...
 Directed testing
 Manually craft test cases
 High maintenance costs
 Good for finite condition spaces
 Random testin...
11
Peculiarities of ASIC verification
Bugs in code escape to silicon => ”re-spin”
70% effort and critical path
Exhaustive ...
 CPU designs are increasingly complex
 New hardware architectures
 multicore
 multithreading
 …
 Requirements on sma...
 Test automation and regression testing
 As often as possible:
 Check out the latest source code
 Run test cases
 Rep...
Large amounts of test results
Test result matrices used to hide information
14
The backside…
Wikimedia Commons, Brukar:Bep...
 One of the largest chip
manufacturers worldwide
 Development of a CPU
 Source code
 61,200 files (~3.8 GB)
 Verilog,...
We target three use cases:
1. General exploration of the design-under-test (DUT)
2. Localization of error-prone parts of t...
 Why game engine?
 Interaction out-of-the-box
 Why Unity?
 Fairly simple
 Scales well
 Very popular
 Unity???
 Cro...
18
 Each building is a file, height represents #commits
 Organized in blocks based on folder structure
 Color indicates fr...
 Two senior verification engineers in India
 Generally positive
 First-person shooter controls not necessarily intuitiv...
 Filtering of commit sets
 Search lights
 Tooltips
 Search for folder name
 Export to log files
21
Prototype evolved ...
22
Tool demo
 Explore city layouts adhering to
physical mapping, i.e., “floor plan”
 Evaluate in focus groups with
additional verific...
24
markus.borg@ri.se
@mrksbrg
mrksbrg.com
Automation in ASIC verification generates large
amounts of test results
Visual a...
Nächste SlideShare
Wird geladen in …5
×

Enabling Visual Analytics with Unity - Exploring Regression Test Results in ASIC Verification

300 Aufrufe

Veröffentlicht am

Invited talk at TOCSYC project meeting, November 17, 2017. Västerås, Sweden.

Veröffentlicht in: Wissenschaft
  • Als Erste(r) kommentieren

  • Gehören Sie zu den Ersten, denen das gefällt!

Enabling Visual Analytics with Unity - Exploring Regression Test Results in ASIC Verification

  1. 1. Research Institutes of Sweden ENABLING VISUAL ANALYTICS WITH - Exploring Regression Test Results in ASIC Verification Markus Borg, Andreas Brytting, Daniel Hansson November 17, 2017 Swedish Institute of Computer Science Software and Systems Engineering Laboratory
  2. 2. 2 Automation in ASIC verification generates large amounts of test results Visual analytics through a game engine helps focusing design and verification effort
  3. 3. Development engineer, ABB, Malmö, Sweden 2007-2010  Editor and compiler development  Process automation PhD student, Lund University, Sweden 2010-2015  Machine learning for software engineering  Bug report management and traceability Senior researcher, RISE SICS AB, Lund, Sweden 2015-  Software engineering for machine learning  V&V for self-driving cars 3 Markus Borg?
  4. 4.  MSc thesis student Andreas Brytting  ASIC verification tool vendor  Research project 4 Credits
  5. 5. 5 WP3 Test effectiveness Task 2 Test result overload
  6. 6.  ASIC design vs. software engineering  Challenge and solution proposal  Tool demo  Evaluation plans 6 Outline
  7. 7. 7 ASIC development ASIC Verification ASIC Design output from design phase register-transfer level (coding) functional correctness probing, temperatures system level, customer applications
  8. 8.  Abstraction  hardware description languages (VHDL, Verilog, SystemVerilog, … )  Version control systems  Test automation  Coverage testing  Steps toward continous integration 8 Software engineering influences ASIC development (NCSU 2017)
  9. 9.  What to verify?  Functional verification  Timing verification  Performance verification  How to verify?  Simulation-based verification  Emulation-based verification (FPGA)  Formal verification 9 ASIC verification
  10. 10.  Directed testing  Manually craft test cases  High maintenance costs  Good for finite condition spaces  Random testing  Automatically generate test cases  Simple to build generator  No maintenance  Needs infinite compute cycles 10 Test case creation constrained random testing & coverage metrics
  11. 11. 11 Peculiarities of ASIC verification Bugs in code escape to silicon => ”re-spin” 70% effort and critical path Exhaustive testing possible? Huge ”test farms” Tool vendors get paid www.erikjohanssonphoto.com
  12. 12.  CPU designs are increasingly complex  New hardware architectures  multicore  multithreading  …  Requirements on small form factors  Trade-off between high performance and low power consumption  Business demands on short time-to- market 12 Contemporary challenges in ASIC verification Purna Mohanty, Aims Technology
  13. 13.  Test automation and regression testing  As often as possible:  Check out the latest source code  Run test cases  Report results  Blame all committers since last pass 13 State-of-practice solution
  14. 14. Large amounts of test results Test result matrices used to hide information 14 The backside… Wikimedia Commons, Brukar:Bep Creative Commons CC-BY 2.5
  15. 15.  One of the largest chip manufacturers worldwide  Development of a CPU  Source code  61,200 files (~3.8 GB)  Verilog, SystemVerilog, assembler, C, and shell scripts  95 committers  Regression testing  500 test cases  2 h execution time  3-8 executions per day 15 Case under study – An ASIC project Wikimedia Commons, User: Zollo, CC-BY 2.0
  16. 16. We target three use cases: 1. General exploration of the design-under-test (DUT) 2. Localization of error-prone parts of the design 3. Detection of potential coverage holes 16 Visual analytics (Illinois Applied Research Institute)
  17. 17.  Why game engine?  Interaction out-of-the-box  Why Unity?  Fairly simple  Scales well  Very popular  Unity???  Cross-platform game engine and IDE  Drag-and-drop 2D and 3D scenes  Scripting in C# 17
  18. 18. 18
  19. 19.  Each building is a file, height represents #commits  Organized in blocks based on folder structure  Color indicates fraction of failed test executions 19 Prototype visualization – test execution cityscape
  20. 20.  Two senior verification engineers in India  Generally positive  First-person shooter controls not necessarily intuitive 20 Prototype evaluation Flickr: billsoPHOTO CC BY-SA 2.0
  21. 21.  Filtering of commit sets  Search lights  Tooltips  Search for folder name  Export to log files 21 Prototype evolved into a tool
  22. 22. 22 Tool demo
  23. 23.  Explore city layouts adhering to physical mapping, i.e., “floor plan”  Evaluate in focus groups with additional verification engineers  Controlled experiment with students  Task: distribute verification effort  Treatment: Cityscape or matrix  Measurement: Time and coverage 23 Future work Flickr: jamesjoel CC BY-ND 2.0
  24. 24. 24 markus.borg@ri.se @mrksbrg mrksbrg.com Automation in ASIC verification generates large amounts of test results Visual analytics through a game engine helps focusing design and verification effort

×