Windows Hardware Certification Kit (HCK) is a set of tools, processes, and tests for certifying HW devices, device drivers and systems. Being a great test environment for QEMU devices, Windows Guest device drivers and related Host subsystems, it's still frightening due to deployment complexity. We'll share a way to deploy HCK setup(s) on top of QEMU VMs in just a few minutes.
3. What is HCK/WHQL?
The Windows certification program (previously known
as the Windows Logo Program) for devices and system
A comprehensive devicesystem test toolkit by
Microsoft
Devices which pass HCK are certified and their binaries
are digitally signed
Daynix Computing LTD
3
4. HCK test kit
Introduced with Windows 8 (previous kit know as WLK)
Predefined test suites for each device type
Ability to add your own tests for additional testing
outside of the certification process
Reproducible scenarios
Daynix Computing LTD
4
5. Benefits for QEMU and
Linux community
QEMU device testing (not only virtio)
Host subsystems testing, for example networking
Extensive coverage for defined device categories
Regression testing
Stress testing
Daynix Computing LTD
5
6. HCK PCI compliance test
Device under test compliance to PCI spec
Applicable for:
New QEMU chipsets
PCIe devices
Next generation of virtio devices
Existing PCI devices
Daynix Computing LTD
6
7. HCK NDIS (network) tester (1)
Comprehensive networking test suite
Send/receive,
Stress tests (MPE)
Offload, VLAN, priority, packet filtering
etc
Not once used to test the robustness of the host networking
stack
Daynix Computing LTD
7
8. HCK NDIS (network) tester (2)
Applicable for
QEMU network devices
New devices
New features
Host network components
Bridge
Open vSwitch
TAP
Kernel
Daynix Computing LTD
8
9. HCK Windows OS tests
Device path exerciser
Identifies drivers that do not correctly handle the
different IO calls
Driver verifier
Captures deadlocks, memory leaks, memory
corruption
Verified correctness of kernel API usage
Daynix Computing LTD
9
10. HCK power management tests
The test cycling through different sleep states
There are variations of IO and PNP (disable/enable)
during, before and after power transitions
Applicable for:
QEMU Devices power management support
SeaBIOS
Daynix Computing LTD
10
11. Our past experience
Transfer hangs with vhost.
Offload/VLAN related corner cases with network
devices (virtio-net, vmxnet3, vmxnet).
vmxnet3 and pvscsi in QEMU
Open vSwitch out of order issues
RSC (Windows GRO)
PCI compliance
Daynix Computing LTD
11
12. HCK - great, but frightening
Cumbersome setup requirements
Complex network configuration
Hard to replicate setups
Setup conflicts
No power management tests automation support
Daynix Computing LTD
12
14. Introducing VirtHCK
Framework for automating HCK test execution
Based on easily configurable host and guest scripts
Creates isolated virtual network environments
Allows running different HCK scenarios in automated
manner
OSS, distributed under BSD license
Daynix Computing LTD
14
15. Repository and WiKi
Get VirtHCK from https://github.com/daynix/
bricklets.git
VirtHCK wiki with steps for deployment and usage:
https://github.com/daynix/bricklets/wiki/VirtHCKbricklet-documentation
Daynix Computing LTD
15
16. HCK issues solved by VirtHCK
Setup replication
Multiple setups sharing the same host
Setup isolation solves the following issues:
HCK controller controls all HCK clients in the network
Inability to use VM templates without manual VM
reconfiguration
Automatic network topology setup during network
device tests
Daynix Computing LTD
16
17. Setup types supported
Network (virtio-net, vmxnet3, vmxnet)
Storage (virtio-block, virtio-scsi, pvscsi)
Simple PCI (virtio-serial, virtio-balloon)
Easily extendable by small changes in configuration
files
Daynix Computing LTD
17
18. Typical HCK deployment
Test Client
Control NIC
Test Client
Control NIC
Test Client
Control NIC
WHQL control network
Control NIC - e1000
HCK
Controller
Daynix Computing LTD
18
19. VirtHCK setup
Disconnected from outside world network during test
run
Isolated NICs under test network
Isolated WHQL control network
A different number of test clients can run according to
test type
Daynix Computing LTD
19
20. VirtHCK setup for storage and
other devices
Test VM 1
Control NIC - e1000
WHQL control bridge
Control NIC - e1000
Studio VM
“World” bridge
Physical NIC
Outside world NIC
Daynix Computing LTD
20
21. VirtHCK setup for storage and
other devices - running
Test VM 1
Control NIC - e1000
WHQL control bridge
Control NIC - e1000
“World” bridge
Studio VM
Outside world NIC
Physical NIC
X
Daynix Computing LTD
21
22. VirtHCK setup for NICs
NIC under test
Test bridge
NIC under test
Test VM 2
Test VM 1
Control NIC - e1000
Control NIC - e1000
WHQL control bridge
Control NIC - e1000
Studio VM
“World” bridge
Physical NIC
Outside world NIC
Daynix Computing LTD
22
23. VirtHCK setup for NICs - running
NIC under test
NIC under test
Test bridge
Test VM 2
Test VM 1
Control NIC - e1000
Control NIC - e1000
WHQL control bridge
Control NIC - e1000
“World” bridge
Studio VM
Outside world NIC
Physical NIC
X
Daynix Computing LTD
23
24. VirtHCK - quick start
Clone VirtHCK from github
Create VM images in /images directory (studio and test
client)
Update STUDIO_IMAGE/CLIENT1_IMAGE/
CLIENT2_IMAGE variables in hck_setup.cfg
Run VirtHCK - “sudo ./VirtHCK/hck.sh”
Install Microsoft HCK Studio and Clients software
according to Microsoft documentation Windows HCK
Getting Started and following Checklist for a new client
VM guidelines.
Daynix Computing LTD
24
25. Directory tree
VirtHCK/ - Directory of all VirtHCK scripts, supposed
to run in Linux environment
VirtHCK/guest_tools/ - Directory for scripts required
for Windows guests
Images directory where VM's images should be located
Daynix Computing LTD
25
27. hck_setup.cfg
VirtHCK setup configuration file
Environment configuration
VM start up parameters
Configured accordingly to required HCK setup type
Allows multiple setups on one host
Daynix Computing LTD
27
28. Running multiple setups
Use UNIQ_ID to configure multiple setups
UNIQ_ID will be used as a base for creation of:
VNC or Spice ports
Bridges names
Network interface names, MAC addresses and etc.
Daynix Computing LTD
28
36. Host scripts - hck.sh
Main script to run HCK enviroment
Dumps configuration
Creates network configuration for HCK setup
Runs VMs in loop (crucial during PM tests)
Cleans up all the configurations on exit
Daynix Computing LTD
36
37. VM execution scripts
Executed by hck.sh
run_hck_studio.sh - starts up Studio VM
run_hck_client.sh -starts up Client VM
Configuration according hck_setup.cfg
Daynix Computing LTD
37
38. Network scripts
Passed as parameter to QEMU to add network
interface to appropriate bridge
hck_ctrl_bridge_ifup.sh
hck_test_bridge_ifup.sh
hck_world_bridge_ifup.sh
Daynix Computing LTD
38
40. Studio VM
Setup HCK according to Microsoft’s documentation
Follow the checklist - https://github.com/daynix/bricklets/wiki/VirtHCKbricklet-documentation#wiki-Checklist_for_a_new_studio_VM:
Do not run HCK studio directly
Use run_hck_studio.bat or run_hck_studio_8.1.bat located in
guest_tools
The above scripts will disconnect the HCK Studio VM from outside
network
Daynix Computing LTD
40
41. Test client VMs
Follow the checklist for test client - https://github.com/
daynix/bricklets/wiki/VirtHCK-brickletdocumentation#wiki-Checklist_for_a_new_client_VM:
Install HCK client according to Microsoft’s
documentation from hck-studioHCKInstallClient
setup.exe
Daynix Computing LTD
41
43. What’s next
HCK automation on the guest
SVVP configuration
Xen support (other hypervisors?)
Adding more device types
Using external configuration
Using libvirt
More guest scripts to automate configuration of test
clients
Daynix Computing LTD
43
44. HCK automation on the
guest
Goal:
Run full test cycle for specific device type without
user interaction.
Daynix Computing LTD
44
45. Adding more device types
Goal
Test all the devices in QEMU and not only virtio (PCI
bridges, USB controllers and etc)
Daynix Computing LTD
45
46. More plans
Using external configuration
Multiple setups on one host without replicating
VirtHCK and images
Use libvirt
Arbitrary number of test clients on one setup
Xen support
Daynix Computing LTD
46
47. Call to action
Test QEMU devices with VirtHCK (not only virtio
devices)
Contribute
Daynix Computing LTD
47