Without continuous testing there is no continuous delivery. Everyone in DevOps talks about continuous integration and continuous delivery but those are just two ends of the DevOps pipeline. In the middle of DevOpsis continuous testing (CT), and many organizations are struggling to implement continuous testing effectively. Lab-As-A- Service (LaaS) enhances CT with dynamic on-demand self-serve test topologies. CT together with LAAS make a powerful combination that perfectly serves complex software development and delivery pipelines.
Software Defined Networks (SDNs) turn the network into a flexible configurable system of software components and connections which is powerful but presents new complexities and challenges for network software development, delivery and deployments.
This talk explains how DevOps best practices for continuous testing when combined with LaaS are make an excellent combination to accelerate SDN realization and operations. The talk describes people, process and products required for SDN continuous testing and online self-service continuous lab and testing services.
11. 11Spirent Communications
• Central Admin
• Security
• User Groups
• Pooled resource inventory
• Physical and virtual
• Work-orders
Total Lab Management
16. 16Spirent Communications
Case Study
360 developers, six sites
36 MLOC+, 100+ features
86 build targets
complex interdependencies
200+ topologies
12K test cases
80+ deployment packages
(physical and virtual)
1,280,000,000
test possibilities
17. 17Spirent Communications
The primary contributing
factor was CT and CM !
CT Case Results
Metric
Major release
(#months)
Minor release
(#weeks)
# Features
Defects
Integrations / day
Tests / day
Automated tests
After
3
2
150
10
100
10
85%
Before
6
4
113
1260
0.5
0.3
5%
This presentation explains how continuous testing and Lab-as-a-Service is a perfect match to support complex test topologies such as SDN.
Spirent is a Test and Measurement company that provides systems and services for testing and lab automation, especially targeted to network applications.
It is clear that DevOps is all about continuous delivery and rapid development to delivery pipelines. Well there is no continuous delivery without testing.
While you hear that DevOps CT “shifts left” testing, to get test results early in the process, it is also true that CT practices are across the entire pipeline from beginning to end.
Next generation software applications across many industries involved networked applications arranged in complex network topologies.
DevOps needs to optimize around these topologies.
SDN and NFV are example of networked applications which this presentation uses to CT for networked applications and topologies.
There are many configuration choices which add complexity for testing.
Customers are looking forward to efficiency, operations management improvement and savings.
The test environment for SDN and NFV require both physical and virtual topologies.
Separate SDN and NFV test topologies are required for different types of tests to address concerns such as COST, INTEGRATION and SECURITY.
Topology-centric CT matters because of requirements for topology resource management, connectivity, setup times, flexibility and efficiency.
To address all of the needed for NextGen networked apps and topology-centric CT requires a NextGen CT platform capabilities for lab management, LaaS, Lab Automation, distributed test capabilities and results aggregation.
The large number of nodes, resources, topologies and users demands a total lab management solution.
Users need to be able to create and use topologies from their desktops.
Admin staff need automated applications to help them manage the complexity of labs.
Business apps
Customer service topologies
Inventory audits
Warranty and License Mngt
Security checks
Demonstration setups
Training setups
Operations Apps
Access controls
Inventory audits
Ticket workflow automation
Health check lab infrastructure
Infrastructure resiliency testing
Backup/Restore
Disaster prevention and recovery
Facility shut-down and turn-up
Automate manual workflows
Production Apps
Popular user requests
Change Management
Topologies for Dev, QA, Mnftg
Reduce False Verdicts
System commissioning
Management Apps
KPIs, Dashboards, messages, thresholds, alerts, and escalations
SLAs, Dashboards, messages, thresholds, alerts, and escalations
Compliance audits
Testing of distributed network applications and nodes work best when the test environment tools are topology-capable.
Results from all the nodes need to be aggregated efficiently.
The case study was for a large scale DevOps environment for a network product that was suffering from late deliveries largely due to poor CT implementation.
In this example the customer, a business unit of a large network equipment manufacturer, had disconnected processes and mostly manual testing.
As you can see from the before and after numbers the implementation of a CT system, integrated with CI and CM had the effect of doubling the speed of releases, while at the same time increasing the number of features delivered and greatly reducing the number of defect delivered.
The underlying mechanisms that facilitated the business transformation were the increase in integrations and test cycles per day and the increase in automated testing from 5% to 85%.
Spirent has developed a number of Best Practices tools for topics important to DevOps, Lab management and Test Automation.
The tool is essentially a survey tool that organizes answers to Best Practices questions and uses the answers to compute a GAP score for each practice and each group of practice categories.
Ok so let me recap the thesis of my talk which hopefully has been made clear by now.
While it should not be a secret, for some reason many organizations, even many scholarly publications, have not fully recognized the strategic nature of getting Continuous Testing right in order for DevOps to accomplish it’s primary success goals of innovation, quality, time to market, and ROI.
There is much more to CT than just running some tests after a build, especially for nextgen topology-centric software applications that need to be networked together to operate together such as SDN and NFV.
Topology-centric tools for testing and lab management simplify CT for networked applications.
There are several “free take-ways” that you may be interested in. A number of DevOps CT white papers, an eBook and an Assessment tool.
Thank-you.
If you have any questions feel free to contact me.
