As known from the Internet of Things (IoT) testing there also exist multiple challenges for the Edge Computing (EC) quality assurance and automated testing process. Developers and QA experts need to understand specific requirements and possible approaches to be applied in Edge Computing Test design, definition and execution. Special attention will be given to existing approaches, testing techniques and tools which follow standardized methods, are freely available and successfully applied for various mobile and fixed network solutions.

1. Fraunhofer FOKUS
Institute for Open Communication Systems
Testing Challenges and Approaches
in Edge Computing
Axel Rennoch, Dr. Alexander Willner, Sascha Hackel | ECW Berlin | 09.03.21

2. Agenda
1 Background
2 Edge Computing
3 EC Testing Challenges
4 Testing Approaches for EC
5 Conclusion
2

3. 1 Background

4. How is our view on Edge Computing characterized?
4

5. Fraunhofer is Europe’s largest application oriented research
organization:
Fraunhofer Society
©
Michael
Zalewski/
Fraunhofer
FOKUS
©
Fraunhofer-Gesellschaft
> 29.000
Employees
75
Institutes and research units
> 2.8 billion €
Budget (1/3 government, 1/3 public, 1/3 industry)
5
Fraunhofer Institute
for Open Communication Systems (FOKUS)
We connect everything
secure, reliable, sustainable
Fraunhofer Institute FOKUS
The largest Fraunhofer ICT institute (~450 employees).
Located in Berlin.

6. Fields of application and strategic topics of Fraunhofer FOKUS
6
STRATEGIC TOPICS
FIELDS OF APPLICATION
Sustainability
Artificial
Intelligence
Digital
Life
Security/
Certification
Digital
Governance
Digital
Networking
(e.g. 5G/6G)
Quantum
Computing

7. 2 Edge Computing

8. What is Edge Computing?
8

9. Based on: Peter Levine: Return to the Edge and the End of Cloud Computing
Edge Computing: A Distributed Computing Paradigm

10. Who needs Edge Computing?
10

11. Source: Based on Texas Instruments and Moor Insights & Strategy's report Segmenting the Internet of Things (IoT)
Various application domains with specific requirements
11
INDUSTRIAL
Internet of Things
CONSUMER
Internet of Things
Business
Value
Personal
Interest
Smart Manufacturing
Smart Energy
Smart Agriculture
Smart City
Smart Transportation
Smart Phones
Smart Wearables
Smart TV
Smart Appliances
Smart Home
Connectivity
Data Exchange
e-Health
Smart Plugs

12. Where is the Edge?
12

13. Where is the Edge?
13
13

14. Is there more?
14

15. The Reference Architecture Model Edge Computing (RAMEC)
15
Willner, A., & Gowtham, V. (2020, August 10). Towards a Reference Architecture Model for Industrial Edge Computing. IEEE Communications Standards Magazine

16. The Reference Architecture Model Edge Computing (RAMEC)
18
Willner, A., & Gowtham, V. (2020, August 10). Towards a Reference Architecture Model for Industrial Edge Computing. IEEE Communications Standards Magazine
Administration Shell
- Information Model
- APIs
- Connectivity
- …
Assets
Manifest
Component
Manager
Asset
Asset
…

17. An initial approach to cluster the activities
19
Initiatives
➢ International (IIC, ECC,
EECC, …)
➢ EU (GAIA-X, EPI, …)
➢ Germany (PI4.0/ LNI4.0/
SCI4.0/Bitkom/ZVEI/…)
➢ France (AIDF)
➢ Italy (I4.0)
➢ Austria (I4.0Ö)
➢ Spain (IC4.0)
➢ Poland (Przemysł 4.0)
➢ Netherlands (Smart Industry)
➢ UK (4IR)
➢ Ukraine (APPAU)
➢ …
Software Focus
➢ LF Edge: Akraino,
Baetyl, EdgeX
Foundry, EVE, Fledge,
HomeEdge, Open
Glossary
➢ LF ELISA
➢ Linaro
➢ ONF CORD
➢ OEC
➢ OSADL
➢ OpenStack EC
➢ KubeEdge
➢ Discovery
➢ …
Standards
➢ ETSI (MEC)
➢ ETSI (MANO)
➢ IEEE (TSN)
➢ IEEE (1934)
➢ ISO/IEC (SC38)
➢ ISO/IEC (62541)
➢ ISO/IEC (61499)
➢ DIN (92222)
➢ IETF
➢ OMG
➢ ITU
➢ NIST
➢ MSP
➢ …
Verticals
➢ 5G-ACIA
➢ SmartFactoryKL
➢ OPC Foundation
➢ ODVA NewTec
➢ FFPA
➢ OI4.0
➢ AECC
➢ 5GAA
➢ MobileEdgeX
➢ TIP EC
➢ OPAF
➢ …

18. ➢ Industrial Internet Consortium
❖ Parallel Edge and Testbed working groups
➢ Alliance for Internet of Things Innovation (AIOTI)
❖ Multiple working groups and IG Testbeds
➢ ETSI ISG MEC
❖ Conformance test developments and hackathons
❖ Online edge emulation environment (Sandbox)
➢ oneM2M
❖ Conformance test development and plugfests
Testing initiatives and standards
20

19. Agenda
1 Background
2 Edge Computing
3 EC Testing Challenges
4 Testing Approaches for EC
5 Conclusion
21

20. 3 EC Testing Challenges

21. ➢ Distributed, heterogeneous architectures and interfaces
❖ Interoperability and Security requirements
❖ Multiple vendor devices/platforms
❖ Domains specific applications, objects and functions (under test), use cases
❖ Synergies and reusability of test harness from different verticals?
➢ Management at the edge (nodes)
❖ Mobility of devices
❖ Elasticity and fluctuation of data workload (container/kubernetes pods etc.)
❖ Provisioning, orchestration and coordination (deterministic capabilities, SDN)
❖ Edge Artificial Intelligence
➢ Performance/robustness and resource restrictions at IoT layer
❖ Big amount of devices (capacity and scalability)
❖ Ultra-low latency (≈ ms) is critical for real-time type of application
❖ Limited connectivity and power consumption (limited autonomy)
Specific EC aspects for testing
23

22. ➢ Distributed, heterogeneous architectures and interfaces:
Scope / Testing types for EC
❖ Conformance and Security (before commissioning safety-critical systems)
❖ The human element must be considered in all aspects of safety
❖ Interoperability
❖ Performance, real-time aspects
➢ Management at the edge (nodes):
Systematic Test definition/specification
❖ Test suite structuring: Test configurations; Implementation under test (IUT) roles (server/client)
❖ Test objectives (test purpose catalogs), incl. new AI requirements (e.g. robustness, comprehensibility)
❖ Use case scenarios (test case implementation)
➢ Performance/robustness and resource restrictions at IoT layer:
(Automated) Testbeds and software architecture (testware/tools/simulations)
❖ Demonstrate readiness of domain-specific solutions
❖ Certification
Challenges for Edge Computing Testing
24

23. 4 Testing Approaches for EC

24. ➢ Requirements
❖ Open for different software platform and toolsets
❖ Vendor independence
❖ Abstract description of scenarios
❖ Open source
➢ Testing specification and implementations
❖ Robot
❖ TDL
❖ TTCN-3
Testing Methods and Techniques
26
Edge
Testing
Software
Testing
System
Testing
Security
Testing
Test
Automation
Protocol
Testing

25. ➢ Recall Mobile communication and IoT domains
❖ Multiple layer and protocols
❖ Import of domain specific notations
❖ Interoperability and Security issues
❖ Inclusion of operational phase
➢ Recall Existing standardized Testing Frameworks
❖ 3GPP RAN5: Mobile terminal conformance testing
❖ ETSI TC MTS: Methodology for RESTful APIs
specifications and testing
❖ ETSI ISG MEC: Multi-access Edge Computing (MEC)
testing
❖ ETSI ISG NFV: Network Functions Virtualisation (NFV)
testing methodology
❖ oneM2M TDE: Testing and Developers Ecosystem
(TDE)
Learning from existing Test approaches
27
IIC
Connectivity
Framework

26. ➢ MEC Testing Framework
(ETSI GR MEC-DEC 025 V2.1.1 (2019-06))
❖ Abstract Test Method
❖ Conformance Test Purposes
❖ Interoperability Test Descriptions
➢ API Conformance Test Specification
(ETSI GS MEC-DEC 032 V2.1.1 (2020-12))
❖ Part 1: Test Requirements and
Implementation Conformance Statement (ICS)
❖ Part 2: Test Purposes (TP)
❖ Part 3: Abstract Test Suite (ATS)
Multi-access Edge Computing (MEC)
28
MEC
Testing
and
Conformance

27. ➢ Eclipse IoT-Testware
❖ MQTT and CoAP
❖ Executable conformance test suite (open source)
➢ Testbeds/Simulations
❖ 5G Playground (FOKUS)
❖ Federated testbeds (FED4FIRE+)
❖ IoTSim-Edge: A simulation framework
➢ Monitoring, prediction etc.
❖ Continuous testing during operational phase
❖ Information Security Indicator Monitoring (sources,
databases)
Further Examples
29

28. ➢ Systematic collection of requirements and related test purpose definition
❖ Unique specification and Common interpretation
➢ Application of Standardized (Abstract) Test approaches
❖ Approved Methods and Basis for certification (extended CommonCriteria)
➢ High performance challenges
❖ Apply simulation and/or continuous offline analysis (from operational phase)
➢ Consideration of new AI requirements and testing methods
❖ E.g. adversarial/corruption robustness evaluation
❖ Risk analysis and Collection of common test data
Main recommendation for EC testing
30

29. 5 Conclusion

30. ➢ Multiple different aspects of EC are under discussion
❖ Various working groups of standardization bodies and
industrial associations
❖ Technical viewpoints differ due to the various
stakeholders
➢ Landscape documents already support the interested
experts and public community
❖ Missing QA, Testing and Certification
➢ A need for harmonization and common strategies
❖ More emphasise on quality and testing
➢ Edge Computing Consortium (EECC): in Europe, in
preparation
❖ ECW Europe Workshop: 11th March 2021
Summary
32

31. Fraunhofer FOKUS
Institute for Open Communication Systems
Kaiserin-Augusta-Allee 31
10589 Berlin, Germany
https://www.fokus.fraunhofer.de/en/sqc
https://www.fokus.fraunhofer.de/ngni
SQC: Axel Rennoch & Sascha Hackel
Phone +49 (30) 3463 – 7344 / 7255
axel.rennoch@fokus.fraunhofer.de
sascha.hackel@fokus.fraunhofer.de
NGNI: Dr. Alexander Willner
Phone +49 (30) 3463 - 7116
alexander.willner@fokus.fraunhofer.de
Thank you for your attention!