SlideShare ist ein Scribd-Unternehmen logo
1 von 49
Downloaden Sie, um offline zu lesen
Challenges and opportunities in the
integration of the
Systems Engineering process and
the AI/ML Lifecycle
Jose María Alvarez Rodríguez | Associate Professor | josemaria.alvarez@uc3m.es
Let’s speak some AI…
Digitalization
Digital technologies are
an “opportunity”.
European companies
9/10 Adopted 2 of 9 the key
digital technologies
Technology adoption
35% Increase in their annual turnover
being 24% BD and 5% AI
Big Data Analytics & AI
46%
E. Union, “Digital Transformation Scoreboard 2018 EU businesses go digital: Opportunities, outcomes and uptake,” COSME programme of the
European Union, Luxembourg, 978-92-9202-353–9, 2018.
Context
Evolving and competitive market for Cyber-physical
systems (CPS)
Legislation
Business needs and models
Quality
• Safety
• Reliability
• Cyber-security
• Certification
• Environment
• Standardization
• Ethics
• …
Engineering process , methods, technology & tooling
AI/ML & Deep Learning
Self-Driving Cars
Source: https://www.micron.com/insight/on-the-road-to-full-autonomy-self-driving-cars-
will-rely-on-ai-and-innovative-memorya
What can you see?
Self-Driving Cars (object recognition)
Source .https://towardsdatascience.com/how-do-self-driving-cars-see-13054aee2503
Source: https://www.theverge.com/2015/7/17/8985699/stanford-neural-networks-image-
recognition-google-study
Self-Driving Cars (object recognition)
An example of architecture
Is it a Car?
Is it a kind of a computer
(software and hardware
intensive)?
Sensors/Actuators
GPUs
AI/ML models
5G
Edge Computing
…
Source: https://www.aptiv.com/media/article/2018/01/07/the-autonomous-driving-platform-how-will-cars-actually-drive-themselves
Critical factors
Human factors
Adoption of intelligent evolving
systems
Safety & Security
Ethics
Conceptual &
technological
From business needs/value to system
capabilities
Software & Systems
Engineering
AI/ML lifecycle
Perspective
Software Engineering 12
Standard notations
Languages, tools, quality
management, maturity
models, design patterns,
components, etc.
Agile methods & DevOps
Scrum, Kanban, BDD, TDD,
etc.
Software production automation
Model-Driven Engineering/Architecture
Software Product Lines
etc.
Initial Focus
Methods and methodology to
tackle the problem of
software reliability
The way through
There is a huge body of knowledge and practice that allow
us to acknowledge which ARE, and which ARE NOT, the
software engineering practices that really represent
timeless scientific and technological foundations for a
proper software development process.
(My own summary based on [1])
Prof. Barry Boehm• “A Conversation with Barry Boehm: Recollections from 50 Years of
Software Engineering”. IEEE Software, vol. 35 (5). 2018
• “A View of 20th and 21st Century Software Engineering “
Software & Systems Engineering process
ISO 15288:2015ISO 12207:2017
Mats Berglund (Ericsson)
http://www.ices.kth.se/upload/events/13/84404189f85d41a6a7d1cafd0db4ee80.pdf
Engineering (and corporate)
environment
Source: http://beyondplm.com/2014/07/22/plm-implementations-nuts-and-bolts-of-data-silos/
Artificial Intelligence
ELIZA
Medical Support Systems
“We can solve everything”
Turing Test, Conversational Systems &
Expert Systems
AI/ML is not cool!
“We can not solve anything”
AI/ML WINTER
Theoretical foundations
Algorithms & Techniques
From Deep Blue to Alexa
Lack of data, computational power, etc. (not anymore*)
AI/ML Boost
Any x / x ε {domain, problem, place, one, sector, etc.}
AI/ML SPRING & Fear
01
02
03
04
15
The way through
“The new spring in AI is the most significant development
in computing in my lifetime. Every month, there are
stunning new applications and transformative new
techniques. But such powerful tools also bring with them
new questions and responsibilities”
Sergey Brin
INCOSE IS 2019
“AI for Systems Development”
AI/ML Techniques
Data Science Lifecycle
Source: https://docs.microsoft.com/en-us/azure/machine-learning/team-data-science-process/overview
“My” Data Science Lifecycle
Technique
Persistence
Optimization
Validation
Training
Data
Selection
Acquisition
Access
Persistence
Structure
Enriching
Cleaning
Business Needs
Applications
(value)
Exploitation
Analysis
Preliminary
Analysis
(stats / visual)
Data Management
Lifecycle
DML
AI/ML
function
Evolution
Frameworks/Platforms/…→AI/ML as a Service 21
Source: https://www.ibm.com/blogs/research/2018/03/deep-learning-advances/
Source: https://www.morpheusdata.com/blog/2017-09-14-cloud-and-ai-a-work-in-progress-with-unlimited-upside
Source: https://azure.microsoft.com/es-es/services/machine-learning-studio
Source: https://algorithmia.com/
Source: https://cloud.google.com/automl/?hl=es-419
Technical debt in AI/ML functions [2, 4] 22
Feedback loops Correction cascades
Garbage features
Source: https://towardsdatascience.com/technical-debt-in-machine-learning-8b0fae938657
Software & Systems
Engineering
AI/ML lifecycle
Harmonization
Engineering (of→ for) AI/ML
Current trend
Challenges ahead
Opportunities ahead
No clear view
Lack of expertise: from
deterministic to
evolving systems
Engineering of evolving intelligent systems
How to integrate the engineering
process with the AI/ML lifecycle?
Technology available
Methods not ready
Change resistance
Room for improvement
Cultural change and new mindset
Evolving intelligent systems for engineering
How to use intelligence to help
engineering processes?
24
Engineering more intelligent complex
products/services…
…also requires more intelligent
engineering methods and tools.
Harmonization of the SE process and the AI/ML function
Paying the “hidden” technical debt
10 major challenges & coffee evaluation
How we do things now?
Status
Can we define a challenge?
Concept
What to do to tackle the
challenge?
Opportunity
Is it really a challenge?
Is there an opportunity?
Debate
26
…and the necessary tea/coffee cups per hour
• Catalogue of technology
and tools.
• Specify the DML stages.
• Describe a DML using
standards.
• Orchestration of DML
activities.
• Access to functionalities as a
service.
Challenge #1: to describe the needs and capabilities of the
resources which are part of the AI/ML model lifecycle
Status Concept Opportunity Debate
• AI/ML model lifecycle→several
inter-related activities
• Data management lifecycle
(DML) not completely clear
• Tangled environment of
technology and tools
• Lack of standardization
• Description of needs and
capabilities of the AI/ML
model lifecycle
• DML has been already
managed in the context of Big
Data applications and Data
Science.
27
• New set of subsystems: AI/ML
based.
• New set of metrics.
• New set of patterns to specify
requirements/models about AI/ML
functions.
• Adaptation of the CCC in the
context AI/ML functions.
• Blocks for non-deterministic
functions.
• Consistency over time
(evolvability).
• Human factors.
Challenge #2: to integrate the AI/ML model lifecycle within
the specification process of a system
Status Concept Opportunity Debate
• Stages of the AI/ML model lifecycle:
data, training, test, optimization
and operation.
• Many models with different
characteristics and applications.
• Performance metrics: precision,
recall, accuracy, etc. depending on
the type of model.
• Specification of
requirements/models for AI/ML
functions.
• It mainly requires the extension of
existing methods.
28
DSL for requirements (patterns, metrics, etc.) 29
Patterns
P1: The <AI_system> shall <action> <entity>
with a <metric> of <value> <unit> under
<weather_condition_type>.
Vocabulary
Techniques, metrics, types of problems to solve, etc.
Requirements
R1: “The entity recognition subsystem shall
provide an explanation of at least 10 words
about the recognized objects.
R2: “The entity recognition subsystem shall
recognize a person with an accuracy of 90%
under light snow.”
Quality metrics
Consistency over time
Consistency between techniques and
metrics.
Consistency between requirements and
techniques.
• Methods to connect a digital
twin environment.
• Methods to deriving testing
environments.
• Methods for re-verification, re-
validation, re-certification, re-
qualification.
• Virtual and Physical
• Continuous RE-V&V
• Consistency over-time.
• Simulation++++ (and its
orchestration)
Challenge #3: to integrate the AI/ML model lifecycle within the
verification and validation (V&V) process of a system.
Status Concept Opportunity Debate
• AI/ML already used for testing
(data, test case generation,
execution).
• Languages to specify (safety)
test cases, e.g. Open Scenario
(AVL).
• Digital twin environment.
• V&V of evolving systems.
• How to make
deterministic a non-
deterministic system?
• Feedback loop:
• In-lab V&V vs physical
V&V
• Connection between both
sides of the Vee model.
• W model.
• Legal framework.
30
V&V IN THE LOOP! 31
https://www.mathworks.com/content/dam/mathworks/mathworks-dot-com/solutions/automotive/files/in-expo-2014/mil-sil-pil-a-new-paradigm-in-model-based-development.pdf
Digital twin environment 32
Source: https://www.mdpi.com/2079-8954/7/1/7/pdf
Boeing Diamond System Lifecycle
• Automatic generation of
execution environments
(DevOps approach).
• Artificial “Intelligence as a
Service” (AIaaS)
• “Machine Learning as a
Service” (MLaaS) [5]
• Share models?
(competitive advantage)
Challenge #4: to integrate the technology stack of the AI/ML
model lifecycle within the system execution environment
Status Concept Opportunity Debate
• Many technology providers • Include a new technology
stack:
• Hardware
• Software
• Communication system
• Computation system
• In the car industry, there are
already strategic alliances
between:
• Car manufacturers
(OEM)
• Hardware providers
• AI/ML providers
• (Software)
infrastructure providers
• Research centers
34
Technology and Car Companies 35
Source: https://www.bloomberg.com/graphics/2016-merging-tech-and-cars/
• Proper management of
baselines and versions
• Round trip of data.
• Change impact analysis
acquires a new dimension.
• Artefact synch
(consistency +
traceability).
• See Challenge #3
• …
Challenge #5: to integrate the AI/ML model lifecycle within
the configuration and change management processes
Status Concept Opportunity Debate
• Many tools • Configuration more relevant
than ever→more software and
more data dependencies
• Existing techniques and tools
are ready to manage
changes and configuration in
a digital twin environment.
36
• Methods to check the
reusability factor of an
AI/ML model.
• Licensing and catalogue of
datasets (market).
• Licensing and catalogue of
AI/ML models (e.g. like
Modelica libraries)
(market).
• See Challenge #4. Services.
Challenge #6: to ease the reuse of existing AI/ML models
Status Concept Opportunity Debate
• Services • Reuse of existing of AI/ML
models can save time and
costs.
• Increase reliability.
• The Big Data and Data
Science communities have
already passed over here.
37
• Update of methods and
standards.
• Update of architectural
frameworks.
• Sustainability.
• Bring new professionals to
a SE team.
Challenge #7: to standardize the integration of AI/ML model
lifecycle within the engineering process
Status Concept Opportunity Debate
• A new knowledge area comes
to the Systems Engineering
discipline.
• Avoid ad-hoc, non-
controlled or vendor-lock-in
integrations.
• It can be considered as
another part of the software
development process.
38
Framework for Highly Automated Vehicle Safety Validation 39
Source: https://users.ece.cmu.edu/~koopman/pubs/koopman18_av_safety_validation.pdf
“First Comprehensive Autonomous Product Safety Standard (UL 4600)”
Source: https://edge-case-research.com/
• New courses and training
methods.
• New positions: AI/ML
explainer, AI/ML engineer,
etc.
Challenge #8: to educate a new wave of professionals to deal
with AI/ML applying engineering methods
Status Concept Opportunity Debate
• A new knowledge area comes
to the Systems Engineering
discipline.
• Equip people with the
required skills to understand
AI/ML functions.
• These are already covered as
in any other discipline.
40
New roles and positions in AI 41
Source: http://ilp.mit.edu/media/news_articles/smr/2017/58416.pdf
• Study strategies for
optimization of AI/ML
functions at all levels
(hardware, software,
communications, storage).
• Reuse as a primitive.
Challenge #9: to fulfill environmental requirements
Status Concept Opportunity Debate
• A new knowledge area comes
to the Systems Engineering
discipline.
• Ensure the environment
efficiency of the new systems
(more than ever).
• It is something that is already
in the roadmap of any
engineering product.
42
• Certification of ethicality.
Challenge #10: to ensure the construction of ethical machines
Status Concept Opportunity Debate
• MIT Moral Machine
• Ethical guidelines of AI
Systems from the EC.
• Ethics, a necessary non-
functional requirement.
• How to check ethics?
43
Summary
Data Management
Lifecycle
Specification of
AI/ML functions
Continuous V&V
of evolving sys.
Technology stack
integration
Skills, Training &
Education
Standards update Reuse of AI/ML
models
Configuration
and change mgt.
Environmental
footprint
Ethics Legal framework …
01 02 03 04
08 07 06 05
09 10 11 12
44
Keypoints
Conclusions
Key points and next steps 46
Research on AI/ML and Systems
Engineering is a key/active area.
Cyber-physical systems must be safer, cost-
effective and environmentally friendly and…deliver
on time (reacting to market needs).
Engineering processes must integrate a new
discipline and adapt engineering methods to
support the new technical capabilities.
Engineering of intelligent and evolving
systems also requires “intelligent” tools.
1
2
3
4
Ethics is IMPORTANT, Safety a MUST.5
Do NOT be a “technological” slave
Build intelligent systems…
• Integrate the background of AI/ML
experts as a new knowledge area.
• Understand the context of an
evolving system and its implications
in the system lifecycle.
…with intelligence
• Make use of existing and new tools
(do not be afraid of change).
• Build a skilled “intelligent” team.
FUSE WG INCOSEFollow up→SENSEI project
Systems ENgineering methodS for Evolving Intelligent systems
References
1. C. Ebert, “50 Years of Software Engineering: Progress and Perils,” IEEE Softw., vol. 35, no. 5, pp. 94–101, Sep.
2018.
2. F. Khomh, B. Adams, J. Cheng, M. Fokaefs, and G. Antoniol, “Software Engineering for Machine-Learning
Applications: The Road Ahead,” IEEE Softw., vol. 35, no. 5, pp. 81–84, Sep. 2018.
3. D. Sculley et al., “Hidden Technical Debt in Machine Learning Systems,” in Proceedings of the 28th
International Conference on Neural Information Processing Systems - Volume 2, Cambridge, MA, USA, 2015, pp.
2503–2511.
4. D. Sculley et al., “Machine Learning: The High Interest Credit Card of Technical Debt”. Google Research, 2014.
5. M. Ribeiro, K. Grolinger, and M. A. M. Capretz, “MLaaS: Machine Learning as a Service,” in 2015 IEEE 14th
International Conference on Machine Learning and Applications (ICMLA), Miami, FL, USA, 2015, pp. 896–902.

Weitere ähnliche Inhalte

Was ist angesagt?

Machine Learning Deep Learning AI and Data Science
Machine Learning Deep Learning AI and Data Science Machine Learning Deep Learning AI and Data Science
Machine Learning Deep Learning AI and Data Science Venkata Reddy Konasani
 
Harry Surden - Artificial Intelligence and Law Overview
Harry Surden - Artificial Intelligence and Law OverviewHarry Surden - Artificial Intelligence and Law Overview
Harry Surden - Artificial Intelligence and Law OverviewHarry Surden
 
Overview of Artificial Intelligence in Cybersecurity
Overview of Artificial Intelligence in CybersecurityOverview of Artificial Intelligence in Cybersecurity
Overview of Artificial Intelligence in CybersecurityOlivier Busolini
 
Credit card fraud detection using machine learning Algorithms
Credit card fraud detection using machine learning AlgorithmsCredit card fraud detection using machine learning Algorithms
Credit card fraud detection using machine learning Algorithmsankit panigrahy
 
The Top Trends in Artificial Intelligence
The Top Trends in Artificial IntelligenceThe Top Trends in Artificial Intelligence
The Top Trends in Artificial IntelligenceErling Hesselberg
 
Machine Learning and AI in Risk Management
Machine Learning and AI in Risk ManagementMachine Learning and AI in Risk Management
Machine Learning and AI in Risk ManagementQuantUniversity
 
Artificial intelligence for speech recognition
Artificial intelligence for speech recognitionArtificial intelligence for speech recognition
Artificial intelligence for speech recognitionsowmith chatlapally
 
Attendance system based on face recognition using python by Raihan Sikdar
Attendance system based on face recognition using python by Raihan SikdarAttendance system based on face recognition using python by Raihan Sikdar
Attendance system based on face recognition using python by Raihan Sikdarraihansikdar
 
Attendance Management System using Face Recognition
Attendance Management System using Face RecognitionAttendance Management System using Face Recognition
Attendance Management System using Face RecognitionNanditaDutta4
 
Ethical Considerations in the Design of Artificial Intelligence
Ethical Considerations in the Design of Artificial IntelligenceEthical Considerations in the Design of Artificial Intelligence
Ethical Considerations in the Design of Artificial IntelligenceJohn C. Havens
 
Attendance Using Facial Recognition
Attendance Using Facial RecognitionAttendance Using Facial Recognition
Attendance Using Facial RecognitionVikramaditya Tarai
 
"I don't trust AI": the role of explainability in responsible AI
"I don't trust AI": the role of explainability in responsible AI"I don't trust AI": the role of explainability in responsible AI
"I don't trust AI": the role of explainability in responsible AIErika Agostinelli
 

Was ist angesagt? (20)

Machine Learning Deep Learning AI and Data Science
Machine Learning Deep Learning AI and Data Science Machine Learning Deep Learning AI and Data Science
Machine Learning Deep Learning AI and Data Science
 
Harry Surden - Artificial Intelligence and Law Overview
Harry Surden - Artificial Intelligence and Law OverviewHarry Surden - Artificial Intelligence and Law Overview
Harry Surden - Artificial Intelligence and Law Overview
 
RPA ppt.pptx
RPA ppt.pptxRPA ppt.pptx
RPA ppt.pptx
 
Membrane computing
Membrane computingMembrane computing
Membrane computing
 
Web search vs ir
Web search vs irWeb search vs ir
Web search vs ir
 
Overview of Artificial Intelligence in Cybersecurity
Overview of Artificial Intelligence in CybersecurityOverview of Artificial Intelligence in Cybersecurity
Overview of Artificial Intelligence in Cybersecurity
 
Credit card fraud detection using machine learning Algorithms
Credit card fraud detection using machine learning AlgorithmsCredit card fraud detection using machine learning Algorithms
Credit card fraud detection using machine learning Algorithms
 
Computer vision
Computer visionComputer vision
Computer vision
 
The Top Trends in Artificial Intelligence
The Top Trends in Artificial IntelligenceThe Top Trends in Artificial Intelligence
The Top Trends in Artificial Intelligence
 
Machine Learning and AI in Risk Management
Machine Learning and AI in Risk ManagementMachine Learning and AI in Risk Management
Machine Learning and AI in Risk Management
 
Artificial intelligence for speech recognition
Artificial intelligence for speech recognitionArtificial intelligence for speech recognition
Artificial intelligence for speech recognition
 
Attendance system based on face recognition using python by Raihan Sikdar
Attendance system based on face recognition using python by Raihan SikdarAttendance system based on face recognition using python by Raihan Sikdar
Attendance system based on face recognition using python by Raihan Sikdar
 
Attendance Management System using Face Recognition
Attendance Management System using Face RecognitionAttendance Management System using Face Recognition
Attendance Management System using Face Recognition
 
Generative AI
Generative AIGenerative AI
Generative AI
 
What is MLOps
What is MLOpsWhat is MLOps
What is MLOps
 
Fraud and Risk in Big Data
Fraud and Risk in Big DataFraud and Risk in Big Data
Fraud and Risk in Big Data
 
Model bias in AI
Model bias in AIModel bias in AI
Model bias in AI
 
Ethical Considerations in the Design of Artificial Intelligence
Ethical Considerations in the Design of Artificial IntelligenceEthical Considerations in the Design of Artificial Intelligence
Ethical Considerations in the Design of Artificial Intelligence
 
Attendance Using Facial Recognition
Attendance Using Facial RecognitionAttendance Using Facial Recognition
Attendance Using Facial Recognition
 
"I don't trust AI": the role of explainability in responsible AI
"I don't trust AI": the role of explainability in responsible AI"I don't trust AI": the role of explainability in responsible AI
"I don't trust AI": the role of explainability in responsible AI
 

Ähnlich wie AI4SE: Challenges and opportunities in the integration of Systems Engineering and the AI/ML lifecycle

Challenges in the integration of Systems Engineering and the AI/ML model life...
Challenges in the integration of Systems Engineering and the AI/ML model life...Challenges in the integration of Systems Engineering and the AI/ML model life...
Challenges in the integration of Systems Engineering and the AI/ML model life...CARLOS III UNIVERSITY OF MADRID
 
2020 09-16-ai-engineering challanges
2020 09-16-ai-engineering challanges2020 09-16-ai-engineering challanges
2020 09-16-ai-engineering challangesIvica Crnkovic
 
Research Problem Presentation - Research in Supply Chain Digital Twins
Research Problem Presentation - Research in Supply Chain Digital TwinsResearch Problem Presentation - Research in Supply Chain Digital Twins
Research Problem Presentation - Research in Supply Chain Digital TwinsArwa Abougharib
 
AI for Software Engineering
AI for Software EngineeringAI for Software Engineering
AI for Software EngineeringMiroslaw Staron
 
Software Engineering Challenges in building AI-based complex systems
Software Engineering Challenges in building AI-based complex systemsSoftware Engineering Challenges in building AI-based complex systems
Software Engineering Challenges in building AI-based complex systemsIvica Crnkovic
 
Explainable AI with H2O Driverless AI's MLI module
Explainable AI with H2O Driverless AI's MLI moduleExplainable AI with H2O Driverless AI's MLI module
Explainable AI with H2O Driverless AI's MLI moduleMartin Dvorak
 
Finding the right digital tools for your internal communication strategy
Finding the right digital tools for your internal communication strategyFinding the right digital tools for your internal communication strategy
Finding the right digital tools for your internal communication strategyStephan Schillerwein
 
First Year Report, PhD presentation
First Year Report, PhD presentationFirst Year Report, PhD presentation
First Year Report, PhD presentationBang Xiang Yong
 
AI & ML in Cyber Security - Why Algorithms Are Dangerous
AI & ML in Cyber Security - Why Algorithms Are DangerousAI & ML in Cyber Security - Why Algorithms Are Dangerous
AI & ML in Cyber Security - Why Algorithms Are DangerousRaffael Marty
 
Mykola Mykytenko: MLOps: your way from nonsense to valuable effect (approache...
Mykola Mykytenko: MLOps: your way from nonsense to valuable effect (approache...Mykola Mykytenko: MLOps: your way from nonsense to valuable effect (approache...
Mykola Mykytenko: MLOps: your way from nonsense to valuable effect (approache...Lviv Startup Club
 
Data Science as a Service: Intersection of Cloud Computing and Data Science
Data Science as a Service: Intersection of Cloud Computing and Data ScienceData Science as a Service: Intersection of Cloud Computing and Data Science
Data Science as a Service: Intersection of Cloud Computing and Data SciencePouria Amirian
 
Data Science as a Service: Intersection of Cloud Computing and Data Science
Data Science as a Service: Intersection of Cloud Computing and Data ScienceData Science as a Service: Intersection of Cloud Computing and Data Science
Data Science as a Service: Intersection of Cloud Computing and Data SciencePouria Amirian
 
Machine Learning for Finance Master Class
Machine Learning for Finance Master Class Machine Learning for Finance Master Class
Machine Learning for Finance Master Class QuantUniversity
 
Model governance in the age of data science & AI
Model governance in the age of data science & AIModel governance in the age of data science & AI
Model governance in the age of data science & AIQuantUniversity
 
The Machine Learning Audit. MIS ITAC 2017 Keynote
The Machine Learning Audit. MIS ITAC 2017 KeynoteThe Machine Learning Audit. MIS ITAC 2017 Keynote
The Machine Learning Audit. MIS ITAC 2017 KeynoteAndrew Clark
 
230208 MLOps Getting from Good to Great.pptx
230208 MLOps Getting from Good to Great.pptx230208 MLOps Getting from Good to Great.pptx
230208 MLOps Getting from Good to Great.pptxArthur240715
 
Webinar: Machine Learning para Microcontroladores
Webinar: Machine Learning para MicrocontroladoresWebinar: Machine Learning para Microcontroladores
Webinar: Machine Learning para MicrocontroladoresEmbarcados
 

Ähnlich wie AI4SE: Challenges and opportunities in the integration of Systems Engineering and the AI/ML lifecycle (20)

Challenges in the integration of Systems Engineering and the AI/ML model life...
Challenges in the integration of Systems Engineering and the AI/ML model life...Challenges in the integration of Systems Engineering and the AI/ML model life...
Challenges in the integration of Systems Engineering and the AI/ML model life...
 
INCOSE IS 2019: AI and Systems Engineering
INCOSE IS 2019: AI and Systems EngineeringINCOSE IS 2019: AI and Systems Engineering
INCOSE IS 2019: AI and Systems Engineering
 
SESE 2021: Where Systems Engineering meets AI/ML
SESE 2021: Where Systems Engineering meets AI/MLSESE 2021: Where Systems Engineering meets AI/ML
SESE 2021: Where Systems Engineering meets AI/ML
 
2020 09-16-ai-engineering challanges
2020 09-16-ai-engineering challanges2020 09-16-ai-engineering challanges
2020 09-16-ai-engineering challanges
 
Research Problem Presentation - Research in Supply Chain Digital Twins
Research Problem Presentation - Research in Supply Chain Digital TwinsResearch Problem Presentation - Research in Supply Chain Digital Twins
Research Problem Presentation - Research in Supply Chain Digital Twins
 
AI for Software Engineering
AI for Software EngineeringAI for Software Engineering
AI for Software Engineering
 
Software Engineering Challenges in building AI-based complex systems
Software Engineering Challenges in building AI-based complex systemsSoftware Engineering Challenges in building AI-based complex systems
Software Engineering Challenges in building AI-based complex systems
 
IoT won't work without AI
IoT won't work without AIIoT won't work without AI
IoT won't work without AI
 
Explainable AI with H2O Driverless AI's MLI module
Explainable AI with H2O Driverless AI's MLI moduleExplainable AI with H2O Driverless AI's MLI module
Explainable AI with H2O Driverless AI's MLI module
 
Finding the right digital tools for your internal communication strategy
Finding the right digital tools for your internal communication strategyFinding the right digital tools for your internal communication strategy
Finding the right digital tools for your internal communication strategy
 
First Year Report, PhD presentation
First Year Report, PhD presentationFirst Year Report, PhD presentation
First Year Report, PhD presentation
 
AI & ML in Cyber Security - Why Algorithms Are Dangerous
AI & ML in Cyber Security - Why Algorithms Are DangerousAI & ML in Cyber Security - Why Algorithms Are Dangerous
AI & ML in Cyber Security - Why Algorithms Are Dangerous
 
Mykola Mykytenko: MLOps: your way from nonsense to valuable effect (approache...
Mykola Mykytenko: MLOps: your way from nonsense to valuable effect (approache...Mykola Mykytenko: MLOps: your way from nonsense to valuable effect (approache...
Mykola Mykytenko: MLOps: your way from nonsense to valuable effect (approache...
 
Data Science as a Service: Intersection of Cloud Computing and Data Science
Data Science as a Service: Intersection of Cloud Computing and Data ScienceData Science as a Service: Intersection of Cloud Computing and Data Science
Data Science as a Service: Intersection of Cloud Computing and Data Science
 
Data Science as a Service: Intersection of Cloud Computing and Data Science
Data Science as a Service: Intersection of Cloud Computing and Data ScienceData Science as a Service: Intersection of Cloud Computing and Data Science
Data Science as a Service: Intersection of Cloud Computing and Data Science
 
Machine Learning for Finance Master Class
Machine Learning for Finance Master Class Machine Learning for Finance Master Class
Machine Learning for Finance Master Class
 
Model governance in the age of data science & AI
Model governance in the age of data science & AIModel governance in the age of data science & AI
Model governance in the age of data science & AI
 
The Machine Learning Audit. MIS ITAC 2017 Keynote
The Machine Learning Audit. MIS ITAC 2017 KeynoteThe Machine Learning Audit. MIS ITAC 2017 Keynote
The Machine Learning Audit. MIS ITAC 2017 Keynote
 
230208 MLOps Getting from Good to Great.pptx
230208 MLOps Getting from Good to Great.pptx230208 MLOps Getting from Good to Great.pptx
230208 MLOps Getting from Good to Great.pptx
 
Webinar: Machine Learning para Microcontroladores
Webinar: Machine Learning para MicrocontroladoresWebinar: Machine Learning para Microcontroladores
Webinar: Machine Learning para Microcontroladores
 

Mehr von CARLOS III UNIVERSITY OF MADRID

Engineering 4.0: Digitization through task automation and reuse
Engineering 4.0:  Digitization through task automation and reuseEngineering 4.0:  Digitization through task automation and reuse
Engineering 4.0: Digitization through task automation and reuseCARLOS III UNIVERSITY OF MADRID
 
LOTAR-PDES: Engineering digitalization through task automation and reuse in t...
LOTAR-PDES: Engineering digitalization through task automation and reuse in t...LOTAR-PDES: Engineering digitalization through task automation and reuse in t...
LOTAR-PDES: Engineering digitalization through task automation and reuse in t...CARLOS III UNIVERSITY OF MADRID
 
Sailing the V: Engineering digitalization through task automation and reuse i...
Sailing the V: Engineering digitalization through task automation and reuse i...Sailing the V: Engineering digitalization through task automation and reuse i...
Sailing the V: Engineering digitalization through task automation and reuse i...CARLOS III UNIVERSITY OF MADRID
 
OSLC KM: Elevating the meaning of data and operations within the toolchain
OSLC KM: Elevating the meaning of data and operations within the toolchainOSLC KM: Elevating the meaning of data and operations within the toolchain
OSLC KM: Elevating the meaning of data and operations within the toolchainCARLOS III UNIVERSITY OF MADRID
 
OSLC KM (Knowledge Management): elevating the meaning of data and operations ...
OSLC KM (Knowledge Management): elevating the meaning of data and operations ...OSLC KM (Knowledge Management): elevating the meaning of data and operations ...
OSLC KM (Knowledge Management): elevating the meaning of data and operations ...CARLOS III UNIVERSITY OF MADRID
 
Systems and Software Architecture: an introduction to architectural modelling
Systems and Software Architecture: an introduction to architectural modellingSystems and Software Architecture: an introduction to architectural modelling
Systems and Software Architecture: an introduction to architectural modellingCARLOS III UNIVERSITY OF MADRID
 
Detection of fraud in financial blockchain-based transactions through big dat...
Detection of fraud in financial blockchain-based transactions through big dat...Detection of fraud in financial blockchain-based transactions through big dat...
Detection of fraud in financial blockchain-based transactions through big dat...CARLOS III UNIVERSITY OF MADRID
 
News headline generation with sentiment and patterns: A case study of sports ...
News headline generation with sentiment and patterns: A case study of sports ...News headline generation with sentiment and patterns: A case study of sports ...
News headline generation with sentiment and patterns: A case study of sports ...CARLOS III UNIVERSITY OF MADRID
 
EMOOCs-2017: Measuring the degree of innovation in higher education through M...
EMOOCs-2017: Measuring the degree of innovation in higher education through M...EMOOCs-2017: Measuring the degree of innovation in higher education through M...
EMOOCs-2017: Measuring the degree of innovation in higher education through M...CARLOS III UNIVERSITY OF MADRID
 

Mehr von CARLOS III UNIVERSITY OF MADRID (20)

Proyecto IVERES-UC3M
Proyecto IVERES-UC3MProyecto IVERES-UC3M
Proyecto IVERES-UC3M
 
RTVE: Sustainable Development Goal Radar
RTVE: Sustainable Development Goal  RadarRTVE: Sustainable Development Goal  Radar
RTVE: Sustainable Development Goal Radar
 
Engineering 4.0: Digitization through task automation and reuse
Engineering 4.0:  Digitization through task automation and reuseEngineering 4.0:  Digitization through task automation and reuse
Engineering 4.0: Digitization through task automation and reuse
 
LOTAR-PDES: Engineering digitalization through task automation and reuse in t...
LOTAR-PDES: Engineering digitalization through task automation and reuse in t...LOTAR-PDES: Engineering digitalization through task automation and reuse in t...
LOTAR-PDES: Engineering digitalization through task automation and reuse in t...
 
Sailing the V: Engineering digitalization through task automation and reuse i...
Sailing the V: Engineering digitalization through task automation and reuse i...Sailing the V: Engineering digitalization through task automation and reuse i...
Sailing the V: Engineering digitalization through task automation and reuse i...
 
Deep Learning Notes
Deep Learning NotesDeep Learning Notes
Deep Learning Notes
 
H2020-AHTOOLS Use Case 3 Functional Design
H2020-AHTOOLS Use Case 3 Functional DesignH2020-AHTOOLS Use Case 3 Functional Design
H2020-AHTOOLS Use Case 3 Functional Design
 
Blockchain en la Industria Musical
Blockchain en la Industria MusicalBlockchain en la Industria Musical
Blockchain en la Industria Musical
 
OSLC KM: Elevating the meaning of data and operations within the toolchain
OSLC KM: Elevating the meaning of data and operations within the toolchainOSLC KM: Elevating the meaning of data and operations within the toolchain
OSLC KM: Elevating the meaning of data and operations within the toolchain
 
Blockchain y sector asegurador
Blockchain y sector aseguradorBlockchain y sector asegurador
Blockchain y sector asegurador
 
OSLC KM (Knowledge Management): elevating the meaning of data and operations ...
OSLC KM (Knowledge Management): elevating the meaning of data and operations ...OSLC KM (Knowledge Management): elevating the meaning of data and operations ...
OSLC KM (Knowledge Management): elevating the meaning of data and operations ...
 
Systems and Software Architecture: an introduction to architectural modelling
Systems and Software Architecture: an introduction to architectural modellingSystems and Software Architecture: an introduction to architectural modelling
Systems and Software Architecture: an introduction to architectural modelling
 
Detection of fraud in financial blockchain-based transactions through big dat...
Detection of fraud in financial blockchain-based transactions through big dat...Detection of fraud in financial blockchain-based transactions through big dat...
Detection of fraud in financial blockchain-based transactions through big dat...
 
News headline generation with sentiment and patterns: A case study of sports ...
News headline generation with sentiment and patterns: A case study of sports ...News headline generation with sentiment and patterns: A case study of sports ...
News headline generation with sentiment and patterns: A case study of sports ...
 
Blockchain y la industria musical
Blockchain y la industria musicalBlockchain y la industria musical
Blockchain y la industria musical
 
Preparing your Big Data start-up pitch
Preparing your Big Data start-up pitchPreparing your Big Data start-up pitch
Preparing your Big Data start-up pitch
 
Internet of Things (IoT) in a nutshell
Internet of Things (IoT) in a nutshellInternet of Things (IoT) in a nutshell
Internet of Things (IoT) in a nutshell
 
Blockchain in a nutshell
Blockchain in a nutshellBlockchain in a nutshell
Blockchain in a nutshell
 
Proyecto SMART: Arquitectura para Big Data
Proyecto SMART: Arquitectura para Big DataProyecto SMART: Arquitectura para Big Data
Proyecto SMART: Arquitectura para Big Data
 
EMOOCs-2017: Measuring the degree of innovation in higher education through M...
EMOOCs-2017: Measuring the degree of innovation in higher education through M...EMOOCs-2017: Measuring the degree of innovation in higher education through M...
EMOOCs-2017: Measuring the degree of innovation in higher education through M...
 

Kürzlich hochgeladen

Test of Significance of Large Samples for Mean = µ.pptx
Test of Significance of Large Samples for Mean = µ.pptxTest of Significance of Large Samples for Mean = µ.pptx
Test of Significance of Large Samples for Mean = µ.pptxHome
 
Lifting Plan | Lifting Plan for Different Process Equipment | Gaurav Singh Ra...
Lifting Plan | Lifting Plan for Different Process Equipment | Gaurav Singh Ra...Lifting Plan | Lifting Plan for Different Process Equipment | Gaurav Singh Ra...
Lifting Plan | Lifting Plan for Different Process Equipment | Gaurav Singh Ra...Gaurav Singh Rajput
 
Gender Bias in Engineer, Honors 203 Project
Gender Bias in Engineer, Honors 203 ProjectGender Bias in Engineer, Honors 203 Project
Gender Bias in Engineer, Honors 203 Projectreemakb03
 
Clutches and brkesSelect any 3 position random motion out of real world and d...
Clutches and brkesSelect any 3 position random motion out of real world and d...Clutches and brkesSelect any 3 position random motion out of real world and d...
Clutches and brkesSelect any 3 position random motion out of real world and d...sahb78428
 
Graphics Primitives and CG Display Devices
Graphics Primitives and CG Display DevicesGraphics Primitives and CG Display Devices
Graphics Primitives and CG Display DevicesDIPIKA83
 
Strategies of Urban Morphologyfor Improving Outdoor Thermal Comfort and Susta...
Strategies of Urban Morphologyfor Improving Outdoor Thermal Comfort and Susta...Strategies of Urban Morphologyfor Improving Outdoor Thermal Comfort and Susta...
Strategies of Urban Morphologyfor Improving Outdoor Thermal Comfort and Susta...amrabdallah9
 
cloud computing notes for anna university syllabus
cloud computing notes for anna university syllabuscloud computing notes for anna university syllabus
cloud computing notes for anna university syllabusViolet Violet
 
A Seminar on Electric Vehicle Software Simulation
A Seminar on Electric Vehicle Software SimulationA Seminar on Electric Vehicle Software Simulation
A Seminar on Electric Vehicle Software SimulationMohsinKhanA
 
sdfsadopkjpiosufoiasdoifjasldkjfl a asldkjflaskdjflkjsdsdf
sdfsadopkjpiosufoiasdoifjasldkjfl a asldkjflaskdjflkjsdsdfsdfsadopkjpiosufoiasdoifjasldkjfl a asldkjflaskdjflkjsdsdf
sdfsadopkjpiosufoiasdoifjasldkjfl a asldkjflaskdjflkjsdsdfJulia Kaye
 
Vertical- Machining - Center - VMC -LMW-Machine-Tool-Division.pptx
Vertical- Machining - Center - VMC -LMW-Machine-Tool-Division.pptxVertical- Machining - Center - VMC -LMW-Machine-Tool-Division.pptx
Vertical- Machining - Center - VMC -LMW-Machine-Tool-Division.pptxLMW Machine Tool Division
 
Engineering Mechanics Chapter 5 Equilibrium of a Rigid Body
Engineering Mechanics  Chapter 5  Equilibrium of a Rigid BodyEngineering Mechanics  Chapter 5  Equilibrium of a Rigid Body
Engineering Mechanics Chapter 5 Equilibrium of a Rigid BodyAhmadHajasad2
 
Dev.bg DevOps March 2024 Monitoring & Logging
Dev.bg DevOps March 2024 Monitoring & LoggingDev.bg DevOps March 2024 Monitoring & Logging
Dev.bg DevOps March 2024 Monitoring & LoggingMarian Marinov
 
Nodal seismic construction requirements.pptx
Nodal seismic construction requirements.pptxNodal seismic construction requirements.pptx
Nodal seismic construction requirements.pptxwendy cai
 
nvidia AI-gtc 2024 partial slide deck.pptx
nvidia AI-gtc 2024 partial slide deck.pptxnvidia AI-gtc 2024 partial slide deck.pptx
nvidia AI-gtc 2024 partial slide deck.pptxjasonsedano2
 
دليل تجارب الاسفلت المختبرية - Asphalt Experiments Guide Laboratory
دليل تجارب الاسفلت المختبرية - Asphalt Experiments Guide Laboratoryدليل تجارب الاسفلت المختبرية - Asphalt Experiments Guide Laboratory
دليل تجارب الاسفلت المختبرية - Asphalt Experiments Guide LaboratoryBahzad5
 
Modelling Guide for Timber Structures - FPInnovations
Modelling Guide for Timber Structures - FPInnovationsModelling Guide for Timber Structures - FPInnovations
Modelling Guide for Timber Structures - FPInnovationsYusuf Yıldız
 
GENERAL CONDITIONS FOR CONTRACTS OF CIVIL ENGINEERING WORKS
GENERAL CONDITIONS  FOR  CONTRACTS OF CIVIL ENGINEERING WORKS GENERAL CONDITIONS  FOR  CONTRACTS OF CIVIL ENGINEERING WORKS
GENERAL CONDITIONS FOR CONTRACTS OF CIVIL ENGINEERING WORKS Bahzad5
 
Best-NO1 Best Rohani Amil In Lahore Kala Ilam In Lahore Kala Jadu Amil In Lah...
Best-NO1 Best Rohani Amil In Lahore Kala Ilam In Lahore Kala Jadu Amil In Lah...Best-NO1 Best Rohani Amil In Lahore Kala Ilam In Lahore Kala Jadu Amil In Lah...
Best-NO1 Best Rohani Amil In Lahore Kala Ilam In Lahore Kala Jadu Amil In Lah...Amil baba
 

Kürzlich hochgeladen (20)

Test of Significance of Large Samples for Mean = µ.pptx
Test of Significance of Large Samples for Mean = µ.pptxTest of Significance of Large Samples for Mean = µ.pptx
Test of Significance of Large Samples for Mean = µ.pptx
 
Lifting Plan | Lifting Plan for Different Process Equipment | Gaurav Singh Ra...
Lifting Plan | Lifting Plan for Different Process Equipment | Gaurav Singh Ra...Lifting Plan | Lifting Plan for Different Process Equipment | Gaurav Singh Ra...
Lifting Plan | Lifting Plan for Different Process Equipment | Gaurav Singh Ra...
 
Gender Bias in Engineer, Honors 203 Project
Gender Bias in Engineer, Honors 203 ProjectGender Bias in Engineer, Honors 203 Project
Gender Bias in Engineer, Honors 203 Project
 
Clutches and brkesSelect any 3 position random motion out of real world and d...
Clutches and brkesSelect any 3 position random motion out of real world and d...Clutches and brkesSelect any 3 position random motion out of real world and d...
Clutches and brkesSelect any 3 position random motion out of real world and d...
 
Graphics Primitives and CG Display Devices
Graphics Primitives and CG Display DevicesGraphics Primitives and CG Display Devices
Graphics Primitives and CG Display Devices
 
Strategies of Urban Morphologyfor Improving Outdoor Thermal Comfort and Susta...
Strategies of Urban Morphologyfor Improving Outdoor Thermal Comfort and Susta...Strategies of Urban Morphologyfor Improving Outdoor Thermal Comfort and Susta...
Strategies of Urban Morphologyfor Improving Outdoor Thermal Comfort and Susta...
 
cloud computing notes for anna university syllabus
cloud computing notes for anna university syllabuscloud computing notes for anna university syllabus
cloud computing notes for anna university syllabus
 
A Seminar on Electric Vehicle Software Simulation
A Seminar on Electric Vehicle Software SimulationA Seminar on Electric Vehicle Software Simulation
A Seminar on Electric Vehicle Software Simulation
 
sdfsadopkjpiosufoiasdoifjasldkjfl a asldkjflaskdjflkjsdsdf
sdfsadopkjpiosufoiasdoifjasldkjfl a asldkjflaskdjflkjsdsdfsdfsadopkjpiosufoiasdoifjasldkjfl a asldkjflaskdjflkjsdsdf
sdfsadopkjpiosufoiasdoifjasldkjfl a asldkjflaskdjflkjsdsdf
 
Vertical- Machining - Center - VMC -LMW-Machine-Tool-Division.pptx
Vertical- Machining - Center - VMC -LMW-Machine-Tool-Division.pptxVertical- Machining - Center - VMC -LMW-Machine-Tool-Division.pptx
Vertical- Machining - Center - VMC -LMW-Machine-Tool-Division.pptx
 
Engineering Mechanics Chapter 5 Equilibrium of a Rigid Body
Engineering Mechanics  Chapter 5  Equilibrium of a Rigid BodyEngineering Mechanics  Chapter 5  Equilibrium of a Rigid Body
Engineering Mechanics Chapter 5 Equilibrium of a Rigid Body
 
Dev.bg DevOps March 2024 Monitoring & Logging
Dev.bg DevOps March 2024 Monitoring & LoggingDev.bg DevOps March 2024 Monitoring & Logging
Dev.bg DevOps March 2024 Monitoring & Logging
 
Nodal seismic construction requirements.pptx
Nodal seismic construction requirements.pptxNodal seismic construction requirements.pptx
Nodal seismic construction requirements.pptx
 
nvidia AI-gtc 2024 partial slide deck.pptx
nvidia AI-gtc 2024 partial slide deck.pptxnvidia AI-gtc 2024 partial slide deck.pptx
nvidia AI-gtc 2024 partial slide deck.pptx
 
دليل تجارب الاسفلت المختبرية - Asphalt Experiments Guide Laboratory
دليل تجارب الاسفلت المختبرية - Asphalt Experiments Guide Laboratoryدليل تجارب الاسفلت المختبرية - Asphalt Experiments Guide Laboratory
دليل تجارب الاسفلت المختبرية - Asphalt Experiments Guide Laboratory
 
Modelling Guide for Timber Structures - FPInnovations
Modelling Guide for Timber Structures - FPInnovationsModelling Guide for Timber Structures - FPInnovations
Modelling Guide for Timber Structures - FPInnovations
 
GENERAL CONDITIONS FOR CONTRACTS OF CIVIL ENGINEERING WORKS
GENERAL CONDITIONS  FOR  CONTRACTS OF CIVIL ENGINEERING WORKS GENERAL CONDITIONS  FOR  CONTRACTS OF CIVIL ENGINEERING WORKS
GENERAL CONDITIONS FOR CONTRACTS OF CIVIL ENGINEERING WORKS
 
Best-NO1 Best Rohani Amil In Lahore Kala Ilam In Lahore Kala Jadu Amil In Lah...
Best-NO1 Best Rohani Amil In Lahore Kala Ilam In Lahore Kala Jadu Amil In Lah...Best-NO1 Best Rohani Amil In Lahore Kala Ilam In Lahore Kala Jadu Amil In Lah...
Best-NO1 Best Rohani Amil In Lahore Kala Ilam In Lahore Kala Jadu Amil In Lah...
 
Lecture 4 .pdf
Lecture 4                              .pdfLecture 4                              .pdf
Lecture 4 .pdf
 
計劃趕得上變化
計劃趕得上變化計劃趕得上變化
計劃趕得上變化
 

AI4SE: Challenges and opportunities in the integration of Systems Engineering and the AI/ML lifecycle

  • 1. Challenges and opportunities in the integration of the Systems Engineering process and the AI/ML Lifecycle Jose María Alvarez Rodríguez | Associate Professor | josemaria.alvarez@uc3m.es
  • 3. Digitalization Digital technologies are an “opportunity”. European companies 9/10 Adopted 2 of 9 the key digital technologies Technology adoption 35% Increase in their annual turnover being 24% BD and 5% AI Big Data Analytics & AI 46% E. Union, “Digital Transformation Scoreboard 2018 EU businesses go digital: Opportunities, outcomes and uptake,” COSME programme of the European Union, Luxembourg, 978-92-9202-353–9, 2018.
  • 4. Context Evolving and competitive market for Cyber-physical systems (CPS) Legislation Business needs and models Quality • Safety • Reliability • Cyber-security • Certification • Environment • Standardization • Ethics • … Engineering process , methods, technology & tooling AI/ML & Deep Learning
  • 7. Self-Driving Cars (object recognition) Source .https://towardsdatascience.com/how-do-self-driving-cars-see-13054aee2503 Source: https://www.theverge.com/2015/7/17/8985699/stanford-neural-networks-image- recognition-google-study
  • 9. An example of architecture Is it a Car? Is it a kind of a computer (software and hardware intensive)? Sensors/Actuators GPUs AI/ML models 5G Edge Computing … Source: https://www.aptiv.com/media/article/2018/01/07/the-autonomous-driving-platform-how-will-cars-actually-drive-themselves
  • 10. Critical factors Human factors Adoption of intelligent evolving systems Safety & Security Ethics Conceptual & technological From business needs/value to system capabilities
  • 11. Software & Systems Engineering AI/ML lifecycle Perspective
  • 12. Software Engineering 12 Standard notations Languages, tools, quality management, maturity models, design patterns, components, etc. Agile methods & DevOps Scrum, Kanban, BDD, TDD, etc. Software production automation Model-Driven Engineering/Architecture Software Product Lines etc. Initial Focus Methods and methodology to tackle the problem of software reliability
  • 13. The way through There is a huge body of knowledge and practice that allow us to acknowledge which ARE, and which ARE NOT, the software engineering practices that really represent timeless scientific and technological foundations for a proper software development process. (My own summary based on [1]) Prof. Barry Boehm• “A Conversation with Barry Boehm: Recollections from 50 Years of Software Engineering”. IEEE Software, vol. 35 (5). 2018 • “A View of 20th and 21st Century Software Engineering “
  • 14. Software & Systems Engineering process ISO 15288:2015ISO 12207:2017 Mats Berglund (Ericsson) http://www.ices.kth.se/upload/events/13/84404189f85d41a6a7d1cafd0db4ee80.pdf Engineering (and corporate) environment Source: http://beyondplm.com/2014/07/22/plm-implementations-nuts-and-bolts-of-data-silos/
  • 15. Artificial Intelligence ELIZA Medical Support Systems “We can solve everything” Turing Test, Conversational Systems & Expert Systems AI/ML is not cool! “We can not solve anything” AI/ML WINTER Theoretical foundations Algorithms & Techniques From Deep Blue to Alexa Lack of data, computational power, etc. (not anymore*) AI/ML Boost Any x / x ε {domain, problem, place, one, sector, etc.} AI/ML SPRING & Fear 01 02 03 04 15
  • 16. The way through “The new spring in AI is the most significant development in computing in my lifetime. Every month, there are stunning new applications and transformative new techniques. But such powerful tools also bring with them new questions and responsibilities” Sergey Brin
  • 17. INCOSE IS 2019 “AI for Systems Development”
  • 19. Data Science Lifecycle Source: https://docs.microsoft.com/en-us/azure/machine-learning/team-data-science-process/overview
  • 20. “My” Data Science Lifecycle Technique Persistence Optimization Validation Training Data Selection Acquisition Access Persistence Structure Enriching Cleaning Business Needs Applications (value) Exploitation Analysis Preliminary Analysis (stats / visual) Data Management Lifecycle DML AI/ML function Evolution
  • 21. Frameworks/Platforms/…→AI/ML as a Service 21 Source: https://www.ibm.com/blogs/research/2018/03/deep-learning-advances/ Source: https://www.morpheusdata.com/blog/2017-09-14-cloud-and-ai-a-work-in-progress-with-unlimited-upside Source: https://azure.microsoft.com/es-es/services/machine-learning-studio Source: https://algorithmia.com/ Source: https://cloud.google.com/automl/?hl=es-419
  • 22. Technical debt in AI/ML functions [2, 4] 22 Feedback loops Correction cascades Garbage features Source: https://towardsdatascience.com/technical-debt-in-machine-learning-8b0fae938657
  • 23. Software & Systems Engineering AI/ML lifecycle Harmonization
  • 24. Engineering (of→ for) AI/ML Current trend Challenges ahead Opportunities ahead No clear view Lack of expertise: from deterministic to evolving systems Engineering of evolving intelligent systems How to integrate the engineering process with the AI/ML lifecycle? Technology available Methods not ready Change resistance Room for improvement Cultural change and new mindset Evolving intelligent systems for engineering How to use intelligence to help engineering processes? 24 Engineering more intelligent complex products/services… …also requires more intelligent engineering methods and tools.
  • 25. Harmonization of the SE process and the AI/ML function Paying the “hidden” technical debt
  • 26. 10 major challenges & coffee evaluation How we do things now? Status Can we define a challenge? Concept What to do to tackle the challenge? Opportunity Is it really a challenge? Is there an opportunity? Debate 26 …and the necessary tea/coffee cups per hour
  • 27. • Catalogue of technology and tools. • Specify the DML stages. • Describe a DML using standards. • Orchestration of DML activities. • Access to functionalities as a service. Challenge #1: to describe the needs and capabilities of the resources which are part of the AI/ML model lifecycle Status Concept Opportunity Debate • AI/ML model lifecycle→several inter-related activities • Data management lifecycle (DML) not completely clear • Tangled environment of technology and tools • Lack of standardization • Description of needs and capabilities of the AI/ML model lifecycle • DML has been already managed in the context of Big Data applications and Data Science. 27
  • 28. • New set of subsystems: AI/ML based. • New set of metrics. • New set of patterns to specify requirements/models about AI/ML functions. • Adaptation of the CCC in the context AI/ML functions. • Blocks for non-deterministic functions. • Consistency over time (evolvability). • Human factors. Challenge #2: to integrate the AI/ML model lifecycle within the specification process of a system Status Concept Opportunity Debate • Stages of the AI/ML model lifecycle: data, training, test, optimization and operation. • Many models with different characteristics and applications. • Performance metrics: precision, recall, accuracy, etc. depending on the type of model. • Specification of requirements/models for AI/ML functions. • It mainly requires the extension of existing methods. 28
  • 29. DSL for requirements (patterns, metrics, etc.) 29 Patterns P1: The <AI_system> shall <action> <entity> with a <metric> of <value> <unit> under <weather_condition_type>. Vocabulary Techniques, metrics, types of problems to solve, etc. Requirements R1: “The entity recognition subsystem shall provide an explanation of at least 10 words about the recognized objects. R2: “The entity recognition subsystem shall recognize a person with an accuracy of 90% under light snow.” Quality metrics Consistency over time Consistency between techniques and metrics. Consistency between requirements and techniques.
  • 30. • Methods to connect a digital twin environment. • Methods to deriving testing environments. • Methods for re-verification, re- validation, re-certification, re- qualification. • Virtual and Physical • Continuous RE-V&V • Consistency over-time. • Simulation++++ (and its orchestration) Challenge #3: to integrate the AI/ML model lifecycle within the verification and validation (V&V) process of a system. Status Concept Opportunity Debate • AI/ML already used for testing (data, test case generation, execution). • Languages to specify (safety) test cases, e.g. Open Scenario (AVL). • Digital twin environment. • V&V of evolving systems. • How to make deterministic a non- deterministic system? • Feedback loop: • In-lab V&V vs physical V&V • Connection between both sides of the Vee model. • W model. • Legal framework. 30
  • 31. V&V IN THE LOOP! 31 https://www.mathworks.com/content/dam/mathworks/mathworks-dot-com/solutions/automotive/files/in-expo-2014/mil-sil-pil-a-new-paradigm-in-model-based-development.pdf
  • 32. Digital twin environment 32 Source: https://www.mdpi.com/2079-8954/7/1/7/pdf
  • 34. • Automatic generation of execution environments (DevOps approach). • Artificial “Intelligence as a Service” (AIaaS) • “Machine Learning as a Service” (MLaaS) [5] • Share models? (competitive advantage) Challenge #4: to integrate the technology stack of the AI/ML model lifecycle within the system execution environment Status Concept Opportunity Debate • Many technology providers • Include a new technology stack: • Hardware • Software • Communication system • Computation system • In the car industry, there are already strategic alliances between: • Car manufacturers (OEM) • Hardware providers • AI/ML providers • (Software) infrastructure providers • Research centers 34
  • 35. Technology and Car Companies 35 Source: https://www.bloomberg.com/graphics/2016-merging-tech-and-cars/
  • 36. • Proper management of baselines and versions • Round trip of data. • Change impact analysis acquires a new dimension. • Artefact synch (consistency + traceability). • See Challenge #3 • … Challenge #5: to integrate the AI/ML model lifecycle within the configuration and change management processes Status Concept Opportunity Debate • Many tools • Configuration more relevant than ever→more software and more data dependencies • Existing techniques and tools are ready to manage changes and configuration in a digital twin environment. 36
  • 37. • Methods to check the reusability factor of an AI/ML model. • Licensing and catalogue of datasets (market). • Licensing and catalogue of AI/ML models (e.g. like Modelica libraries) (market). • See Challenge #4. Services. Challenge #6: to ease the reuse of existing AI/ML models Status Concept Opportunity Debate • Services • Reuse of existing of AI/ML models can save time and costs. • Increase reliability. • The Big Data and Data Science communities have already passed over here. 37
  • 38. • Update of methods and standards. • Update of architectural frameworks. • Sustainability. • Bring new professionals to a SE team. Challenge #7: to standardize the integration of AI/ML model lifecycle within the engineering process Status Concept Opportunity Debate • A new knowledge area comes to the Systems Engineering discipline. • Avoid ad-hoc, non- controlled or vendor-lock-in integrations. • It can be considered as another part of the software development process. 38
  • 39. Framework for Highly Automated Vehicle Safety Validation 39 Source: https://users.ece.cmu.edu/~koopman/pubs/koopman18_av_safety_validation.pdf “First Comprehensive Autonomous Product Safety Standard (UL 4600)” Source: https://edge-case-research.com/
  • 40. • New courses and training methods. • New positions: AI/ML explainer, AI/ML engineer, etc. Challenge #8: to educate a new wave of professionals to deal with AI/ML applying engineering methods Status Concept Opportunity Debate • A new knowledge area comes to the Systems Engineering discipline. • Equip people with the required skills to understand AI/ML functions. • These are already covered as in any other discipline. 40
  • 41. New roles and positions in AI 41 Source: http://ilp.mit.edu/media/news_articles/smr/2017/58416.pdf
  • 42. • Study strategies for optimization of AI/ML functions at all levels (hardware, software, communications, storage). • Reuse as a primitive. Challenge #9: to fulfill environmental requirements Status Concept Opportunity Debate • A new knowledge area comes to the Systems Engineering discipline. • Ensure the environment efficiency of the new systems (more than ever). • It is something that is already in the roadmap of any engineering product. 42
  • 43. • Certification of ethicality. Challenge #10: to ensure the construction of ethical machines Status Concept Opportunity Debate • MIT Moral Machine • Ethical guidelines of AI Systems from the EC. • Ethics, a necessary non- functional requirement. • How to check ethics? 43
  • 44. Summary Data Management Lifecycle Specification of AI/ML functions Continuous V&V of evolving sys. Technology stack integration Skills, Training & Education Standards update Reuse of AI/ML models Configuration and change mgt. Environmental footprint Ethics Legal framework … 01 02 03 04 08 07 06 05 09 10 11 12 44
  • 46. Key points and next steps 46 Research on AI/ML and Systems Engineering is a key/active area. Cyber-physical systems must be safer, cost- effective and environmentally friendly and…deliver on time (reacting to market needs). Engineering processes must integrate a new discipline and adapt engineering methods to support the new technical capabilities. Engineering of intelligent and evolving systems also requires “intelligent” tools. 1 2 3 4 Ethics is IMPORTANT, Safety a MUST.5
  • 47. Do NOT be a “technological” slave Build intelligent systems… • Integrate the background of AI/ML experts as a new knowledge area. • Understand the context of an evolving system and its implications in the system lifecycle. …with intelligence • Make use of existing and new tools (do not be afraid of change). • Build a skilled “intelligent” team.
  • 48. FUSE WG INCOSEFollow up→SENSEI project Systems ENgineering methodS for Evolving Intelligent systems
  • 49. References 1. C. Ebert, “50 Years of Software Engineering: Progress and Perils,” IEEE Softw., vol. 35, no. 5, pp. 94–101, Sep. 2018. 2. F. Khomh, B. Adams, J. Cheng, M. Fokaefs, and G. Antoniol, “Software Engineering for Machine-Learning Applications: The Road Ahead,” IEEE Softw., vol. 35, no. 5, pp. 81–84, Sep. 2018. 3. D. Sculley et al., “Hidden Technical Debt in Machine Learning Systems,” in Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2, Cambridge, MA, USA, 2015, pp. 2503–2511. 4. D. Sculley et al., “Machine Learning: The High Interest Credit Card of Technical Debt”. Google Research, 2014. 5. M. Ribeiro, K. Grolinger, and M. A. M. Capretz, “MLaaS: Machine Learning as a Service,” in 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA), Miami, FL, USA, 2015, pp. 896–902.