第86回 Machine Learning 15minutes! Hybrid (2024/02/24) https://machine-learning15minutes.connpass.com/event/307875/ LLMOps with Azure Machine Learning prompt flow (Machine Learning 15minutes! Hybrid #86) https://satonaoki.wordpress.com/2024/02/24/llmops-azure-machine-learning-prompt-flow-ml15min/

1. LLMOps with
Azure Machine Learning
prompt flow
SATO Naoki (Neo)
Senior Software Engineer, Microsoft

2. Access to thousands of
LLMs from OpenAI,
Meta, Hugging Face
Azure Machine Learning for Generative AI
Prompt engineering/
evaluation
Built-in safety and
responsible AI
Continuous monitoring
for LLMs
Purpose-built AI
infrastructure

3. The paradigm shift (MLOps vs LLMOps)
Traditional MLOps LLMOps
Target audiences
Assets to share
Metrics/evaluations
ML models
ML Engineers
Data Scientists
ML Engineers
App developers
Model, data,
environments, features
LLM, agents, plugins,
prompts, chains, APIs
Accuracy
Accuracy, fairness,
groundedness, relevance,
coherence
Build from scratch Pre-built, fine-tune

4. Operationalize LLM
app development
with prompt flow
LLMOps is a complex process.
Customers want:
• Private data access and controls
• Prompt engineering
• CI/CD
• Iterative experimentation
• Versioning and reproducibility
• Deployment and optimization
• Safe and Responsible AI
Design and development
Develop flow based on prompt
to extend the capability
Debug, run, and evaluate
flow with small data
Modify flow (prompts and
tools etc.)
No If satisfied
Yes
Evaluation and refinement
No
Evaluate flow against large
dataset with different metrics
(quality, relevance, safety, etc.)
If satisfied
Yes
Optimization and production
Optimize flow
Deploy and
monitor flow
Get end user
feedback

5. Streamline prompt engineering projects
Azure Machine Learning
prompt flow
Customer Benefits
• Create AI workflows that connect various language models,
APIs, and data sources to ground LLMs on your data.
• One platform to design, construct, tune, evaluate, test, and
deploy LLM workflows
• Evaluate the quality of workflows with rich set of pre-built
metrics and safety system.
• Easy prompt tuning, comparison of prompt variants, and
version-control.
Documentation: https://aka.ms/prompt_flow

6. Studio UI

7. Azure Machine Learning prompt flow (1/7)
Capabilities Overview
• Develop workflows
• Develop flows that connect to
various language models, external data sources,
tools, and custom code
• Test and evaluate
• Test flows with large datasets in parallel
• Evaluate the AI quality of the workflows with
metrics like performance, groundedness, and
accuracy
• Prompt tuning
• Easily tune prompts​ with variants and versions
• Compare and deploy
• Visually compare across experiments
• One-click deploy to a managed endpoint for
rapid integration

8. Azure Machine Learning prompt flow (2/7)
Prompt flow authoring
Develop your LLM flow from scratch
• Construct a flow using pre-built tools
• Support custom code
• Clone flows from samples
• Track run history

9. Azure Machine Learning prompt flow (3/7)
Connections
Manage APIs and external data sources
• Seamless integration with pre-built LLMs like
Azure OpenAI Service
• Built-in safety system with Azure AI Content
Safety
• Effectively manage credentials or secrets for
APIs
• Create your own connections in Python tools

10. Azure Machine Learning prompt flow (4/7)
Variants
• Create dynamic prompts using external
data and few shot samples
• Edit your complex prompts in full screen
• Quickly tune prompt and LLM
configuration with variants

11. Azure Machine Learning prompt flow (5/7)
Evaluation
• Evaluate flow performance with your own
data
• Use pre-built evaluation flows
• Build your own custom evaluation flows
Tune Variant 0
Tune Variant 1
Tune Variant 2
Flow variants
Evaluation
Bulk Test

12. Azure Machine Learning prompt flow (6/7)
Evaluation
• Compare multiple variants or runs to pick
best flow
• Add new evaluations to a finished run
• Ensure accuracy by scaling the size of data
in evaluation
Tune Variant 0
Tune Variant 1
Tune Variant 2
Flow variants
Evaluation
Bulk Test

13. Azure Machine Learning prompt flow (7/7)
Deploy
• Seamless transition from development to
production with AzureML’s managed online
endpoints
Production
Tune Variant 0
Tune Variant 1
Tune Variant 2
Flow variants
Test App

14. What is prompt flow code experience ？
Use code to define flow
File based flow, organized in a well-defined folder structure​
Support CLI/SDK​
Smooth transition between cloud and local
Download flow to local, import flow to cloud​
Develop, test, debug, deploy on local ​
Submit run from local to cloud​
From local deploy to cloud​
Manage runs/evaluation in cloud
Integrate with your CI/CD automation
SDK/CLI to init, execute, evaluate, visualize flow and metrics
VS Code Extension
Flow editor​
Local connection management​
Run history​
Collaboration
or share cross
workspace
Submit flow
runs to cloud
from your repo
(anywhere)

15. Demo - Azure Machine Learning prompt flow
1. Upload PDF files and create a vector search index in Azure AI Search
2. Create a new chat flow in prompt flow
3. Configure Azure OpenAI Service (LLM) and Azure AI Search (vector search)
4. Run the chat flow
5. Evaluate the chat flow
6. Deploy the chat flow as a REST API

37. Learn More
• What is Azure Machine Learning prompt flow - Azure Machine Learning |
Microsoft Learn
• Prompt flow — Prompt flow documentation (microsoft.github.io)