Building and deploying LLM applications with Apache Airflow

Kaxil Naik
Kaxil NaikDirector of Airflow Engineering team @ Astronomer um Astronomer
Building and
deploying LLM
applications with
Apache Airflow
Kaxil Naik
Apache Airflow Committer & PMC Member
Director of Eng @ Astronomer
Julian LaNeve
Senior Product Manager @
Astronomer
Why Airflow should be at the centre of LLMOps?
Real Use-case & reference architecture
Next Steps: Community collaboration
Agenda
A powerful new class of large
language models is making it
possible for machines to
write, code, draw, and create
with credible and sometimes
superhuman results.
Generative AI:
A Creative New World
Normally, for ML, you need to…
Ingest Data Train Model Prediction
…but now you can
You hit a pre-trained model
instead of your own model
…but now you can:
Ingest Data Train Model Prediction
Less Data
Building and deploying LLM applications with Apache Airflow
■ Ingestion from several sources
■ Day 2 operations on data pipelines
■ Data preparation
■ Data privacy
■ Data freshness
■ Model deployment & monitoring
■ Scaling Models
■ Experimentation & fine-tuning
■ Feedback Loops
Going from “Idea to Production” with LLM Apps involves
solving a lot of data engineering problems:
Source: https://python.langchain.com/docs/use_cases/question_answering/
Typical Architecture for Q&A use-case using LLM
Legacy Data Store
Retrieval Output
Storage
Splitting
Document Loading
Vectorstore
Database
PDFs
URLs
LLM <Answer>
Prompt
Splits
Relevant
Splits
Query
<Question>
Python Native
The language of data scientists and
ML engineers.
Pluggable Compute
GPUs, Kubernetes, EC2, VMs etc.
Common Interface
Between Data Engineering, Data
Science, ML Engineering and
Operations.
Data Agnostic
But data aware.
Extensible
Standardize custom operators and
templates for common DS tasks
across the organization.
Monitoring & Alerting
Built in features for logging,
monitoring and alerting to external
systems.
Ingestion
Extract and load data into
vectordbs and other destinations
Day 2 Ops
Handle retries, dependencies, and
all other day 2 ops associated with
data pipelines
Airflow is a Natural Fit…
Document Parsing
Decorator and pythonic interfaces
for standard LLM tools
Let’s Talk About a
Real Use Case
Problem Statement:
We have customers, employees, and community members
that ask questions about our product with answers that
exist across several sources of documentation.
How do we provide an easy interface for folks to
get their questions answered without adding
further strain to the team?
Building and deploying LLM applications with Apache Airflow
■ Airflow gives a framework to load data
from APIs & other sources into LangChain
■ LangChain helps pre-process and split
documents into smaller chunks
depending on content type
■ After content is split into chunks, each
chunk is embedded into vectors (semantic
representations)
■ Those vectors are written to Weaviate for
later retrieval
Data Ingestion, Processing, and Embedding
Embed chunks Write to Weaviate
Pre-process and split into
chunks
🦜🔗 LangChain
Docs (.md)
files
Slack
Messages
GitHub
issues
Docs (.md)
files
Users can interact with UI or
Slack Bot; they both use the
same API
■ Original prompt gets reworded 3x using gpt-3.5-turbo
■ Answer is generated by combining docs from each prompt
and making a gpt-4 call
■ State is stored in Firestore and prompt tracing is done through
LangSmith
🦜🔗LangChain
User Asks
a Question
Web App
Slack Bot
Original Prompt Rewording 2
Rewording 1
Rewording 3
Reword to get more
related documents
Vector DB search
with prompts
Combine docs
and make final
LLM call to
answer
🦜
🔗
Prompt Orchestration and Answering
■ Airflow DAGs process feedback async to evaluate answers on helpfulness,, relevance,
and publicness
■ If answer is good, it gets stored in Weaviate and can be used as a source for future
questions
■ UI also shows the most recent good prompts on the homepage
When a user submits feedback, it
gets stored in Firestore and
LangSmith for later use
User Rates
Answer
🦜
🔗
Fetch new runs: input,
output, and user feedback
Classify Q&A according
to helpfulness,
relevance, and public
🦜🔗 LangChain
If good answer, write to
vector DB to use in future
answers
If good answer, mark as
good to show on Ask Astro
homepage
On schedule
LLM & Product Feedback Loops
Running this in production meant:
■ Experimenting with different sources of data to ingest
■ Running the pipelines on a schedule and ad-hoc
■ Running the same workloads with variable chunking
strategies
■ Needing to retry tasks due to finicky python libraries and
unreliable external services
■ Giving different parts of the workload variable compute
■ Creating standard interfaces to interact with external
systems
■ Experimenting with different sources of data to ingest
■ Running the pipelines on a schedule and ad-hoc
■ Running the same workloads with variable chunking
strategies
■ Needing to retry tasks due to finicky python libraries and
unreliable external services
■ Giving different parts of the workload variable compute
■ Creating standard interfaces to interact with external
systems
Running this in production meant:
Which is
what
Airflow’s
great at!
ask.astronomer.io
github.com/astronomer/ask-astro
a16z’s Emerging LLM App Stack
Orchestration
(Python/DIY,
LangChain,
LlamaIndex,
ChatGPT)
APIs/Plugins
(Serp, Wolfram,
Zapier, etc.)
App Hosting
(Vercel, Steamship,
Streamlit, Modal)
Query
Output
Prompt
Few-shot
examples
Contextual
data
Playground
(OpenAI, nat.dev,
Humanloop)
Data Pipelines
(Databricks, Airflow,
Unstructured, etc.)
Embedding Model
(OpenAI, Cohere,
Hugging Face)
Vector Database
(Pinecone, Weaviate,
Chroma, pgvector)
LLM Cache
(Redis, SQLite,
GPTCache)
Logging/LLMops
(Weights & Biases, MLflow,
PromptLayer, Helicone)
Validation
(Guardrails, Rebuff,
Guidance, LMQL)
Proprietary API
(OpenAI, Anthropic)
Open API
(Hugging Face, Replicate)
Opinionated Cloud
(Databricks, Anyscale,
Mosaic, Modal, Runpod)
Cloud Provider
(AWS, GCP, Azure,
Coreweave)
LLM APIs and Hosting
Gray boxes show key components of the stack, with leading tools /
systems listed. Arrows show the flow of data through the stack.
Contextual data provided by app developers to condition
LLM outputs
Prompts and few-shot examples that are sent to the LLM
Queries submitted by users
Output returned to users
Legend
AskAstro has a few parts of this…
Orchestration
(Python/DIY,
LangChain,
LlamaIndex,
ChatGPT)
APIs/Plugins
(Serp, Wolfram,
Zapier, etc.)
App Hosting
(Vercel, Steamship,
Streamlit, Modal)
Query
Output
Prompt
Few-shot
examples
Contextual
data
Playground
(OpenAI, nat.dev,
Humanloop)
Data Pipelines
(Databricks, Airflow,
Unstructured, etc.)
Embedding Model
(OpenAI, Cohere,
Hugging Face)
Vector Database
(Pinecone, Weaviate,
Chroma, pgvector)
LLM Cache
(Redis, SQLite,
GPTCache)
Logging/LLMops
(Weights & Biases, MLflow,
PromptLayer, Helicone)
Validation
(Guardrails, Rebuff,
Guidance, LMQL)
Proprietary API
(OpenAI, Anthropic)
Open API
(Hugging Face, Replicate)
Opinionated Cloud
(Databricks, Anyscale,
Mosaic, Modal, Runpod)
Cloud Provider
(AWS, GCP, Azure,
Coreweave)
LLM APIs and Hosting
Gray boxes show key components of the stack, with leading tools /
systems listed. Arrows show the flow of data through the stack.
Contextual data provided by app developers to condition
LLM outputs
Prompts and few-shot examples that are sent to the LLM
Queries submitted by users
Output returned to users
Legend
Airflow is
foundational
to best
practices for
all of this.
Data Governance
■ How do you account for private data?
■ How do you provide transparency into data lineage?
Fine Tuning
■ Does it improve results?
■ How much does it cost?
Feedback Loops
■ Semantic cache for correct responses
■ Ranking sources based on accuracy and ranking accordingly
■ Prompt clustering – what are people asking?
…but there’s even more to consider.
Michael Gregory
Philippe Gagnon
Thanks to the AskAstro Team:
Community Collaboration
Providers Interfaces
Patterns and
Use Cases
What are all the providers the ecosystem needs?
pgvector
What’s the
interface that
feels right for
LLMOps?
What’s the
interface that
feels right for
LLMOps?
Patterns
What are the
best practices
for building
pipelines for
LLM Apps?
■ Do you use one task to ingest and write?
■ Can you use dynamic task mapping to break it out?
■ Do you write to disk?
■ Can you store embedding values in XCOMs?
■ How do you reconcile Airflow orchestration with
prompt orchestration?
Let’s do this all in the open source!
1 von 29

Recomendados

AI and ML Series - Introduction to Generative AI and LLMs - Session 1 von
AI and ML Series - Introduction to Generative AI and LLMs - Session 1AI and ML Series - Introduction to Generative AI and LLMs - Session 1
AI and ML Series - Introduction to Generative AI and LLMs - Session 1DianaGray10
1.2K views38 Folien
Generative-AI-in-enterprise-20230615.pdf von
Generative-AI-in-enterprise-20230615.pdfGenerative-AI-in-enterprise-20230615.pdf
Generative-AI-in-enterprise-20230615.pdfLiming Zhu
827 views7 Folien
Build an LLM-powered application using LangChain.pdf von
Build an LLM-powered application using LangChain.pdfBuild an LLM-powered application using LangChain.pdf
Build an LLM-powered application using LangChain.pdfStephenAmell4
939 views22 Folien
Explainable AI von
Explainable AIExplainable AI
Explainable AIDinesh V
3.8K views19 Folien
Responsible Generative AI von
Responsible Generative AIResponsible Generative AI
Responsible Generative AICMassociates
235 views12 Folien
MLOps by Sasha Rosenbaum von
MLOps by Sasha RosenbaumMLOps by Sasha Rosenbaum
MLOps by Sasha RosenbaumSasha Rosenbaum
2.6K views51 Folien

Más contenido relacionado

Was ist angesagt?

Large Language Models - Chat AI.pdf von
Large Language Models - Chat AI.pdfLarge Language Models - Chat AI.pdf
Large Language Models - Chat AI.pdfDavid Rostcheck
685 views19 Folien
MLOps Using MLflow von
MLOps Using MLflowMLOps Using MLflow
MLOps Using MLflowDatabricks
2K views26 Folien
AI and ML Series - Leveraging Generative AI and LLMs Using the UiPath Platfor... von
AI and ML Series - Leveraging Generative AI and LLMs Using the UiPath Platfor...AI and ML Series - Leveraging Generative AI and LLMs Using the UiPath Platfor...
AI and ML Series - Leveraging Generative AI and LLMs Using the UiPath Platfor...DianaGray10
307 views38 Folien
Regulating Generative AI - LLMOps pipelines with Transparency von
Regulating Generative AI - LLMOps pipelines with TransparencyRegulating Generative AI - LLMOps pipelines with Transparency
Regulating Generative AI - LLMOps pipelines with TransparencyDebmalya Biswas
230 views16 Folien
From Data Science to MLOps von
From Data Science to MLOpsFrom Data Science to MLOps
From Data Science to MLOpsCarl W. Handlin
445 views29 Folien
Using MLOps to Bring ML to Production/The Promise of MLOps von
Using MLOps to Bring ML to Production/The Promise of MLOpsUsing MLOps to Bring ML to Production/The Promise of MLOps
Using MLOps to Bring ML to Production/The Promise of MLOpsWeaveworks
5.4K views89 Folien

Was ist angesagt?(20)

Large Language Models - Chat AI.pdf von David Rostcheck
Large Language Models - Chat AI.pdfLarge Language Models - Chat AI.pdf
Large Language Models - Chat AI.pdf
David Rostcheck685 views
MLOps Using MLflow von Databricks
MLOps Using MLflowMLOps Using MLflow
MLOps Using MLflow
Databricks2K views
AI and ML Series - Leveraging Generative AI and LLMs Using the UiPath Platfor... von DianaGray10
AI and ML Series - Leveraging Generative AI and LLMs Using the UiPath Platfor...AI and ML Series - Leveraging Generative AI and LLMs Using the UiPath Platfor...
AI and ML Series - Leveraging Generative AI and LLMs Using the UiPath Platfor...
DianaGray10307 views
Regulating Generative AI - LLMOps pipelines with Transparency von Debmalya Biswas
Regulating Generative AI - LLMOps pipelines with TransparencyRegulating Generative AI - LLMOps pipelines with Transparency
Regulating Generative AI - LLMOps pipelines with Transparency
Debmalya Biswas230 views
Using MLOps to Bring ML to Production/The Promise of MLOps von Weaveworks
Using MLOps to Bring ML to Production/The Promise of MLOpsUsing MLOps to Bring ML to Production/The Promise of MLOps
Using MLOps to Bring ML to Production/The Promise of MLOps
Weaveworks5.4K views
LanGCHAIN Framework von Keymate.AI
LanGCHAIN FrameworkLanGCHAIN Framework
LanGCHAIN Framework
Keymate.AI1.4K views
MLOps Virtual Event: Automating ML at Scale von Databricks
MLOps Virtual Event: Automating ML at ScaleMLOps Virtual Event: Automating ML at Scale
MLOps Virtual Event: Automating ML at Scale
Databricks684 views
Using the power of Generative AI at scale von Maxim Salnikov
Using the power of Generative AI at scaleUsing the power of Generative AI at scale
Using the power of Generative AI at scale
Maxim Salnikov910 views
Exploring Opportunities in the Generative AI Value Chain.pdf von Dung Hoang
Exploring Opportunities in the Generative AI Value Chain.pdfExploring Opportunities in the Generative AI Value Chain.pdf
Exploring Opportunities in the Generative AI Value Chain.pdf
Dung Hoang267 views
Generative AI for the rest of us von Massimo Ferre'
Generative AI for the rest of usGenerative AI for the rest of us
Generative AI for the rest of us
Massimo Ferre'167 views
MLOps and Data Quality: Deploying Reliable ML Models in Production von Provectus
MLOps and Data Quality: Deploying Reliable ML Models in ProductionMLOps and Data Quality: Deploying Reliable ML Models in Production
MLOps and Data Quality: Deploying Reliable ML Models in Production
Provectus211 views
Episode 2: The LLM / GPT / AI Prompt / Data Engineer Roadmap von Anant Corporation
Episode 2: The LLM / GPT / AI Prompt / Data Engineer RoadmapEpisode 2: The LLM / GPT / AI Prompt / Data Engineer Roadmap
Episode 2: The LLM / GPT / AI Prompt / Data Engineer Roadmap
Anant Corporation655 views
Suresh Poopandi_Generative AI On AWS-MidWestCommunityDay-Final.pdf von AWS Chicago
Suresh Poopandi_Generative AI On AWS-MidWestCommunityDay-Final.pdfSuresh Poopandi_Generative AI On AWS-MidWestCommunityDay-Final.pdf
Suresh Poopandi_Generative AI On AWS-MidWestCommunityDay-Final.pdf
AWS Chicago718 views
Large Language Models, No-Code, and Responsible AI - Trends in Applied NLP in... von David Talby
Large Language Models, No-Code, and Responsible AI - Trends in Applied NLP in...Large Language Models, No-Code, and Responsible AI - Trends in Applied NLP in...
Large Language Models, No-Code, and Responsible AI - Trends in Applied NLP in...
David Talby659 views
Microsoft + OpenAI: Recent Updates (Machine Learning 15minutes! Broadcast #74) von Naoki (Neo) SATO
Microsoft + OpenAI: Recent Updates (Machine Learning 15minutes! Broadcast #74)Microsoft + OpenAI: Recent Updates (Machine Learning 15minutes! Broadcast #74)
Microsoft + OpenAI: Recent Updates (Machine Learning 15minutes! Broadcast #74)
Naoki (Neo) SATO1.2K views

Similar a Building and deploying LLM applications with Apache Airflow

Boulder/Denver BigData: Cluster Computing with Apache Mesos and Cascading von
Boulder/Denver BigData: Cluster Computing with Apache Mesos and CascadingBoulder/Denver BigData: Cluster Computing with Apache Mesos and Cascading
Boulder/Denver BigData: Cluster Computing with Apache Mesos and CascadingPaco Nathan
4.9K views87 Folien
Technology Stack Discussion von
Technology Stack DiscussionTechnology Stack Discussion
Technology Stack DiscussionZaiyang Li
586 views25 Folien
Why apache Flink is the 4G of Big Data Analytics Frameworks von
Why apache Flink is the 4G of Big Data Analytics FrameworksWhy apache Flink is the 4G of Big Data Analytics Frameworks
Why apache Flink is the 4G of Big Data Analytics FrameworksSlim Baltagi
30.7K views118 Folien
System design for Web Application von
System design for Web ApplicationSystem design for Web Application
System design for Web ApplicationMichael Choi
2.5K views5 Folien
Integrating Splunk into your Spring Applications von
Integrating Splunk into your Spring ApplicationsIntegrating Splunk into your Spring Applications
Integrating Splunk into your Spring ApplicationsDamien Dallimore
11.2K views68 Folien
MLflow: Infrastructure for a Complete Machine Learning Life Cycle with Mani ... von
 MLflow: Infrastructure for a Complete Machine Learning Life Cycle with Mani ... MLflow: Infrastructure for a Complete Machine Learning Life Cycle with Mani ...
MLflow: Infrastructure for a Complete Machine Learning Life Cycle with Mani ...Databricks
1.7K views41 Folien

Similar a Building and deploying LLM applications with Apache Airflow(20)

Boulder/Denver BigData: Cluster Computing with Apache Mesos and Cascading von Paco Nathan
Boulder/Denver BigData: Cluster Computing with Apache Mesos and CascadingBoulder/Denver BigData: Cluster Computing with Apache Mesos and Cascading
Boulder/Denver BigData: Cluster Computing with Apache Mesos and Cascading
Paco Nathan4.9K views
Technology Stack Discussion von Zaiyang Li
Technology Stack DiscussionTechnology Stack Discussion
Technology Stack Discussion
Zaiyang Li586 views
Why apache Flink is the 4G of Big Data Analytics Frameworks von Slim Baltagi
Why apache Flink is the 4G of Big Data Analytics FrameworksWhy apache Flink is the 4G of Big Data Analytics Frameworks
Why apache Flink is the 4G of Big Data Analytics Frameworks
Slim Baltagi30.7K views
System design for Web Application von Michael Choi
System design for Web ApplicationSystem design for Web Application
System design for Web Application
Michael Choi2.5K views
Integrating Splunk into your Spring Applications von Damien Dallimore
Integrating Splunk into your Spring ApplicationsIntegrating Splunk into your Spring Applications
Integrating Splunk into your Spring Applications
Damien Dallimore11.2K views
MLflow: Infrastructure for a Complete Machine Learning Life Cycle with Mani ... von Databricks
 MLflow: Infrastructure for a Complete Machine Learning Life Cycle with Mani ... MLflow: Infrastructure for a Complete Machine Learning Life Cycle with Mani ...
MLflow: Infrastructure for a Complete Machine Learning Life Cycle with Mani ...
Databricks1.7K views
Overview of VS2010 and .NET 4.0 von Bruce Johnson
Overview of VS2010 and .NET 4.0Overview of VS2010 and .NET 4.0
Overview of VS2010 and .NET 4.0
Bruce Johnson879 views
.NET per la Data Science e oltre von Marco Parenzan
.NET per la Data Science e oltre.NET per la Data Science e oltre
.NET per la Data Science e oltre
Marco Parenzan180 views
Machine learning model to production von Georg Heiler
Machine learning model to productionMachine learning model to production
Machine learning model to production
Georg Heiler4.4K views
Splunk Conf 2014 - Getting the message von Damien Dallimore
Splunk Conf 2014 - Getting the messageSplunk Conf 2014 - Getting the message
Splunk Conf 2014 - Getting the message
Damien Dallimore2.7K views
Infrastructure as code, using Terraform von Harkamal Singh
Infrastructure as code, using TerraformInfrastructure as code, using Terraform
Infrastructure as code, using Terraform
Harkamal Singh254 views
Web Development Environments: Choose the best or go with the rest von george.james
Web Development Environments:  Choose the best or go with the restWeb Development Environments:  Choose the best or go with the rest
Web Development Environments: Choose the best or go with the rest
george.james5.6K views
Near real-time anomaly detection at Lyft von markgrover
Near real-time anomaly detection at LyftNear real-time anomaly detection at Lyft
Near real-time anomaly detection at Lyft
markgrover1.9K views
Metaflow: The ML Infrastructure at Netflix von Bill Liu
Metaflow: The ML Infrastructure at NetflixMetaflow: The ML Infrastructure at Netflix
Metaflow: The ML Infrastructure at Netflix
Bill Liu523 views
After the LAMP, it's time to get MEAN von Jeff Fox
After the LAMP, it's time to get MEANAfter the LAMP, it's time to get MEAN
After the LAMP, it's time to get MEAN
Jeff Fox752 views
IBM Strategy for Spark von Mark Kerzner
IBM Strategy for SparkIBM Strategy for Spark
IBM Strategy for Spark
Mark Kerzner752 views
OS for AI: Elastic Microservices & the Next Gen of ML von Nordic APIs
OS for AI: Elastic Microservices & the Next Gen of MLOS for AI: Elastic Microservices & the Next Gen of ML
OS for AI: Elastic Microservices & the Next Gen of ML
Nordic APIs264 views
Overview of Apache Flink: Next-Gen Big Data Analytics Framework von Slim Baltagi
Overview of Apache Flink: Next-Gen Big Data Analytics FrameworkOverview of Apache Flink: Next-Gen Big Data Analytics Framework
Overview of Apache Flink: Next-Gen Big Data Analytics Framework
Slim Baltagi32.5K views
Integration Patterns for Big Data Applications von Michael Häusler
Integration Patterns for Big Data ApplicationsIntegration Patterns for Big Data Applications
Integration Patterns for Big Data Applications
Michael Häusler522 views
Making Machine Learning Easy with H2O and WebFlux von Trayan Iliev
Making Machine Learning Easy with H2O and WebFluxMaking Machine Learning Easy with H2O and WebFlux
Making Machine Learning Easy with H2O and WebFlux
Trayan Iliev119 views

Más de Kaxil Naik

Introducing airflowctl: A CLI to streamline getting started with Airflow - Ai... von
Introducing airflowctl: A CLI to streamline getting started with Airflow - Ai...Introducing airflowctl: A CLI to streamline getting started with Airflow - Ai...
Introducing airflowctl: A CLI to streamline getting started with Airflow - Ai...Kaxil Naik
4 views17 Folien
Airflow: Save Tons of Money by Using Deferrable Operators von
Airflow: Save Tons of Money by Using Deferrable OperatorsAirflow: Save Tons of Money by Using Deferrable Operators
Airflow: Save Tons of Money by Using Deferrable OperatorsKaxil Naik
780 views30 Folien
What's new in Airflow 2.3? von
What's new in Airflow 2.3?What's new in Airflow 2.3?
What's new in Airflow 2.3?Kaxil Naik
80 views23 Folien
Upgrading to Apache Airflow 2 | Airflow Summit 2021 von
Upgrading to Apache Airflow 2 | Airflow Summit 2021Upgrading to Apache Airflow 2 | Airflow Summit 2021
Upgrading to Apache Airflow 2 | Airflow Summit 2021Kaxil Naik
85 views49 Folien
Contributing to Apache Airflow | Journey to becoming Airflow's leading contri... von
Contributing to Apache Airflow | Journey to becoming Airflow's leading contri...Contributing to Apache Airflow | Journey to becoming Airflow's leading contri...
Contributing to Apache Airflow | Journey to becoming Airflow's leading contri...Kaxil Naik
119 views63 Folien
Upcoming features in Airflow 2 von
Upcoming features in Airflow 2Upcoming features in Airflow 2
Upcoming features in Airflow 2Kaxil Naik
126 views50 Folien

Más de Kaxil Naik(9)

Introducing airflowctl: A CLI to streamline getting started with Airflow - Ai... von Kaxil Naik
Introducing airflowctl: A CLI to streamline getting started with Airflow - Ai...Introducing airflowctl: A CLI to streamline getting started with Airflow - Ai...
Introducing airflowctl: A CLI to streamline getting started with Airflow - Ai...
Kaxil Naik4 views
Airflow: Save Tons of Money by Using Deferrable Operators von Kaxil Naik
Airflow: Save Tons of Money by Using Deferrable OperatorsAirflow: Save Tons of Money by Using Deferrable Operators
Airflow: Save Tons of Money by Using Deferrable Operators
Kaxil Naik780 views
What's new in Airflow 2.3? von Kaxil Naik
What's new in Airflow 2.3?What's new in Airflow 2.3?
What's new in Airflow 2.3?
Kaxil Naik80 views
Upgrading to Apache Airflow 2 | Airflow Summit 2021 von Kaxil Naik
Upgrading to Apache Airflow 2 | Airflow Summit 2021Upgrading to Apache Airflow 2 | Airflow Summit 2021
Upgrading to Apache Airflow 2 | Airflow Summit 2021
Kaxil Naik85 views
Contributing to Apache Airflow | Journey to becoming Airflow's leading contri... von Kaxil Naik
Contributing to Apache Airflow | Journey to becoming Airflow's leading contri...Contributing to Apache Airflow | Journey to becoming Airflow's leading contri...
Contributing to Apache Airflow | Journey to becoming Airflow's leading contri...
Kaxil Naik119 views
Upcoming features in Airflow 2 von Kaxil Naik
Upcoming features in Airflow 2Upcoming features in Airflow 2
Upcoming features in Airflow 2
Kaxil Naik126 views
What's coming in Airflow 2.0? - NYC Apache Airflow Meetup von Kaxil Naik
What's coming in Airflow 2.0? - NYC Apache Airflow MeetupWhat's coming in Airflow 2.0? - NYC Apache Airflow Meetup
What's coming in Airflow 2.0? - NYC Apache Airflow Meetup
Kaxil Naik182 views
Airflow Best Practises & Roadmap to Airflow 2.0 von Kaxil Naik
Airflow Best Practises & Roadmap to Airflow 2.0Airflow Best Practises & Roadmap to Airflow 2.0
Airflow Best Practises & Roadmap to Airflow 2.0
Kaxil Naik338 views
Apache Airflow in the Cloud: Programmatically orchestrating workloads with Py... von Kaxil Naik
Apache Airflow in the Cloud: Programmatically orchestrating workloads with Py...Apache Airflow in the Cloud: Programmatically orchestrating workloads with Py...
Apache Airflow in the Cloud: Programmatically orchestrating workloads with Py...
Kaxil Naik2.9K views

Último

VoxelNet von
VoxelNetVoxelNet
VoxelNettaeseon ryu
9 views21 Folien
[DSC Europe 23] Spela Poklukar & Tea Brasanac - Retrieval Augmented Generation von
[DSC Europe 23] Spela Poklukar & Tea Brasanac - Retrieval Augmented Generation[DSC Europe 23] Spela Poklukar & Tea Brasanac - Retrieval Augmented Generation
[DSC Europe 23] Spela Poklukar & Tea Brasanac - Retrieval Augmented GenerationDataScienceConferenc1
15 views29 Folien
How Leaders See Data? (Level 1) von
How Leaders See Data? (Level 1)How Leaders See Data? (Level 1)
How Leaders See Data? (Level 1)Narendra Narendra
15 views76 Folien
[DSC Europe 23] Ivana Sesic - Use of AI in Public Health.pptx von
[DSC Europe 23] Ivana Sesic - Use of AI in Public Health.pptx[DSC Europe 23] Ivana Sesic - Use of AI in Public Health.pptx
[DSC Europe 23] Ivana Sesic - Use of AI in Public Health.pptxDataScienceConferenc1
5 views15 Folien
SAP-TCodes.pdf von
SAP-TCodes.pdfSAP-TCodes.pdf
SAP-TCodes.pdfmustafaghulam8181
10 views285 Folien
CRM stick or twist workshop von
CRM stick or twist workshopCRM stick or twist workshop
CRM stick or twist workshopinfo828217
10 views16 Folien

Último(20)

[DSC Europe 23] Spela Poklukar & Tea Brasanac - Retrieval Augmented Generation von DataScienceConferenc1
[DSC Europe 23] Spela Poklukar & Tea Brasanac - Retrieval Augmented Generation[DSC Europe 23] Spela Poklukar & Tea Brasanac - Retrieval Augmented Generation
[DSC Europe 23] Spela Poklukar & Tea Brasanac - Retrieval Augmented Generation
CRM stick or twist workshop von info828217
CRM stick or twist workshopCRM stick or twist workshop
CRM stick or twist workshop
info82821710 views
[DSC Europe 23][Cryptica] Martin_Summer_Digital_central_bank_money_Ideas_init... von DataScienceConferenc1
[DSC Europe 23][Cryptica] Martin_Summer_Digital_central_bank_money_Ideas_init...[DSC Europe 23][Cryptica] Martin_Summer_Digital_central_bank_money_Ideas_init...
[DSC Europe 23][Cryptica] Martin_Summer_Digital_central_bank_money_Ideas_init...
Chapter 3b- Process Communication (1) (1)(1) (1).pptx von ayeshabaig2004
Chapter 3b- Process Communication (1) (1)(1) (1).pptxChapter 3b- Process Communication (1) (1)(1) (1).pptx
Chapter 3b- Process Communication (1) (1)(1) (1).pptx
ayeshabaig20047 views
Data Journeys Hard Talk workshop final.pptx von info828217
Data Journeys Hard Talk workshop final.pptxData Journeys Hard Talk workshop final.pptx
Data Journeys Hard Talk workshop final.pptx
info82821710 views
Ukraine Infographic_22NOV2023_v2.pdf von AnastosiyaGurin
Ukraine Infographic_22NOV2023_v2.pdfUkraine Infographic_22NOV2023_v2.pdf
Ukraine Infographic_22NOV2023_v2.pdf
AnastosiyaGurin1.4K views
Survey on Factuality in LLM's.pptx von NeethaSherra1
Survey on Factuality in LLM's.pptxSurvey on Factuality in LLM's.pptx
Survey on Factuality in LLM's.pptx
NeethaSherra17 views
OECD-Persol Holdings Workshop on Advancing Employee Well-being in Business an... von StatsCommunications
OECD-Persol Holdings Workshop on Advancing Employee Well-being in Business an...OECD-Persol Holdings Workshop on Advancing Employee Well-being in Business an...
OECD-Persol Holdings Workshop on Advancing Employee Well-being in Business an...
[DSC Europe 23] Zsolt Feleki - Machine Translation should we trust it.pptx von DataScienceConferenc1
[DSC Europe 23] Zsolt Feleki - Machine Translation should we trust it.pptx[DSC Europe 23] Zsolt Feleki - Machine Translation should we trust it.pptx
[DSC Europe 23] Zsolt Feleki - Machine Translation should we trust it.pptx
CRM stick or twist.pptx von info828217
CRM stick or twist.pptxCRM stick or twist.pptx
CRM stick or twist.pptx
info82821711 views
[DSC Europe 23][AI:CSI] Dragan Pleskonjic - AI Impact on Cybersecurity and P... von DataScienceConferenc1
[DSC Europe 23][AI:CSI]  Dragan Pleskonjic - AI Impact on Cybersecurity and P...[DSC Europe 23][AI:CSI]  Dragan Pleskonjic - AI Impact on Cybersecurity and P...
[DSC Europe 23][AI:CSI] Dragan Pleskonjic - AI Impact on Cybersecurity and P...
UNEP FI CRS Climate Risk Results.pptx von pekka28
UNEP FI CRS Climate Risk Results.pptxUNEP FI CRS Climate Risk Results.pptx
UNEP FI CRS Climate Risk Results.pptx
pekka2811 views
Short Story Assignment by Kelly Nguyen von kellynguyen01
Short Story Assignment by Kelly NguyenShort Story Assignment by Kelly Nguyen
Short Story Assignment by Kelly Nguyen
kellynguyen0119 views

Building and deploying LLM applications with Apache Airflow

  • 2. Kaxil Naik Apache Airflow Committer & PMC Member Director of Eng @ Astronomer Julian LaNeve Senior Product Manager @ Astronomer
  • 3. Why Airflow should be at the centre of LLMOps? Real Use-case & reference architecture Next Steps: Community collaboration Agenda
  • 4. A powerful new class of large language models is making it possible for machines to write, code, draw, and create with credible and sometimes superhuman results. Generative AI: A Creative New World
  • 5. Normally, for ML, you need to… Ingest Data Train Model Prediction
  • 6. …but now you can You hit a pre-trained model instead of your own model …but now you can: Ingest Data Train Model Prediction Less Data
  • 8. ■ Ingestion from several sources ■ Day 2 operations on data pipelines ■ Data preparation ■ Data privacy ■ Data freshness ■ Model deployment & monitoring ■ Scaling Models ■ Experimentation & fine-tuning ■ Feedback Loops Going from “Idea to Production” with LLM Apps involves solving a lot of data engineering problems:
  • 9. Source: https://python.langchain.com/docs/use_cases/question_answering/ Typical Architecture for Q&A use-case using LLM Legacy Data Store Retrieval Output Storage Splitting Document Loading Vectorstore Database PDFs URLs LLM <Answer> Prompt Splits Relevant Splits Query <Question>
  • 10. Python Native The language of data scientists and ML engineers. Pluggable Compute GPUs, Kubernetes, EC2, VMs etc. Common Interface Between Data Engineering, Data Science, ML Engineering and Operations. Data Agnostic But data aware. Extensible Standardize custom operators and templates for common DS tasks across the organization. Monitoring & Alerting Built in features for logging, monitoring and alerting to external systems. Ingestion Extract and load data into vectordbs and other destinations Day 2 Ops Handle retries, dependencies, and all other day 2 ops associated with data pipelines Airflow is a Natural Fit… Document Parsing Decorator and pythonic interfaces for standard LLM tools
  • 11. Let’s Talk About a Real Use Case
  • 12. Problem Statement: We have customers, employees, and community members that ask questions about our product with answers that exist across several sources of documentation. How do we provide an easy interface for folks to get their questions answered without adding further strain to the team?
  • 14. ■ Airflow gives a framework to load data from APIs & other sources into LangChain ■ LangChain helps pre-process and split documents into smaller chunks depending on content type ■ After content is split into chunks, each chunk is embedded into vectors (semantic representations) ■ Those vectors are written to Weaviate for later retrieval Data Ingestion, Processing, and Embedding Embed chunks Write to Weaviate Pre-process and split into chunks 🦜🔗 LangChain Docs (.md) files Slack Messages GitHub issues Docs (.md) files
  • 15. Users can interact with UI or Slack Bot; they both use the same API ■ Original prompt gets reworded 3x using gpt-3.5-turbo ■ Answer is generated by combining docs from each prompt and making a gpt-4 call ■ State is stored in Firestore and prompt tracing is done through LangSmith 🦜🔗LangChain User Asks a Question Web App Slack Bot Original Prompt Rewording 2 Rewording 1 Rewording 3 Reword to get more related documents Vector DB search with prompts Combine docs and make final LLM call to answer 🦜 🔗 Prompt Orchestration and Answering
  • 16. ■ Airflow DAGs process feedback async to evaluate answers on helpfulness,, relevance, and publicness ■ If answer is good, it gets stored in Weaviate and can be used as a source for future questions ■ UI also shows the most recent good prompts on the homepage When a user submits feedback, it gets stored in Firestore and LangSmith for later use User Rates Answer 🦜 🔗 Fetch new runs: input, output, and user feedback Classify Q&A according to helpfulness, relevance, and public 🦜🔗 LangChain If good answer, write to vector DB to use in future answers If good answer, mark as good to show on Ask Astro homepage On schedule LLM & Product Feedback Loops
  • 17. Running this in production meant: ■ Experimenting with different sources of data to ingest ■ Running the pipelines on a schedule and ad-hoc ■ Running the same workloads with variable chunking strategies ■ Needing to retry tasks due to finicky python libraries and unreliable external services ■ Giving different parts of the workload variable compute ■ Creating standard interfaces to interact with external systems
  • 18. ■ Experimenting with different sources of data to ingest ■ Running the pipelines on a schedule and ad-hoc ■ Running the same workloads with variable chunking strategies ■ Needing to retry tasks due to finicky python libraries and unreliable external services ■ Giving different parts of the workload variable compute ■ Creating standard interfaces to interact with external systems Running this in production meant: Which is what Airflow’s great at!
  • 20. a16z’s Emerging LLM App Stack Orchestration (Python/DIY, LangChain, LlamaIndex, ChatGPT) APIs/Plugins (Serp, Wolfram, Zapier, etc.) App Hosting (Vercel, Steamship, Streamlit, Modal) Query Output Prompt Few-shot examples Contextual data Playground (OpenAI, nat.dev, Humanloop) Data Pipelines (Databricks, Airflow, Unstructured, etc.) Embedding Model (OpenAI, Cohere, Hugging Face) Vector Database (Pinecone, Weaviate, Chroma, pgvector) LLM Cache (Redis, SQLite, GPTCache) Logging/LLMops (Weights & Biases, MLflow, PromptLayer, Helicone) Validation (Guardrails, Rebuff, Guidance, LMQL) Proprietary API (OpenAI, Anthropic) Open API (Hugging Face, Replicate) Opinionated Cloud (Databricks, Anyscale, Mosaic, Modal, Runpod) Cloud Provider (AWS, GCP, Azure, Coreweave) LLM APIs and Hosting Gray boxes show key components of the stack, with leading tools / systems listed. Arrows show the flow of data through the stack. Contextual data provided by app developers to condition LLM outputs Prompts and few-shot examples that are sent to the LLM Queries submitted by users Output returned to users Legend
  • 21. AskAstro has a few parts of this… Orchestration (Python/DIY, LangChain, LlamaIndex, ChatGPT) APIs/Plugins (Serp, Wolfram, Zapier, etc.) App Hosting (Vercel, Steamship, Streamlit, Modal) Query Output Prompt Few-shot examples Contextual data Playground (OpenAI, nat.dev, Humanloop) Data Pipelines (Databricks, Airflow, Unstructured, etc.) Embedding Model (OpenAI, Cohere, Hugging Face) Vector Database (Pinecone, Weaviate, Chroma, pgvector) LLM Cache (Redis, SQLite, GPTCache) Logging/LLMops (Weights & Biases, MLflow, PromptLayer, Helicone) Validation (Guardrails, Rebuff, Guidance, LMQL) Proprietary API (OpenAI, Anthropic) Open API (Hugging Face, Replicate) Opinionated Cloud (Databricks, Anyscale, Mosaic, Modal, Runpod) Cloud Provider (AWS, GCP, Azure, Coreweave) LLM APIs and Hosting Gray boxes show key components of the stack, with leading tools / systems listed. Arrows show the flow of data through the stack. Contextual data provided by app developers to condition LLM outputs Prompts and few-shot examples that are sent to the LLM Queries submitted by users Output returned to users Legend
  • 22. Airflow is foundational to best practices for all of this. Data Governance ■ How do you account for private data? ■ How do you provide transparency into data lineage? Fine Tuning ■ Does it improve results? ■ How much does it cost? Feedback Loops ■ Semantic cache for correct responses ■ Ranking sources based on accuracy and ranking accordingly ■ Prompt clustering – what are people asking? …but there’s even more to consider.
  • 23. Michael Gregory Philippe Gagnon Thanks to the AskAstro Team:
  • 25. What are all the providers the ecosystem needs? pgvector
  • 28. Patterns What are the best practices for building pipelines for LLM Apps? ■ Do you use one task to ingest and write? ■ Can you use dynamic task mapping to break it out? ■ Do you write to disk? ■ Can you store embedding values in XCOMs? ■ How do you reconcile Airflow orchestration with prompt orchestration?
  • 29. Let’s do this all in the open source!