SlideShare ist ein Scribd-Unternehmen logo
1 von 30
Apache Storm
A distributed, real-time computation
system

Ryan Lanman

Some content borrowed from Nathan Marz’
Presentation of a similar name
Objectives
1.Their Motivation
2.Our Motivation
3.Storm Basics
4.Demo
Their Motivation
How Storm Came To Be
What They Wanted
•
•
•
•
•
•

Guaranteed data processing
Horizontal scalability
Fault-tolerance
No intermediate message brokers!
Higher level abstraction than message passing
“Just works”
Our Motivation
Why We Chose Storm
^
Lumify Ingest
Raw
Data

Text Extraction

Entity Extraction

Text Highlighting

Location Extraction

Full Text Indexing
Issues
•
•
•
•
•

No Reducers
High DB Read/Writes
Batch-style processing
M/R Overhead
Zero Fault Tolerance
What We Really Wanted

• Distributed, Stream-type Processing
• Simple Logical DAG
• Better Fault Tolerance
Storm Ingest Workflow
Documents

Raw Data

Content
Sorter

Video

Images

Text
Extraction

Video
Frame
Splitting

Image Text
Extraction

Video
Frame Text
Extraction

Text
…
Storm Basics
What the heck’s a Topology?
Storm Cluster
Supervisor

Zookeeper

Nimbus

Supervisor

Zookeeper

Supervisor

Zookeeper

Supervisor

Supervisor
Storm Cluster
Supervisor

Zookeeper

Nimbus

Supervisor

Zookeeper

Supervisor

Zookeeper

Supervisor

Supervisor
Storm Cluster
Supervisor

Zookeeper

Nimbus

Supervisor

Zookeeper

Supervisor

Zookeeper

Supervisor

Supervisor
Storm Cluster
Supervisor

Zookeeper

Nimbus

Supervisor

Zookeeper

Supervisor

Zookeeper

Supervisor

Supervisor
Storm Data Concepts
•
•
•
•
•

Tuples
Streams
Spouts
Bolts
Topologies
Tuples
• Single unit of data in Storm
• Examples
– Tweet
– User Activity Log Entry
– File Info
Streams

Tuple

Tuple

Tuple

Tuple

Tuple

An unbound sequence of Tuples

Tuple

Tuple
Spouts

Spout

Producers of Streams
Bolts

Tuple

Process input streams to create new streams

Tuple
Examples
Spout Examples
• HDFS Filesystem Spout
• Kafka Queue Spout

Bolt Examples
• Filtering
• Aggregation
• DB Operations
Topologies
Spout

Spout

Spout
Demo
Demo Topology

Twitter

Twitter
Hosebird
Spout
Sentence
Splitter

Word
Count

Accumulo
Demo Topology

Twitter

Twitter
Hosebird
Spout

Shuffle
Grouping

Field
Grouping

Sentence
Splitter

Word
Count

Accumulo

Weitere ähnliche Inhalte

Ähnlich wie Storm - Altamira University Presentation

CRUD Operation of images through XML
CRUD Operation of images through XMLCRUD Operation of images through XML
CRUD Operation of images through XMLAnshudha Maheshwari
 
Webinar: Ransomware - Five Reasons You’re Not As Protected As You Think
Webinar: Ransomware - Five Reasons You’re Not As Protected As You ThinkWebinar: Ransomware - Five Reasons You’re Not As Protected As You Think
Webinar: Ransomware - Five Reasons You’re Not As Protected As You ThinkStorage Switzerland
 
Velocity 2016 Speaking Session - Using Machine Learning to Determine Drivers ...
Velocity 2016 Speaking Session - Using Machine Learning to Determine Drivers ...Velocity 2016 Speaking Session - Using Machine Learning to Determine Drivers ...
Velocity 2016 Speaking Session - Using Machine Learning to Determine Drivers ...SOASTA
 
Using machine learning to determine drivers of bounce and conversion
Using machine learning to determine drivers of bounce and conversionUsing machine learning to determine drivers of bounce and conversion
Using machine learning to determine drivers of bounce and conversionTammy Everts
 
InheritedASecurityDept
InheritedASecurityDeptInheritedASecurityDept
InheritedASecurityDeptAmanda Berlin
 
Watchtowers of the Internet - Source Boston 2012
Watchtowers of the Internet - Source Boston 2012Watchtowers of the Internet - Source Boston 2012
Watchtowers of the Internet - Source Boston 2012Stephan Chenette
 
Life in a Queue - Using Message Queue with django
Life in a Queue - Using Message Queue with djangoLife in a Queue - Using Message Queue with django
Life in a Queue - Using Message Queue with djangoTareque Hossain
 
Zane lackey. security at scale. web application security in a continuous depl...
Zane lackey. security at scale. web application security in a continuous depl...Zane lackey. security at scale. web application security in a continuous depl...
Zane lackey. security at scale. web application security in a continuous depl...Yury Chemerkin
 
Apachecon Euro 2012: Elastic, Multi-tenant Hadoop on Demand
Apachecon Euro 2012: Elastic, Multi-tenant Hadoop on DemandApachecon Euro 2012: Elastic, Multi-tenant Hadoop on Demand
Apachecon Euro 2012: Elastic, Multi-tenant Hadoop on DemandRichard McDougall
 
Scaling Ruby for Enterprise Applications
Scaling Ruby for Enterprise ApplicationsScaling Ruby for Enterprise Applications
Scaling Ruby for Enterprise ApplicationsPro777
 
Continuous Automated Red Teaming (CART) - Bikash Barai
Continuous Automated Red Teaming (CART) - Bikash BaraiContinuous Automated Red Teaming (CART) - Bikash Barai
Continuous Automated Red Teaming (CART) - Bikash BaraiAllanGray11
 
Chirp 2010: Scaling Twitter
Chirp 2010: Scaling TwitterChirp 2010: Scaling Twitter
Chirp 2010: Scaling TwitterJohn Adams
 
Message handler customer deck
Message handler customer deckMessage handler customer deck
Message handler customer deckYves Goeleven
 
Python for pentesters
Python for pentestersPython for pentesters
Python for pentestersRashid feroz
 
Zombie Annihilation Using AWS Big Data - ABD328 - re:Invent 2017
Zombie Annihilation Using AWS Big Data - ABD328 - re:Invent 2017Zombie Annihilation Using AWS Big Data - ABD328 - re:Invent 2017
Zombie Annihilation Using AWS Big Data - ABD328 - re:Invent 2017Amazon Web Services
 

Ähnlich wie Storm - Altamira University Presentation (20)

CRUD Operation of images through XML
CRUD Operation of images through XMLCRUD Operation of images through XML
CRUD Operation of images through XML
 
Webinar: Ransomware - Five Reasons You’re Not As Protected As You Think
Webinar: Ransomware - Five Reasons You’re Not As Protected As You ThinkWebinar: Ransomware - Five Reasons You’re Not As Protected As You Think
Webinar: Ransomware - Five Reasons You’re Not As Protected As You Think
 
Velocity 2016 Speaking Session - Using Machine Learning to Determine Drivers ...
Velocity 2016 Speaking Session - Using Machine Learning to Determine Drivers ...Velocity 2016 Speaking Session - Using Machine Learning to Determine Drivers ...
Velocity 2016 Speaking Session - Using Machine Learning to Determine Drivers ...
 
Using machine learning to determine drivers of bounce and conversion
Using machine learning to determine drivers of bounce and conversionUsing machine learning to determine drivers of bounce and conversion
Using machine learning to determine drivers of bounce and conversion
 
InheritedASecurityDept
InheritedASecurityDeptInheritedASecurityDept
InheritedASecurityDept
 
Watchtowers of the Internet - Source Boston 2012
Watchtowers of the Internet - Source Boston 2012Watchtowers of the Internet - Source Boston 2012
Watchtowers of the Internet - Source Boston 2012
 
Storm Processing Internals
Storm Processing InternalsStorm Processing Internals
Storm Processing Internals
 
Scalability and Availability
Scalability and AvailabilityScalability and Availability
Scalability and Availability
 
Life in a Queue - Using Message Queue with django
Life in a Queue - Using Message Queue with djangoLife in a Queue - Using Message Queue with django
Life in a Queue - Using Message Queue with django
 
Zane lackey. security at scale. web application security in a continuous depl...
Zane lackey. security at scale. web application security in a continuous depl...Zane lackey. security at scale. web application security in a continuous depl...
Zane lackey. security at scale. web application security in a continuous depl...
 
Apache Storm Internals
Apache Storm InternalsApache Storm Internals
Apache Storm Internals
 
Apachecon Euro 2012: Elastic, Multi-tenant Hadoop on Demand
Apachecon Euro 2012: Elastic, Multi-tenant Hadoop on DemandApachecon Euro 2012: Elastic, Multi-tenant Hadoop on Demand
Apachecon Euro 2012: Elastic, Multi-tenant Hadoop on Demand
 
Scaling Ruby for Enterprise Applications
Scaling Ruby for Enterprise ApplicationsScaling Ruby for Enterprise Applications
Scaling Ruby for Enterprise Applications
 
Continuous Automated Red Teaming (CART) - Bikash Barai
Continuous Automated Red Teaming (CART) - Bikash BaraiContinuous Automated Red Teaming (CART) - Bikash Barai
Continuous Automated Red Teaming (CART) - Bikash Barai
 
Talos
TalosTalos
Talos
 
Chirp 2010: Scaling Twitter
Chirp 2010: Scaling TwitterChirp 2010: Scaling Twitter
Chirp 2010: Scaling Twitter
 
Message handler customer deck
Message handler customer deckMessage handler customer deck
Message handler customer deck
 
Python for pentesters
Python for pentestersPython for pentesters
Python for pentesters
 
Zombie Annihilation Using AWS Big Data - ABD328 - re:Invent 2017
Zombie Annihilation Using AWS Big Data - ABD328 - re:Invent 2017Zombie Annihilation Using AWS Big Data - ABD328 - re:Invent 2017
Zombie Annihilation Using AWS Big Data - ABD328 - re:Invent 2017
 
Running Yarn at Scale
Running Yarn at Scale Running Yarn at Scale
Running Yarn at Scale
 

Kürzlich hochgeladen

Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Scriptwesley chun
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationRadu Cotescu
 
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Igalia
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonAnna Loughnan Colquhoun
 
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024The Digital Insurer
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonetsnaman860154
 
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsTop 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsRoshan Dwivedi
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...Neo4j
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slidevu2urc
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityPrincipled Technologies
 
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEarley Information Science
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationSafe Software
 
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...gurkirankumar98700
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)Gabriella Davis
 
Breaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path MountBreaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path MountPuma Security, LLC
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking MenDelhi Call girls
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsMaria Levchenko
 
Developing An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of BrazilDeveloping An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of BrazilV3cube
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Miguel Araújo
 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Enterprise Knowledge
 

Kürzlich hochgeladen (20)

Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
 
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonets
 
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsTop 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 
Breaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path MountBreaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path Mount
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
 
Developing An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of BrazilDeveloping An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of Brazil
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...
 

Storm - Altamira University Presentation