Large-scale data processing with Hadoop in the Cloud

•Als KEY, PDF herunterladen•

1 gefällt mir•420 views

A lightning talk that I gave at CloudCamp San Diego on how Hadoop can be used in the cloud to process more than 100,000,000 records

Technologie Business

Large-scale data processing
with Hadoop in the Cloud

by Patrick Salami
Senior Software Engineer

Hadoop

Distributed computing framework

Runs on commodity hardware

Splits up large data sets

Easy to deploy in AWS with Elastic
MapReduce

Java Code
Raw Product
Normalize
Data
Filter

Categorize
S3
Shipping

Coupons
Hadoop Cluster
Group

Post-Process

Solr
S3 WWW
Index

x32 nodes
compute time $0.17/h $5.44/h
EMR $0.03/h $0.96/h

Total $0.20/h $6.40/h

Source: http://aws.amazon.com/ec2/#pricing

twitter.com/psalami
linkedin.com/in/psalami

Weitere ähnliche Inhalte

Andere mochten auch

Hadoop on the Cloud

Amazon Web Services

Hadoop in the Cloud - The what, why and how from the experts

DataWorks Summit/Hadoop Summit

Hadoop Security in Big-Data-as-a-Service Deployments - Presented at Hadoop Su...

Abhiraj Butala

BlueTalon-Isilon-Validation

Boni Bruno

The purpose of these slides is to give a high-level overview of Big Data concepts and techniques, as well as its related tools and technologies, focusing on Microsoft Azure. It starts by defining what Big Data is, as well as why Big Data platforms are needed. Fundamental components of a Big Data Platform are discussed, followed by a little bit of theory about Distributed Processing & CAP Theorem, and its relevance to how Big Data Solutions compare to Traditional RDBMS. Use case of how Big Data fits in Enterprise Data Platforms are shown. The Hadoop Ecosystem is briefly reviewed before Big Data on Microsoft Azure is discussed. Then some directions of How to get started with Big Data.

Intorducing Big Data and Microsoft Azure

Khalid Salama

Moving towards enterprise ready Hadoop clusters on the cloud

DataWorks Summit/Hadoop Summit

Hadoop in the Cloud – The What, Why and How from the Experts

DataWorks Summit/Hadoop Summit

Big Data in Azure

DataWorks Summit/Hadoop Summit

Microsoft Azure Big Data Analytics

Mark Kromer

Introducing DocumentDB

James Serra

Maintainable cloud architecture_of_hadoop

Kai Sasaki

This talk given at the Hadoop Summit in San Jose on June 28, 2016, analyzes a few major trends in Big Data analytics. These are a few takeaways from this talk: - Adopt Apache Beam for easier development and portability between Big Data Execution Engines. - Adopt stream analytics for faster time to insight, competitive advantages and operational efficiency. - Accelerate your Big Data applications with In-Memory open source tools. - Adopt Rapid Application Development of Big Data applications: APIs, Notebooks, GUIs, Microservices… - Have Machine Learning part of your strategy or passively watch your industry completely transformed! - How to advance your strategy for hybrid integration between cloud and on-premise deployments?

Analysis-of-Major-Trends-in-big-data-analytics-slim-baltagi-hadoop-summit

Slim Baltagi

Compare Clouds: Aws vs Azure vs Google vs SoftLayer

RightScale

The Future of Apache Hadoop an Enterprise Architecture View

DataWorks Summit/Hadoop Summit

cloud computing ppt

himanshuawasthi2109

Andere mochten auch (15)

Hadoop on the Cloud

Hadoop in the Cloud - The what, why and how from the experts

Hadoop Security in Big-Data-as-a-Service Deployments - Presented at Hadoop Su...

BlueTalon-Isilon-Validation

Intorducing Big Data and Microsoft Azure

Moving towards enterprise ready Hadoop clusters on the cloud

Hadoop in the Cloud – The What, Why and How from the Experts

Big Data in Azure

Microsoft Azure Big Data Analytics

Introducing DocumentDB

Maintainable cloud architecture_of_hadoop

Analysis-of-Major-Trends-in-big-data-analytics-slim-baltagi-hadoop-summit

Compare Clouds: Aws vs Azure vs Google vs SoftLayer

The Future of Apache Hadoop an Enterprise Architecture View

cloud computing ppt

Kürzlich hochgeladen

Discord is a free app offering voice, video, and text chat functionalities, primarily catering to the gaming community. It serves as a hub for users to create and join servers tailored to their interests. Discord’s ecosystem comprises servers, each functioning as a distinct online community with its own channels dedicated to specific topics or activities. Users can engage in text-based discussions, voice calls, or video chats within these channels. Understanding Discord Servers Discord servers are virtual spaces where users congregate to interact, share content, and build communities. Servers may revolve around gaming, hobbies, interests, or fandoms, providing a platform for like-minded individuals to connect. Communication Features Discord offers a range of communication tools, including text channels for messaging, voice channels for real-time audio conversations, and video channels for face-to-face interactions. These features facilitate seamless communication and collaboration. What Does NSFW Mean? The acronym NSFW stands for “Not Safe For Work,” indicating content that may be inappropriate for professional or public settings. NSFW Content NSFW content encompasses material that is sexually explicit, violent, or otherwise graphic in nature. It often includes nudity, profanity, or depictions of sensitive topics.

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

UK Journal

🐬 The future of MySQL is Postgres 🐘

RTylerCroy

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Martijn de Jong

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

What is a good lead in your organisation? Which leads are priority? What happens to leads? When sales and marketing give different answers to these questions, or perhaps aren't sure of the answers at all, frustrations build and opportunities are left on the table. Join us for an illuminating session with Cian McLoughlin, HubSpot Principal Customer Success Manager, as we look at that crucial piece of the customer journey in which leads are transferred from marketing to sales.

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

HampshireHUG

As privacy and data protection regulations evolve rapidly, organizations operating in multiple jurisdictions face mounting challenges to ensure compliance and safeguard customer data. With state-specific privacy laws coming up in multiple states this year, it is essential to understand what their unique data protection regulations will require clearly. How will data privacy evolve in the US in 2024? How to stay compliant? Our panellists will guide you through the intricacies of these states' specific data privacy laws, clarifying complex legal frameworks and compliance requirements. This webinar will review: - The essential aspects of each state's privacy landscape and the latest updates - Common compliance challenges faced by organizations operating in multiple states and best practices to achieve regulatory adherence - Valuable insights into potential changes to existing regulations and prepare your organization for the evolving landscape

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc

Data Cloud, More than a CDP by Matt Robison

Anna Loughnan Colquhoun

[2024]Digital Global Overview Report 2024 Meltwater.pdf

hans926745

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

sammart93

Abhishek Deb(1), Mr Abdul Kalam(2) M. Des (UX) , School of Design, DIT University , Dehradun. This paper explores the future potential of AI-enabled smartphone processors, aiming to investigate the advancements, capabilities, and implications of integrating artificial intelligence (AI) into smartphone technology. The research study goals consist of evaluating the development of AI in mobile phone processors, analyzing the existing state as well as abilities of AI-enabled cpus determining future patterns as well as chances together with reviewing obstacles as well as factors to consider for more growth.

Exploring the Future Potential of AI-Enabled Smartphone Processors

debabhi2

A Domino Admins Adventures (Engage 2024)

Gabriella Davis

Building Digital Trust in a Digital Economy Veronica Tan, Director - Cyber Security Agency of Singapore Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

apidays

Histor y of HAM Radio presentation slide

vu2urc

Enterprise Knowledge’s Urmi Majumder, Principal Data Architecture Consultant, and Fernando Aguilar Islas, Senior Data Science Consultant, presented "Driving Behavioral Change for Information Management through Data-Driven Green Strategy" on March 27, 2024 at Enterprise Data World (EDW) in Orlando, Florida. In this presentation, Urmi and Fernando discussed a case study describing how the information management division in a large supply chain organization drove user behavior change through awareness of the carbon footprint of their duplicated and near-duplicated content, identified via advanced data analytics. Check out their presentation to gain valuable perspectives on utilizing data-driven strategies to influence positive behavioral shifts and support sustainability initiatives within your organization. In this session, participants gained answers to the following questions: - What is a Green Information Management (IM) Strategy, and why should you have one? - How can Artificial Intelligence (AI) and Machine Learning (ML) support your Green IM Strategy through content deduplication? - How can an organization use insights into their data to influence employee behavior for IM? - How can you reap additional benefits from content reduction that go beyond Green IM?

Driving Behavioral Change for Information Management through Data-Driven Gree...

Enterprise Knowledge

Axa Assurance Maroc - Insurer Innovation Award 2024

The Digital Insurer

BooK Now Call us at +918448380779 to hire a gorgeous and seductive call girl for sex. Take a Delhi Escort Service. The help of our escort agency is mostly meant for men who want sexual Indian Escorts In Delhi NCR. It should be noted that any impersonator will get 100 attention from our Young Girls Escorts in Delhi. They will assume the position of reliable allies. VIP Call Girl With Original Photos Book Tonight +918448380779 Our Cheap Price 1 Hour not available 2 Hours 5000 Full Night 8000 TAG: Call Girls in Delhi, Noida, Gurgaon, Ghaziabad, Connaught Place, Greater Kailash Delhi, Lajpat Nagar Delhi, Mayur Vihar Delhi, Chanakyapuri Delhi, New Friends Colony Delhi, Majnu Ka Tilla, Karol Bagh, Malviya Nagar, Saket, Khan Market, Noida Sector 18, Noida Sector 76, Noida Sector 51, Gurgaon Mg Road, Iffco Chowk Gurgaon, Rajiv Chowk Gurgaon All Delhi Ncr Free Home Deliver

08448380779 Call Girls In Friends Colony Women Seeking Men

Delhi Call girls

08448380779 Call Girls In Civil Lines Women Seeking Men

Delhi Call girls

How to convert PDF to text with Nanonets

naman860154

In an era where artificial intelligence (AI) stands at the forefront of business innovation, Information Architecture (IA) is at the core of functionality. See “There’s No AI Without IA” – (from 2016 but even more relevant today) Understanding and leveraging how Information Architecture (IA) supports AI synergies between knowledge engineering and prompt engineering is critical for senior leaders looking to successfully deploy AI for internal and externally facing knowledge processes. This webinar be a high-level overview of the methodologies that can elevate AI-driven knowledge processes supporting both employees and customers. Core Insights Include: Strategic Knowledge Engineering: Delve into how structuring AI's knowledge base is required to prevent hallucinations, enable contextual retrieval of accurate information. This will include discussion of gold standard libraries of use cases support testing various LLMs and structures and configurations of knowledge base. Precision in Prompt Engineering: Learn the art of crafting prompts that direct AI to deliver targeted, relevant responses, thereby optimizing customer experiences and business outcomes. Unified Approach for Enhanced AI Performance: Explore the intersection of knowledge and prompt engineering to develop AI systems that are not only more responsive but also aligned with overarching business strategies. Guiding Principles for Implementation: Equip yourself with best practices, ethical guidelines, and strategic considerations for embedding these technologies into your business ecosystem effectively. This webinar is designed to empower business and technology leaders with the knowledge to harness the full potential of AI, ensuring their organizations not only keep pace with digital transformation but lead the charge. Join us to map a roadmap to fully leverage Information Architecture (IA) and AI chart a course towards a future where AI is a key pillar of strategic innovation and business success.

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx

Earley Information Science

Kürzlich hochgeladen (20)

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

🐬 The future of MySQL is Postgres 🐘

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

Data Cloud, More than a CDP by Matt Robison

[2024]Digital Global Overview Report 2024 Meltwater.pdf

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Exploring the Future Potential of AI-Enabled Smartphone Processors

A Domino Admins Adventures (Engage 2024)

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

Histor y of HAM Radio presentation slide

Driving Behavioral Change for Information Management through Data-Driven Gree...

Axa Assurance Maroc - Insurer Innovation Award 2024

08448380779 Call Girls In Friends Colony Women Seeking Men

08448380779 Call Girls In Civil Lines Women Seeking Men

How to convert PDF to text with Nanonets

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx