Apache Hadoop India Summit 2011 talk "Adaptive Parallel Computing over Distributed Military Computing Infrastructures" by Rituraj Kumar

•Als PPTX, PDF herunterladen•

0 gefällt mir•421 views

Yahoo Developer Network

Adaptive parallel computing over distributed military computing infrastructures RiturajKumar Center of Artificial Intelligence and Robotics – DRDO Lab

2 Contents ,[object Object]

Adaptive Parallel Computing

Approaches for APC

MPI + DDS

Hadoop

Conclusions,[object Object]

Parallel Computing is the only feasible solution that would guarantee reduced response time.

Tactical deployment would not amenable for –

Large Clusters in the field

Back-hauling

Intelligent use of the existing spare compute capacity of the computing devices within the cloud is essential.,[object Object]

Weitere ähnliche Inhalte

Was ist angesagt?

F07-Cloud-Hadoop-BAM

F07-Cloud-Hadoop-BAM

F07-Cloud-Hadoop-BAM

Bioinformatics Open Source Conference

HTML slides and longer abstract can be found at https://github.com/ljdursi/EuroMPI2016. For years, the academic science and engineering community was almost alone in pursuing very large-scale numerical computing, and MPI was the lingua franca for such work. But starting in the mid-2000s, we were no longer alone. First internet-scale companies like Google and Yahoo! started performing fairly basic analytics tasks at enormous scale, and since then others have begun tackling increasingly complex and data-heavy machine-learning computations, which involve very familiar scientific computing primitives such as linear algebra, unstructured mesh decomposition, and numerical optimization. These new communities have created programming environments which emphasize what we’ve learned about computer science and programmability since 1994 – with greater levels of abstraction and encapsulation, separating high-level computation from the low-level implementation details. At about the same time, new academic research communities began using computing at scale to attack their problems - but in many cases, an ideal distributed-memory application for them begins to look more like the new concurrent distributed databases than a large CFD simulation, with data structures like dynamic hash tables and Bloom trees playing more important roles than rectangular arrays or unstructured meshes. These new academic communities are among the first to adopt emerging big-data technologies over traditional HPC options; but as big-data technologies improve their tightly-coupled number-crunching capabilities, they are unlikely to be the last. In this talk, I sketch out the landscape of distributed technical computing frameworks and environments, and look to see where MPI and the MPI community fits in to this new ecosystem.

EuroMPI 2016 Keynote: How Can MPI Fit Into Today's Big Computing

EuroMPI 2016 Keynote: How Can MPI Fit Into Today's Big Computing

EuroMPI 2016 Keynote: How Can MPI Fit Into Today's Big Computing

OpenPOWER Webinar on Machine Learning for Academic Research

OpenPOWER Webinar on Machine Learning for Academic Research

OpenPOWER Webinar on Machine Learning for Academic Research

Ganesan Narayanasamy

IBM Bayesian Optimization Accelerator (BOA) is a do-it-yourself toolkit to apply state-of-the-art Bayesian inferencing techniques and obtain optimal solutions for complex, real-world design simulations without requiring deep machine learning skills. This talk will describe IBM BOA, its differentiation and ease of use, and how researchers can take advantage of it for optimizing any arbitrary HPC simulation.

IBM BOA for POWER

IBM BOA for POWER

IBM BOA for POWER

Ganesan Narayanasamy

In this deck, Jem Davies (VP Engineering and ARM Fellow) gives a brief introduction to Machine Learning and explains how it is used in devices such as smartphones, autos, and drones. "I do think that machine learning altogether is probably going to be one of the biggest shifts in computing that we'll see in quite a few years. I'm reluctant to put a number on it like -- the biggest thing in 25 years or whatever," said Jem Davies in a recent investor call. "But this is going to be big. It is going to affect all of us. It affects quite a lot of ARM, in fact." Watch the video presentation: http://insidehpc.com/2017/03/slidecast-arm-steps-machine-learning/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

ARM and Machine Learning

ARM and Machine Learning

ARM and Machine Learning

inside-BigData.com

Support Vector Machines (SVMs) have proven to yield high accuracy and have been used widespread in recent years. However, the standard versions of the SVM algorithm are very time-consuming and computationally intensive; which places a challenge on engineers to explore other hardware architectures than CPU, capable of performing real-time training and classifications while maintaining low power consumption in embedded systems. This paper proposes an overview of works based on the two most popular parallel processing devices: GPU and FPGA, with a focus on multiclass training process. Since diﬀerent techniques have been evaluated using diﬀerent experimentation platforms and methodologies, we only focus on the improvements realized in each study.

Hardware Acceleration of SVM Training for Real-time Embedded Systems: An Over...

Hardware Acceleration of SVM Training for Real-time Embedded Systems: An Over...

Hardware Acceleration of SVM Training for Real-time Embedded Systems: An Over...

MATLAB Small Projects Research Assistance

MATLAB Small Projects Research Assistance

MATLAB Small Projects Research Assistance

Matlab Simulation

SAMOS 2018: LEGaTO: first steps towards energy-efficient toolset for heteroge...

SAMOS 2018: LEGaTO: first steps towards energy-efficient toolset for heteroge...

SAMOS 2018: LEGaTO: first steps towards energy-efficient toolset for heteroge...

Deploying and Managing HPC Clusters with IBM Platform and Intel Xeon Phi Copr...

Deploying and Managing HPC Clusters with IBM Platform and Intel Xeon Phi Copr...

Deploying and Managing HPC Clusters with IBM Platform and Intel Xeon Phi Copr...

Intel IT Center

Shubhankar pawade Resume

Shubhankar pawade Resume

Shubhankar pawade Resume

shubhankar pawade

Vlsi titles 2014 2013 2012 2011

Vlsi titles 2014 2013 2012 2011

Vlsi titles 2014 2013 2012 2011

vamsi_allamsetti

Was ist angesagt? (11)

F07-Cloud-Hadoop-BAM

F07-Cloud-Hadoop-BAM

F07-Cloud-Hadoop-BAM

EuroMPI 2016 Keynote: How Can MPI Fit Into Today's Big Computing

EuroMPI 2016 Keynote: How Can MPI Fit Into Today's Big Computing

EuroMPI 2016 Keynote: How Can MPI Fit Into Today's Big Computing

OpenPOWER Webinar on Machine Learning for Academic Research

OpenPOWER Webinar on Machine Learning for Academic Research

OpenPOWER Webinar on Machine Learning for Academic Research

IBM BOA for POWER

IBM BOA for POWER

IBM BOA for POWER

ARM and Machine Learning

ARM and Machine Learning

ARM and Machine Learning

Hardware Acceleration of SVM Training for Real-time Embedded Systems: An Over...

Hardware Acceleration of SVM Training for Real-time Embedded Systems: An Over...

Hardware Acceleration of SVM Training for Real-time Embedded Systems: An Over...

MATLAB Small Projects Research Assistance

MATLAB Small Projects Research Assistance

MATLAB Small Projects Research Assistance

SAMOS 2018: LEGaTO: first steps towards energy-efficient toolset for heteroge...

SAMOS 2018: LEGaTO: first steps towards energy-efficient toolset for heteroge...

SAMOS 2018: LEGaTO: first steps towards energy-efficient toolset for heteroge...

Deploying and Managing HPC Clusters with IBM Platform and Intel Xeon Phi Copr...

Deploying and Managing HPC Clusters with IBM Platform and Intel Xeon Phi Copr...

Deploying and Managing HPC Clusters with IBM Platform and Intel Xeon Phi Copr...

Shubhankar pawade Resume

Shubhankar pawade Resume

Shubhankar pawade Resume

Vlsi titles 2014 2013 2012 2011

Vlsi titles 2014 2013 2012 2011

Vlsi titles 2014 2013 2012 2011

Andere mochten auch

HC-4022, Towards an Ecosystem for Heterogeneous Parallel Computing, by Wu Feng

HC-4022, Towards an Ecosystem for Heterogeneous Parallel Computing, by Wu Feng

HC-4022, Towards an Ecosystem for Heterogeneous Parallel Computing, by Wu Feng

AMD Developer Central

Parallel Computing Application

Parallel Computing Application

Parallel Computing Application

Casual mass parallel computing

Casual mass parallel computing

Casual mass parallel computing

EU-RESPOND

Engineering

Er. Md. Hasanuzzaman

TechnoXian 2016 robotics championship event series that includes International Robotics League (IRL) and World Robotics Championship (WRC) in affiliation with All India Council for Robotics & Automation (AICRA) and IIT-Delhi (eDC). It will accumulate students from all over the world together to be a part of the biggest world level robotics championship and commemorate with knowledge and create an unmatched ambiance and aura of learning with fun under the expert guidance of research personnel who have proved their worth in the field of academics and industry.

TechnoXian 2016

TechnoXian 2016

TechnoXian 2016

Rajkumar Sharma

Disruptive & Breakthrough innovations alter our world. Some domains of Technology are altering and evolving at a pace that is almost alarming. However, the future is never predictable and a breakthrough technology in a domain can revolutionaries the way the world works and conducts without much warning. The Moore's Law was expected to hit a plateau and now with advent of Quantum computing it has again become relevant and computational speeds may even outpace Moore's Law. The material technologies including nano-science will continue to excite the researchers and Bio-sciences with synergising affects of other domains of science can be predicted to take giant leaps. Artificial Intelligence is probably expected to pervade everything we touch and feel.

How disruptive innovations shall alter our world

How disruptive innovations shall alter our world

How disruptive innovations shall alter our world

Guruprasad Siddalingappa

Daksh remotely operated vehicle (rov), india

Daksh remotely operated vehicle (rov), india

Daksh remotely operated vehicle (rov), india

Fire fighting robot using 8051 microcontroller

Fire fighting robot using 8051 microcontroller

Fire fighting robot using 8051 microcontroller

The Defence Research and Development Organisation (DRDO) is an agency of the Republic of India, charged with the military's research and development, headquartered in New Delhi, India. It was formed in 1958 by the merger of the Technical Development Establishment and the Directorate of Technical Development and Production with the Defence Science Organisation. It is under the administrative control of the Ministry of Defence, Government of India.

D.R.D.O

Introduction to parallel processing

Introduction to parallel processing

Introduction to parallel processing

Applications of paralleL processing

Applications of paralleL processing

Applications of paralleL processing

Parallel computing

Parallel computing

Parallel computing

robotics ppt

Robotics project ppt

Robotics project ppt

Robotics project ppt

Vundavalli Shreya

Hype vs. Reality: The AI Explainer

Hype vs. Reality: The AI Explainer

Hype vs. Reality: The AI Explainer

Andere mochten auch (16)

HC-4022, Towards an Ecosystem for Heterogeneous Parallel Computing, by Wu Feng

HC-4022, Towards an Ecosystem for Heterogeneous Parallel Computing, by Wu Feng

HC-4022, Towards an Ecosystem for Heterogeneous Parallel Computing, by Wu Feng

Parallel Computing Application

Parallel Computing Application

Parallel Computing Application

Casual mass parallel computing

Casual mass parallel computing

Casual mass parallel computing

EU-RESPOND

Engineering

TechnoXian 2016

TechnoXian 2016

TechnoXian 2016

How disruptive innovations shall alter our world

How disruptive innovations shall alter our world

How disruptive innovations shall alter our world

Daksh remotely operated vehicle (rov), india

Daksh remotely operated vehicle (rov), india

Daksh remotely operated vehicle (rov), india

Fire fighting robot using 8051 microcontroller

Fire fighting robot using 8051 microcontroller

Fire fighting robot using 8051 microcontroller

D.R.D.O

Introduction to parallel processing

Introduction to parallel processing

Introduction to parallel processing

Applications of paralleL processing

Applications of paralleL processing

Applications of paralleL processing

Parallel computing

Parallel computing

Parallel computing

robotics ppt

Robotics project ppt

Robotics project ppt

Robotics project ppt

Hype vs. Reality: The AI Explainer

Hype vs. Reality: The AI Explainer

Hype vs. Reality: The AI Explainer

Ähnlich wie Apache Hadoop India Summit 2011 talk "Adaptive Parallel Computing over Distributed Military Computing Infrastructures" by Rituraj Kumar

HPC with Clouds and Cloud Technologies

HPC with Clouds and Cloud Technologies

HPC with Clouds and Cloud Technologies

Inderjeet Singh

Parallel Processing Concepts

Parallel Processing Concepts

Parallel Processing Concepts

Dr Shashikant Athawale

For the full video of this presentation, please visit: https://www.embedded-vision.com/platinum-members/renesas/embedded-vision-training/videos/pages/may-2019-embedded-vision-summit For more information about embedded vision, please visit: http://www.embedded-vision.com Yoshio Sato, Senior Product Marketing Manager in the Industrial Business Unit at Renesas, presents the "Dynamically Reconfigurable Processor Technology for Vision Processing" tutorial at the May 2019 Embedded Vision Summit. The Dynamically Reconfigurable Processing (DRP) block in the Arm Cortex-A9 based RZ/A2M MPU accelerates image processing algorithms with spatially pipelined, time-multiplexed, reconfigurable- hardware compute resources. This hybrid ARM/DRP architecture combines the economy, flexibility and ease-of-use of microprocessors with the high throughput and low latency of performance- optimized hardware. DRP technology achieves silicon area efficiency by dividing large data paths into sub- blocks that can be swapped into the DRP hardware on each clock cycle to accelerate multiple complex algorithms while avoiding the cost and power penalties associated with large FPGAs. Pre-built libraries and a C-language programming environment deliver these benefits without the need for hardware design expertise. Designs can be iteratively enhanced through pre-production and even after mass-market deployment. In this presentation, Sato examines the DRP block’s architecture and operation, presents benchmarks demonstrating performance up to 20x greater than traditional CPUs and introduces resources for developing DRP-based embedded vision systems with the RZ/A2M MPU.

"Dynamically Reconfigurable Processor Technology for Vision Processing," a Pr...

"Dynamically Reconfigurable Processor Technology for Vision Processing," a Pr...

"Dynamically Reconfigurable Processor Technology for Vision Processing," a Pr...

Edge AI and Vision Alliance

In this deck from the 2019 Stanford HPC Conference, Devendar Bureddy from Mellanox presents: SHARP: In-Network Scalable Hierarchical Aggregation and Reduction Protocol. "Increased system size and a greater reliance on utilizing system parallelism to achieve computational needs, requires innovative system architectures to meet the simulation challenges. As a step towards a new network class of co-processors intelligent network devices, which manipulate data traversing the data-center network, SHARP technology designed to ofﬂoad collective operation processing to the network. This tutorial will provide an overview of SHARP technology, integration with MPI, SHARP software components and live example of running MPI collectives. Devendar Bureddy is a Staff Engineer at Mellanox Technologies and has been instrumental in building several key technologies like SHARP, HCOLL, etc. Prior to joining Mellanox, he was a software developer at The Ohio State University in network-Based Computing Laboratory led by Dr. D. K. Panda, involved in the design and development of MVAPICH2, an open-source high-performance implementation of MPI over InfiniBand and 10GigE/iWARP. Devendar received his master’s in Computer Science and Engineering from the Indian Institute of Technology, Kanpur. His research interests include high speed interconnects, parallel programming models and HPC software. Watch the video: https://youtu.be/_EB2Ixy-cNw Learn more: http://www.mellanox.com/page/products_dyn?product_family=261&mtag=sharp and http://hpcadvisorycouncil.com/events/2019/stanford-workshop/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

SHARP: In-Network Scalable Hierarchical Aggregation and Reduction Protocol

SHARP: In-Network Scalable Hierarchical Aggregation and Reduction Protocol

SHARP: In-Network Scalable Hierarchical Aggregation and Reduction Protocol

inside-BigData.com

Programming Models for High-performance Computing

Programming Models for High-performance Computing

Programming Models for High-performance Computing

OpenACC Monthly Highlights: October2020

OpenACC Monthly Highlights: October2020

OpenACC Monthly Highlights: October2020

RAPIDS Overview

RAPIDS Overview

RAPIDS Overview

High performance computing in the cloud is enabling high scale compute- and graphics-intensive workloads across industries, ranging from aerospace, automotive, and manufacturing to life sciences, financial services, and energy. AWS provides application developers and end users with unprecedented computational power for massively parallel applications, in areas such as large-scale fluid and materials simulations, 3D content rendering, financial computing, and deep learning. This session provides an overview of HPC capabilities on AWS, describes the newest generations of accelerated computing instances (including P2), as well as highlighting customer and partner use-cases across industries. Attendees learn about best practices for running HPC workflows in the cloud, including graphical pre- and post-processing, workflow automation, and optimization. Attendees also learn about new and emerging HPC use cases: in particular, deep learning training and inference, large-scale simulations, and high performance data analytics.

AWS re:Invent 2016: High Performance Computing on AWS (CMP207)

AWS re:Invent 2016: High Performance Computing on AWS (CMP207)

AWS re:Invent 2016: High Performance Computing on AWS (CMP207)

Amazon Web Services

STAC Summit 2014 - Building a multitenant Big Data infrastructure

STAC Summit 2014 - Building a multitenant Big Data infrastructure

STAC Summit 2014 - Building a multitenant Big Data infrastructure

This deck was presented by Rich Graham at the 2016 Stanford HPC Conference. "Exascale levels of computing pose many system- and application- level computational challenges. Mellanox Technologies, Inc. as a provider of end-to-end communication services is progressing the foundation of the InfiniBand architecture to meet the exascale challenges. This presentation will focus on recent technology improvements which significantly improve InfiniBand’s scalability, performance, and ease of use." Learn more: http://insidehpc.com/2016-stanford-hpc-conference-video-gallery/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

Interconnect your future

Interconnect your future

Interconnect your future

inside-BigData.com

Qos aware data replication for data-intensive applications in cloud computing...

Qos aware data replication for data-intensive applications in cloud computing...

Qos aware data replication for data-intensive applications in cloud computing...

Papitha Velumani

QoS-Aware Data Replication for Data-Intensive Applications in Cloud Computing...

QoS-Aware Data Replication for Data-Intensive Applications in Cloud Computing...

QoS-Aware Data Replication for Data-Intensive Applications in Cloud Computing...

Papitha Velumani

186 devlin p-poster(2)

186 devlin p-poster(2)

186 devlin p-poster(2)

Paper

Flow Solver: HiFUN

Flow Solver: HiFUN

Flow Solver: HiFUN

Nikhil Vijay Shende

In this deck from the MVAPICH User Group, Gilad Shainer from Mellanox presents: InfiniBand In-Network Computing Technology and Roadmap. "In-Network Computing transforms the data center interconnect to become a "distributed CPU", and "distributed memory", enables to overcome performance barriers and to enable faster and more scalable data analysis. HDR 200G InfiniBand In-Network Computing technology includes several elements - Scalable Hierarchical Aggregation and Reduction Protocol (SHARP), smart Tag Matching and rendezvoused protocol, and more. These technologies are in use at some of the recent large scale supercomputers around the world, including the top TOP500 platforms. The session will discuss the InfiniBand In-Network Computing technology and performance results, as well as view to future roadmap." Watch the video: https://wp.me/p3RLHQ-kIC Learn more: http://mellanox.com and http://mug.mvapich.cse.ohio-state.edu/program/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

InfiniBand In-Network Computing Technology and Roadmap

InfiniBand In-Network Computing Technology and Roadmap

InfiniBand In-Network Computing Technology and Roadmap

inside-BigData.com

Considering new ways and options for reducing operational costs and scaling flexibility of your Apache Hadoop/Spark? Try migrating to Amazon EMR! On-premises Apache Hadoop/Spark clusters are among the top sources of financial pressure for businesses. IT organizations want to reduce spend while still meeting demand, to keep their legacy data applications up and running. Come and learn from experts at Provectus & AWS how you can use Amazon EMR to start driving cost efficiencies in your organization! Agenda - Hadoop market and cost optimizations using Amazon EMR - Cost related and other challenges of on-prem Hadoop clusters - Cost optimizations by using Amazon EMR and migration best practices Intended audience Technology executives & decision makers, manager-level tech roles, data engineers & data scientists, and developers Presenters - Stepan Pushkarev, Chief Technology Officer, Provectus - Pritpal Sahota, Technical Account Manager, Provectus - Nirav Shah, Senior Solutions Architect, AWS - Perry Peterson, Business Development Manager, AWS Feel free to share this presentation with your colleagues and don't hesitate to reach out to us at info@provectus.com if you have any questions! REQUEST WEBINAR: https://provectus.com/cost-optimization-for-apache-hadoop-spark-workloads-with-amazon-emr-june-2020/

Cost Optimization for Apache Hadoop/Spark Workloads with Amazon EMR

Cost Optimization for Apache Hadoop/Spark Workloads with Amazon EMR

Cost Optimization for Apache Hadoop/Spark Workloads with Amazon EMR

Advancements in information technology is contributing to the excessive rate of big data generation recently. Big data refers to datasets that are huge in volume and consumes much time and space to process and transmit using the available resources. Big data also covers data with unstructured and structured formats. Many agencies are currently subscribing to research on big data analytics owing to the failure of the existing data processing techniques to handle the rate at which big data is generated. This paper presents an efficient classification and reduction technique for big data based on parallel generalized Hebbian algorithm (GHA) which is one of the commonly used principal component analysis (PCA) neural network (NN) learning algorithms. The new method proposed in this study was compared to the existing methods to demonstrate its capabilities in reducing the dimensionality of big data. The proposed method in this paper is implemented using Spark Radoop platform.

An effective classification approach for big data with parallel generalized H...

An effective classification approach for big data with parallel generalized H...

An effective classification approach for big data with parallel generalized H...

CFD on Power

Ganesan Narayanasamy

Talk about 4 leading edge projects: 1) Optimal pricing for energy management, online pricing, and truck scheduling @Princeton University 2) Infinite DRAM - RAMCloud @Stanford University Applications: Extremely low latency and very high bandwidth a) Facebook like problems with high read AND write rate, b) advanced analytics, c) what-if scenarios for demand planning 3) Hybrid In-Memory Store @MIT CSAIL 4) Multithreading Real Time Event Platform @MIT Auto-ID Lab 500k events/s and millions of threads in-memory or distributed used for automatic meter reading, online billing, mobile billing and Smart Grid

New Business Applications Powered by In-Memory Technology @MIT Forum for Supp...

New Business Applications Powered by In-Memory Technology @MIT Forum for Supp...

New Business Applications Powered by In-Memory Technology @MIT Forum for Supp...

Ähnlich wie Apache Hadoop India Summit 2011 talk "Adaptive Parallel Computing over Distributed Military Computing Infrastructures" by Rituraj Kumar (20)

HPC with Clouds and Cloud Technologies

HPC with Clouds and Cloud Technologies

HPC with Clouds and Cloud Technologies

Parallel Processing Concepts

Parallel Processing Concepts

Parallel Processing Concepts

"Dynamically Reconfigurable Processor Technology for Vision Processing," a Pr...

"Dynamically Reconfigurable Processor Technology for Vision Processing," a Pr...

"Dynamically Reconfigurable Processor Technology for Vision Processing," a Pr...

SHARP: In-Network Scalable Hierarchical Aggregation and Reduction Protocol

SHARP: In-Network Scalable Hierarchical Aggregation and Reduction Protocol

SHARP: In-Network Scalable Hierarchical Aggregation and Reduction Protocol

Programming Models for High-performance Computing

Programming Models for High-performance Computing

Programming Models for High-performance Computing

OpenACC Monthly Highlights: October2020

OpenACC Monthly Highlights: October2020

OpenACC Monthly Highlights: October2020

RAPIDS Overview

RAPIDS Overview

RAPIDS Overview

AWS re:Invent 2016: High Performance Computing on AWS (CMP207)

AWS re:Invent 2016: High Performance Computing on AWS (CMP207)

AWS re:Invent 2016: High Performance Computing on AWS (CMP207)

STAC Summit 2014 - Building a multitenant Big Data infrastructure

STAC Summit 2014 - Building a multitenant Big Data infrastructure

STAC Summit 2014 - Building a multitenant Big Data infrastructure

Interconnect your future

Interconnect your future

Interconnect your future

Qos aware data replication for data-intensive applications in cloud computing...

Qos aware data replication for data-intensive applications in cloud computing...

Qos aware data replication for data-intensive applications in cloud computing...

QoS-Aware Data Replication for Data-Intensive Applications in Cloud Computing...

QoS-Aware Data Replication for Data-Intensive Applications in Cloud Computing...

QoS-Aware Data Replication for Data-Intensive Applications in Cloud Computing...

186 devlin p-poster(2)

186 devlin p-poster(2)

186 devlin p-poster(2)

Paper

Flow Solver: HiFUN

Flow Solver: HiFUN

Flow Solver: HiFUN

InfiniBand In-Network Computing Technology and Roadmap

InfiniBand In-Network Computing Technology and Roadmap

InfiniBand In-Network Computing Technology and Roadmap

Cost Optimization for Apache Hadoop/Spark Workloads with Amazon EMR

Cost Optimization for Apache Hadoop/Spark Workloads with Amazon EMR

Cost Optimization for Apache Hadoop/Spark Workloads with Amazon EMR

An effective classification approach for big data with parallel generalized H...

An effective classification approach for big data with parallel generalized H...

An effective classification approach for big data with parallel generalized H...

CFD on Power

New Business Applications Powered by In-Memory Technology @MIT Forum for Supp...

New Business Applications Powered by In-Memory Technology @MIT Forum for Supp...

New Business Applications Powered by In-Memory Technology @MIT Forum for Supp...

Mehr von Yahoo Developer Network

Developing Mobile Apps for Performance - Swapnil Patel, Verizon Media

Developing Mobile Apps for Performance - Swapnil Patel, Verizon Media

Developing Mobile Apps for Performance - Swapnil Patel, Verizon Media

Yahoo Developer Network

Athenz - The Open-Source Solution to Provide Access Control in Dynamic Infras...

Athenz - The Open-Source Solution to Provide Access Control in Dynamic Infras...

Athenz - The Open-Source Solution to Provide Access Control in Dynamic Infras...

Yahoo Developer Network

Athenz & SPIFFE, Tatsuya Yano, Yahoo Japan

Athenz & SPIFFE, Tatsuya Yano, Yahoo Japan

Athenz & SPIFFE, Tatsuya Yano, Yahoo Japan

Yahoo Developer Network

Athenz with Istio - Single Access Control Model in Cloud Infrastructures, Tat...

Athenz with Istio - Single Access Control Model in Cloud Infrastructures, Tat...

Athenz with Istio - Single Access Control Model in Cloud Infrastructures, Tat...

Yahoo Developer Network

CICD at Oath using Screwdriver

CICD at Oath using Screwdriver

CICD at Oath using Screwdriver

Yahoo Developer Network

Offline and stream processing of big data sets can be done with tools such as Hadoop, Spark, and Storm, but what if you need to process big data at the time a user is making a request? Vespa (http://www.vespa.ai) allows you to search, organize and evaluate machine-learned models from e.g TensorFlow over large, evolving data sets with latencies in the tens of milliseconds. Vespa is behind the recommendation, ad targeting, and search at Yahoo where it handles billions of daily queries over billions of documents.

Big Data Serving with Vespa - Jon Bratseth, Distinguished Architect, Oath

Big Data Serving with Vespa - Jon Bratseth, Distinguished Architect, Oath

Big Data Serving with Vespa - Jon Bratseth, Distinguished Architect, Oath

Yahoo Developer Network

How @TwitterHadoop Chose Google Cloud, Joep Rottinghuis, Lohit VijayaRenu

How @TwitterHadoop Chose Google Cloud, Joep Rottinghuis, Lohit VijayaRenu

How @TwitterHadoop Chose Google Cloud, Joep Rottinghuis, Lohit VijayaRenu

Yahoo Developer Network

The Future of Hadoop in an AI World, Milind Bhandarkar, CEO, Ampool

The Future of Hadoop in an AI World, Milind Bhandarkar, CEO, Ampool

The Future of Hadoop in an AI World, Milind Bhandarkar, CEO, Ampool

Yahoo Developer Network

Apache YARN Federation and Tez at Microsoft, Anupam Upadhyay, Adrian Nicoara,...

Apache YARN Federation and Tez at Microsoft, Anupam Upadhyay, Adrian Nicoara,...

Apache YARN Federation and Tez at Microsoft, Anupam Upadhyay, Adrian Nicoara,...

Yahoo Developer Network

Containerized Services on Apache Hadoop YARN: Past, Present, and Future, Shan...

Containerized Services on Apache Hadoop YARN: Past, Present, and Future, Shan...

Containerized Services on Apache Hadoop YARN: Past, Present, and Future, Shan...

Yahoo Developer Network

HDFS Scalability and Security, Daryn Sharp, Senior Engineer, Oath

HDFS Scalability and Security, Daryn Sharp, Senior Engineer, Oath

HDFS Scalability and Security, Daryn Sharp, Senior Engineer, Oath

Yahoo Developer Network

Hadoop {Submarine} Project: Running deep learning workloads on YARN, Wangda T...

Hadoop {Submarine} Project: Running deep learning workloads on YARN, Wangda T...

Hadoop {Submarine} Project: Running deep learning workloads on YARN, Wangda T...

Yahoo Developer Network

Moving the Oath Grid to Docker, Eric Badger, Oath

Moving the Oath Grid to Docker, Eric Badger, Oath

Moving the Oath Grid to Docker, Eric Badger, Oath

Yahoo Developer Network

Architecting Petabyte Scale AI Applications

Architecting Petabyte Scale AI Applications

Architecting Petabyte Scale AI Applications

Yahoo Developer Network

Offline and stream processing of big data sets can be done with tools such as Hadoop, Spark, and Storm, but what if you need to process big data at the time a user is making a request? This presentation introduces Vespa (http://vespa.ai) – the open source big data serving engine. Vespa allows you to search, organize, and evaluate machine-learned models from e.g TensorFlow over large, evolving data sets with latencies in the tens of milliseconds. Vespa is behind the recommendation, ad targeting, and search at Yahoo where it handles billions of daily queries over billions of documents and was recently open sourced at http://vespa.ai.

Introduction to Vespa – The Open Source Big Data Serving Engine, Jon Bratseth...

Introduction to Vespa – The Open Source Big Data Serving Engine, Jon Bratseth...

Introduction to Vespa – The Open Source Big Data Serving Engine, Jon Bratseth...

Yahoo Developer Network

In recent times, YARN Capacity Scheduler has improved a lot in terms of some critical features and refactoring. Here is a quick look into some of the recent changes in scheduler: Global Scheduling Support General placement support Better preemption model to handle resource anomalies across and within queue. Absolute resources’ configuration support Priority support between Queues and Applications In this talk, we will deep dive into each of these new features to give a better picture of their usage and performance comparison. We will also provide some more brief overview about the ongoing efforts and how they can help to solve some of the core issues we face today. Speakers: Sunil Govind (Hortonworks), Jian He (Hortonworks)

Jun 2017 HUG: YARN Scheduling – A Step Beyond

Jun 2017 HUG: YARN Scheduling – A Step Beyond

Jun 2017 HUG: YARN Scheduling – A Step Beyond

Yahoo Developer Network

In recent years, Yahoo has brought the big data ecosystem and machine learning together to discover mathematical models for search ranking, online advertising, content recommendation, and mobile applications. We use distributed computing clusters with CPUs and GPUs to train these models from 100’s of petabytes of data. A collection of distributed algorithms have been developed to achieve 10-1000x the scale and speed of alternative solutions. Our algorithms construct regression/classification models and semantic vectors within hours, even for billions of training examples and parameters. We have made our distributed deep learning solutions, CaffeOnSpark and TensorFlowOnSpark, available as open source. In this talk, we highlight Yahoo use cases where big data and machine learning technologies are best exemplified. We explain algorithm/system challenges to scale ML algorithms for massive datasets. We provide a technical overview of CaffeOnSpark and TensorFlowOnSpark to jumpstart your journey of large-scale machine learning. Speakers: Andy Feng is a VP of Architecture at Yahoo, leading the architecture and design of big data and machine learning initiatives. He has architected large-scale systems for personalization, ad serving, NoSQL, and cloud infrastructure. Prior to Yahoo, he was a Chief Architect at Netscape/AOL, and Principal Scientist at Xerox. He received a Ph.D. degree in computer science from Osaka University, Japan.

Jun 2017 HUG: Large-Scale Machine Learning: Use Cases and Technologies

Jun 2017 HUG: Large-Scale Machine Learning: Use Cases and Technologies

Jun 2017 HUG: Large-Scale Machine Learning: Use Cases and Technologies

Yahoo Developer Network

Spark and SQL-on-Hadoop have made it easier than ever for enterprises to create or migrate apps to the big data stack. Thousands of apps are being generated every day in the form of ETL and modeling pipelines, business intelligence and data cubes, deep machine learning, graph analytics, and real-time data streaming. However, the task of reliably operationalizing these big data apps involves many painpoints. Developers may not have the experience in distributed systems to tune apps for efficiency and performance. Diagnosing failures or unpredictable performance of apps can be a laborious process that involves multiple people. Apps may get stuck or steal resources and cause mission-critical apps to miss SLAs. This talk with introduce the audience to these problems and their common causes. We will also demonstrate how to find and fix these problems quickly, as well as prevent such problems from happening in the first place. Speakers: Dr. Shivnath Babu is a Co-founder and CTO of Unravel and Associate Professor of Computer Science at Duke University. With more than a decade of experience researching the ease of use and manageability of data-intensive systems, he leads the Starfish project at Duke, which pioneered the automation of Hadoop application tuning, problem diagnosis, and resource management. Shivnath has more than 80 peer-reviewed publications to his credit and has received the U.S. National Science Foundation CAREER Award, the HP Labs Innovation Award, and three IBM Faculty Awards.

February 2017 HUG: Slow, Stuck, or Runaway Apps? Learn How to Quickly Fix Pro...

February 2017 HUG: Slow, Stuck, or Runaway Apps? Learn How to Quickly Fix Pro...

February 2017 HUG: Slow, Stuck, or Runaway Apps? Learn How to Quickly Fix Pro...

Yahoo Developer Network

Apache Apex (http://apex.apache.org/) is a stream processing platform that helps organizations to build processing pipelines with fault tolerance and strong processing guarantees. It was built to support low processing latency, high throughput, scalability, interoperability, high availability and security. The platform comes with Malhar library - an extensive collection of processing operators and a wide range of input and output connectors for out-of-the-box integration with an existing infrastructure. In the talk I am going to describe how connectors together with the distributed checkpointing (a mechanism used by the Apex to support fault tolerance and high availability) provide exactly-once end-to-end processing guarantees. Speakers: Vlad Rozov is Apache Apex PMC member and back-end engineer at DataTorrent where he focuses on the buffer server, Apex platform network layer, benchmarks and optimizing the core components for low latency and high throughput. Prior to DataTorrent Vlad worked on distributed BI platform at Huawei and on multi-dimensional database (OLAP) at Hyperion Solutions and Oracle.

February 2017 HUG: Exactly-once end-to-end processing with Apache Apex

February 2017 HUG: Exactly-once end-to-end processing with Apache Apex

February 2017 HUG: Exactly-once end-to-end processing with Apache Apex

Yahoo Developer Network

In the analysis of big data there are problematic queries that don’t scale because they require huge compute resources and time to generate exact results. Examples include count distinct, quantiles, most frequent items, joins, matrix computations, and graph analysis. If approximate results are acceptable, there is a class of sub-linear, stochastic streaming algorithms, called "sketches", that can produce results orders-of magnitude faster and with mathematically proven error bounds. For interactive queries there may not be other viable alternatives, and in the case of extracting results for these problem queries in real-time, sketches are the only known solution. For any analysis system that requires these problematic queries from big data, sketches are a required toolkit that should be tightly integrated into the system's analysis capabilities. This technology has helped Yahoo successfully reduce data processing times from days to hours, or minutes to seconds on a number of its internal platforms. This talk covers the current state of our Open Source DataSketches.github.io library, which includes adaptations and example code for Pig, Hive, Spark and Druid and gives architectural examples of use and a case study. Speakers: Jon Malkin is a scientist at Yahoo working to extend the DataSketches library. His previous roles have involved large scale data processing for sponsored search, display advertising, user counting, ad targeting, and cross-device user identity modeling. Alexander Saydakov is a senior software engineer at Yahoo working on the open source Data Sketches project. In his previous roles he has been involved in building large-scale back-end data processing systems and frameworks for data analytics and experimentation based on Torque, Hadoop, Pig, Hive and Druid. Alexander’s education background is in the field of applied mathematics.

February 2017 HUG: Data Sketches: A required toolkit for Big Data Analytics

February 2017 HUG: Data Sketches: A required toolkit for Big Data Analytics

February 2017 HUG: Data Sketches: A required toolkit for Big Data Analytics

Yahoo Developer Network

Mehr von Yahoo Developer Network (20)

Developing Mobile Apps for Performance - Swapnil Patel, Verizon Media

Developing Mobile Apps for Performance - Swapnil Patel, Verizon Media

Developing Mobile Apps for Performance - Swapnil Patel, Verizon Media

Athenz - The Open-Source Solution to Provide Access Control in Dynamic Infras...

Athenz - The Open-Source Solution to Provide Access Control in Dynamic Infras...

Athenz - The Open-Source Solution to Provide Access Control in Dynamic Infras...

Athenz & SPIFFE, Tatsuya Yano, Yahoo Japan

Athenz & SPIFFE, Tatsuya Yano, Yahoo Japan

Athenz & SPIFFE, Tatsuya Yano, Yahoo Japan

Athenz with Istio - Single Access Control Model in Cloud Infrastructures, Tat...

Athenz with Istio - Single Access Control Model in Cloud Infrastructures, Tat...

Athenz with Istio - Single Access Control Model in Cloud Infrastructures, Tat...

CICD at Oath using Screwdriver

CICD at Oath using Screwdriver

CICD at Oath using Screwdriver

Big Data Serving with Vespa - Jon Bratseth, Distinguished Architect, Oath

Big Data Serving with Vespa - Jon Bratseth, Distinguished Architect, Oath

Big Data Serving with Vespa - Jon Bratseth, Distinguished Architect, Oath

How @TwitterHadoop Chose Google Cloud, Joep Rottinghuis, Lohit VijayaRenu

How @TwitterHadoop Chose Google Cloud, Joep Rottinghuis, Lohit VijayaRenu

How @TwitterHadoop Chose Google Cloud, Joep Rottinghuis, Lohit VijayaRenu

The Future of Hadoop in an AI World, Milind Bhandarkar, CEO, Ampool

The Future of Hadoop in an AI World, Milind Bhandarkar, CEO, Ampool

The Future of Hadoop in an AI World, Milind Bhandarkar, CEO, Ampool

Apache YARN Federation and Tez at Microsoft, Anupam Upadhyay, Adrian Nicoara,...

Apache YARN Federation and Tez at Microsoft, Anupam Upadhyay, Adrian Nicoara,...

Apache YARN Federation and Tez at Microsoft, Anupam Upadhyay, Adrian Nicoara,...

Containerized Services on Apache Hadoop YARN: Past, Present, and Future, Shan...

Containerized Services on Apache Hadoop YARN: Past, Present, and Future, Shan...

Containerized Services on Apache Hadoop YARN: Past, Present, and Future, Shan...

HDFS Scalability and Security, Daryn Sharp, Senior Engineer, Oath

HDFS Scalability and Security, Daryn Sharp, Senior Engineer, Oath

HDFS Scalability and Security, Daryn Sharp, Senior Engineer, Oath

Hadoop {Submarine} Project: Running deep learning workloads on YARN, Wangda T...

Hadoop {Submarine} Project: Running deep learning workloads on YARN, Wangda T...

Hadoop {Submarine} Project: Running deep learning workloads on YARN, Wangda T...

Moving the Oath Grid to Docker, Eric Badger, Oath

Moving the Oath Grid to Docker, Eric Badger, Oath

Moving the Oath Grid to Docker, Eric Badger, Oath

Architecting Petabyte Scale AI Applications

Architecting Petabyte Scale AI Applications

Architecting Petabyte Scale AI Applications

Introduction to Vespa – The Open Source Big Data Serving Engine, Jon Bratseth...

Introduction to Vespa – The Open Source Big Data Serving Engine, Jon Bratseth...

Introduction to Vespa – The Open Source Big Data Serving Engine, Jon Bratseth...

Jun 2017 HUG: YARN Scheduling – A Step Beyond

Jun 2017 HUG: YARN Scheduling – A Step Beyond

Jun 2017 HUG: YARN Scheduling – A Step Beyond

Jun 2017 HUG: Large-Scale Machine Learning: Use Cases and Technologies

Jun 2017 HUG: Large-Scale Machine Learning: Use Cases and Technologies

Jun 2017 HUG: Large-Scale Machine Learning: Use Cases and Technologies

February 2017 HUG: Slow, Stuck, or Runaway Apps? Learn How to Quickly Fix Pro...

February 2017 HUG: Slow, Stuck, or Runaway Apps? Learn How to Quickly Fix Pro...

February 2017 HUG: Slow, Stuck, or Runaway Apps? Learn How to Quickly Fix Pro...

February 2017 HUG: Exactly-once end-to-end processing with Apache Apex

February 2017 HUG: Exactly-once end-to-end processing with Apache Apex

February 2017 HUG: Exactly-once end-to-end processing with Apache Apex

February 2017 HUG: Data Sketches: A required toolkit for Big Data Analytics

February 2017 HUG: Data Sketches: A required toolkit for Big Data Analytics

February 2017 HUG: Data Sketches: A required toolkit for Big Data Analytics

Apache Hadoop India Summit 2011 talk "Adaptive Parallel Computing over Distributed Military Computing Infrastructures" by Rituraj Kumar

1. Adaptive parallel computing over distributed military computing infrastructures RiturajKumar Center of Artificial Intelligence and Robotics – DRDO Lab

2.

3. Adaptive Parallel Computing

4. Approaches for APC

7.

8. Parallel Computing is the only feasible solution that would guarantee reduced response time.

9. Tactical deployment would not amenable for –

10. Large Clusters in the field

11. Back-hauling

12.

13. 5 Adaptive Parallel Computing Amdahls Law The Amdahl's law is concerned with the speedup achievable from an improvement to a computation that affects a proportion P of that computation where the improvement has a speedup of S. Where, S = Speedup P = Parallel fraction of Program N = Number of Processors If 95% of the program can be parallelized, the theoretical maximum speedup using parallel computing would be 20×, no matter how many processors are used.

14. 6 Adaptive Parallel Computing Cost for parallel Computation = Computation Cost + Serialization Cost

15.

16.

17.

18.

19. Used extensively in varieties of information systems.

21.

22. Excellent fault tolerance capabilities

23.

24.

25. Dynamic topology poses great challenge for computing infrastructure.

26. MPI framework has few disadvantages

27.