AWS re:Invent 2016: Taking Data to the Extreme (MBL202)

•

3 likes•791 views

As GoPro expands into content networks and launches new products, new challenges have appeared. One of the most critical challenges facing GoPro during this period of rapid growth is their ability to make effective use of massive amounts of data. Every day, GoPro collects increasing amounts of data generated by internet connected consumer devices (smart cameras, smart drones), GoPro mobile apps, GoPro content networks, GoPro e-commerce sales, and social media. This data ranges from raw camera logs to refined and well-structured e-commerce datasets. In the past, it took GoPro months to understand new inbound data and determine how to transform or augment it for analysis. To streamline this process and bridge the gap between tech-savvy engineers and data-savvy analysts, GoPro is creating an analysis loop, which informs product usage trends and product insights. This analysis loop serves a large ecosystem of GoPro executives, product managers, engineers, data scientists, and business analysts through an integrated technology pipeline consisting of Apache Kafka, Apache Spark Streaming, Cloudera’s distribution of Hadoop, and Tableau’s Data Visualization Software as the end user analytical tool. Session sponsored by Tableau Software.

Technology

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Gaurav Kumar, Christian Lam, David Winters
November 30, 2016
Taking Data to the Extreme
MBL202

David Winters
Big Data Architect,
Data Science &
Engineering, GoPro
Gaurav Kumar
Product Lead,
Data Science &
Engineering, GoPro
Christian Lam
Analytics Engineer,
Data Science &
Engineering, GoPro

Origin Story
•Make Friends
•Haul Ass
•Maintain Balance
•No Half-Assery
•Integrity. Always
•Be a HERO
Yes, this comes from the top…

High-Level Architecture
ETL Cluster
• Aggregations and
Joins
• Hive
• Map/Reduce
Secure Data
Mart Cluster
• End User Query
• Impala/Sentry
• Parquet
Analytics Apps
•Hue
•Tableau
•Python
•R
Streaming Ingest
Cluster
•Log file streaming
•RESTful service
•Kafka
•Spark Streaming
•HBase
Batch Induction
Framework
• Batch files
• Scheduled
downloads
• Pre-processing
• Java App
Original
Cluster
JSON
JSON
Parquet
DDL

Data Pipeline
Streaming Ingest Cluster
ELBHTTP
Pipeline for processing of streaming logs
To ETL Cluster

Data Pipeline
/path1/…
/path2/…
/path3/…
ToETL
Cluster
/path4/…

Data Pipeline
ETL Cluster
HDFS
Hive Metastore
To SDM Cluster
From Streaming
Ingest Cluster
Batch
Induction
Framework

Data Delivery!
HDFS
Hive Metastore
Applications
Thrift
ODBC
Server
User
Studio
Studio - Staging
GDA
Report
SDM Cluster
From ETL Cluster

Areas for Improvement
• Isolation of workloads
• Fast ingest
• Secure
• Fast delivery/queries
• Loosely coupled clusters
• Multiple copies of data
• Tightly coupled storage and compute
• Lack of elasticity
• Operational overhead of multiple clusters

Amazon S3
bucket
Future Architecture
Streaming
Ingest Cluster
Batch
Induction
Framework
Hive
Metastore
Ephemeral
ETL
Cluster
JSON
Parquet
+
DDL
Aggregates
Events
+
State Ephemeral
Data Mart
Cluster #1
Ephemeral
Data Mart
Cluster #2
Ephemeral
Data Mart
Cluster #N

Visualization Categories
3 categories for all visualizations…
OperationsPre ETL Analytics

Data Ops
Operations
Data ops dashboards
allow us to monitor
the health of data
streams and detect
anomalies, as well as
Tableau Server it self.
Operations

PRE BUILT AGGREGATIONS
• Aggregates are
important for the
successful adoption
of Tableau
• Example: Karma
flight table
Operations

FEED PRODUCT
INSIGHTS
• Product insights can
result in product design
changes
• How we observed
Camera as A Hub
upload behaviors
Analytics

Reporting
Analytics
CAMERA
CONNECTS…
• Help us with
figuring out
stolen/smuggled
cameras
• Where we
should put our
new marketing
dollars for new
products

DASHBOARDS (GoPro Plus SUBSCRIPTIONS)
• Real-time
subscriber growth
• Utilizes Hive
External Tables
and JSON SERDE
Analytics

PG #
RC Playbook: Your guide to
success at GoPro
Questions?

What's hot

You wouldn't use a scooter to help someone move; likewise, there is no one-size-fits-all data storage solution. AWS provides a wide variety of storage services to address the spectrum of needs, from casual users saving photos to mission-critical, specialized databases utilized at the largest private and public sector entities. This session will give you an overview of these storage offerings, provide you with the groundwork to match these to your use cases. Learn More: https://aws.amazon.com/government-education/

Demystifying Storage on AWS | AWS Public Sector Summit 2017

Amazon Web Services

Startups around the world use AWS services to access the power of the cloud to grow faster and more cost effectively. In this session, Smartsheet talks about how they were able to cost-effectively build their prototype for scale and avoid replatforming at different points in the adoption curve, and Quantcast discusses how they are running a high-performance analytics solution on AWS. They provide several tips and tricks for S3, and show how they removed a traditional MySQL data store from a distributed-image hosting application so that the only required data store is S3. They also show how to avoid common, cumbersome database practices by working with the eventually consistent nature of S3 objects and the fact that objects and directories share the same namespace.

AWS re:Invent 2016: Case Study: How Startups Like Smartsheet and Quantcast Ac...

Amazon Web Services

Organizations need to gain insight and knowledge from a growing number of Internet of Things (IoT), application programming interfaces (API), clickstreams, unstructured and log data sources. However, organizations are also often limited by legacy data warehouses and ETL processes that were designed for transactional data. Building scalable big data pipelines with automated extract-transform-load (ETL) and machine learning processes can address these limitations. JustGiving is the world’s largest social platform for online giving. In this session, we describe how we created several scalable and loosely coupled event-driven ETL and ML pipelines as part of our in-house data science platform called RAVEN. You learn how to leverage AWS Lambda, Amazon S3, Amazon EMR, Amazon Kinesis, and other services to build serverless, event-driven, data and stream processing pipelines in your organization. We review common design patterns, lessons learned, and best practices, with a focus on serverless big data architectures with AWS Lambda.

AWS re:Invent 2016: JustGiving: Serverless Data Pipelines, Event-Driven ETL, ...

Amazon Web Services

Billions of Rows Transformed in Record Time Using Matillion ETL for Amazon Redshift GE Power & Water develops advanced technologies to help solve some of the world’s most complex challenges related to water availability and quality. They had amassed billions of rows of data on on-premises databases, but decided to migrate some of their core big data projects to the AWS Cloud. When they decided to transform and store it all in Amazon Redshift, they knew they needed an ETL/ELT tool that could handle this enormous amount of data and safely deliver it to its destination. In this session, Ryan Oates, Enterprise Architect at GE Water, shares his use case, requirements, outcomes and lessons learned. He also shares the details of his solution stack, including Amazon Redshift and Matillion ETL for Amazon Redshift in AWS Marketplace. You learn best practices on Amazon Redshift ETL supporting enterprise analytics and big data requirements, simply and at scale. You learn how to simplify data loading, transformation and orchestration on to Amazon Redshift and how build out a real data pipeline. Get the insights to deliver your big data project in record time.

AWS re:Invent 2016: Billions of Rows Transformed in Record Time Using Matilli...

Amazon Web Services

During this session Greg Brandt and Liyin Tang, Data Infrastructure engineers from Airbnb, will discuss the design and architecture of Airbnb's streaming ETL infrastructure, which exports data from RDS for MySQL and DynamoDB into Airbnb's data warehouse, using a system called SpinalTap. We will also discuss how we leverage Spark Streaming to compute derived data from tracking topics and/or database tables, and HBase to provide immediate data access and generate cleanly time-partitioned Hive tables.

AWS re:Invent 2016: Streaming ETL for RDS and DynamoDB (DAT315)

Amazon Web Services

As you begin to move out of your data center and develop a cloud-first strategy, you'll need support for large-scale migrations to AWS. In this session, CSC shares details about the journey to AWS by some of our largest enterprise customers. We provide best practices for planning your large-scale migrations and focus on business processes in addition to technology. We show how CSC used this approach to migrate to AWS as part of our separation last year into two publicly traded companies: CSC and CSRA. In less than six months, CSC took our 56-year-old company and broke it into two companies, one of which was brand new and without any infrastructure or enterprise applications. We explain how we leveraged the AWS Partner ecosystem to achieve this incredible IT challenge. Session sponsored by CSC. AWS Competency Partner

AWS re:Invent 2016: Large-scale AWS Migrations (ENT204)

Amazon Web Services

DigitalGlobe, Inc., the world’s leading provider of high-resolution Earth imagery, data, and analysis, is migrating its IT infrastructure, supporting imagery production and storage as well as satellite flight operations, to AWS with plans to close its commercial data centers within four years. DigitalGlobe has utilized AWS Snowmobile to move its 100PB image archive to the cloud. DigitalGlobe built its Geospatial Big Data platform, GBDX, natively on AWS. GBDX utilizes the image archive and combines geospatial big data and analytic tools, partner and customer data and tools, and dynamic cloud compute all in one place. This session will explore cost optimization for data management on AWS, highlighting various storage tiers and data import opportunities. We will focus on cost optimal usage of S3, S3-IA, Glacier, Snowball Edge and Snowmobile – balancing imagery access time with storage costs. Hear how DigitalGlobe utilized some of the newest features of the AWS platform to minimize their costs from storage. Learn More: https://aws.amazon.com/government-education/

Optimizing Data Management Using AWS Storage and Data Migration Products | AW...

Amazon Web Services

Amazon QuickSight is a fast BI service that makes it easy for you to build visualizations, perform ad-hoc analysis, and quickly get business insights from your data. QuickSight is built to harness the power and scalability of the cloud, so you can easily run analysis on large datasets, and support hundreds of thousands of users. In this session, we’ll demonstrate how you can easily get started with Amazon QuickSight, uploading files, connecting to S3 and Redshift and creating analyses from visualizations that are optimized based on the underlying data. Once we’ve built our analysis and dashboard, we’ll show you easy it is to share it with colleagues and stakeholders in just a few seconds. And with SPICE – QuckSight’s in-memory calculation engine – you can go from data to insights, faster than ever.

AWS re:Invent 2016: Visualizing Big Data Insights with Amazon QuickSight (BDM...

Amazon Web Services

Toyota Racing Development (TRD) developed a robust and highly performant real-time data analysis tool for professional racing. In this talk, learn how we structured a reliable, maintainable, decoupled architecture built around Amazon DynamoDB as both a streaming mechanism and a long-term persistent data store. In racing, milliseconds matter and even moments of downtime can cost a race. You'll see how we used DynamoDB together with Amazon Kinesis and Kinesis Firehose to build a real-time streaming data analysis tool for competitive racing.

AWS re:Invent 2016: How Toyota Racing Development Makes Racing Decisions in R...

Amazon Web Services

Amazon Redshift is a fast, simple, cost-effective data warehousing solution, and in this session, we look at the tools and techniques you can use to migrate your existing data warehouse to Amazon Redshift. We will then present a case study on Scholastic’s migration to Amazon Redshift. Scholastic, a large 100-year-old publishing company, was running their business with older, on-premise, data warehousing and analytics solutions, which could not keep up with business needs and were expensive. Scholastic also needed to include new capabilities like streaming data and real time analytics. Scholastic migrated to Amazon Redshift, and achieved agility and faster time to insight while dramatically reducing costs. In this session, Scholastic will discuss how they achieved this, including options considered, technical architecture implemented, results, and lessons learned.

AWS re:Invent 2016: Migrating Your Data Warehouse to Amazon Redshift (DAT202)

Amazon Web Services

Amazon RDS allows you to launch an optimally configured, secure and highly available database with just a few clicks. It provides cost-efficient and resizable capacity, automates time-consuming database administration tasks, and provides you with six familiar database engines to choose from: Amazon Aurora, Oracle, Microsoft SQL Server, PostgreSQL, MySQL and MariaDB. In this session, we will take a close look at the capabilities of Amazon RDS and explain how it works. We’ll also discuss the AWS Database Migration Service and AWS Schema Conversion Tool, which help you migrate databases and data warehouses with minimal downtime from on-premises and cloud environments to Amazon RDS and other Amazon services. Gain your freedom from expensive, proprietary databases while providing your applications with the fast performance, scalability, high availability, and compatibility they need. AWS Speaker: Andrew Kane, Solutions Architect - Amazon Web Services

Migrating Your Databases to AWS: Deep Dive on Amazon RDS and AWS Database Mig...

Amazon Web Services

Ansible is a simple, but powerful automation tool with an agentless footprint that allows for the definition of architecture, intent, and policy as code that can be deployed across both on-prem and cloud infrastructure. This enables customers to extend their enterprise and applications into AWS in a way that maintains a consistent, secure posture as part of a continuous delivery pipeline. Customers can then natively integrate with AWS to seamlessly configure and deploy a range of AWS services such as Amazon Aurora, Amazon Redshift, Amazon EMR, Amazon Athena, Amazon CloudFront, Amazon Route 53, and Elastic Load Balancing from within Red Hat OpenShift across a secure, consistent hybrid cloud infrastructure. In this session, we will demonstrate how infrastructure can be instantiated with code as part of a continuous delivery pipeline and describe how that integrates with an OpenShift hybrid cloud deployment. Learn More: https://aws.amazon.com/government-education/

Integrating Infrastructure as Code into a Continuous Delivery Pipeline | AWS ...

Amazon Web Services

Originally, Hadoop was used as a batch analytics tool; however, this is rapidly changing, as applications move towards real-time processing and streaming. Amazon Elastic MapReduce has made running Hadoop in the cloud easier and more accessible than ever. Each day, tens of thousands of Hadoop clusters are run on the Amazon Elastic MapReduce infrastructure by users of every size — from university students to Fortune 50 companies. We recently launched Amazon Kinesis – a managed service for real-time processing of high volume, streaming data. Amazon Kinesis enables a new class of big data applications which can continuously analyze data at any volume and throughput, in real-time. Adi will discuss each service, dive into how customers are adopting the services for different use cases, and share emerging best practices. Learn how you can architect Amazon Kinesis and Amazon Elastic MapReduce together to create a highly scalable real-time analytics solution which can ingest and process terabytes of data per hour from hundreds of thousands of different concurrent sources. Forever change how you process web site click-streams, marketing and financial transactions, social media feeds, logs and metering data, and location-tracking events.

Real-time Streaming and Querying with Amazon Kinesis and Amazon Elastic MapRe...

Amazon Web Services

Introduction to the AWS Cloud – Russell Hall

Amazon Web Services

Applications running in a typical data center are static entities. Dynamic scaling and resource allocation are the norm in AWS. Technologies such as Amazon EC2, Docker, AWS Lambda, and Auto Scaling make tracking resources and resource utilization a challenge. The days of static server monitoring are over. In this session, we examine trends we’ve observed across thousands of customers using dynamic resource allocation and discuss why dynamic infrastructure fundamentally changes your monitoring strategy. We discuss some of the best practices we’ve learned by working with New Relic customers to build, manage, and troubleshoot applications and dynamic cloud services. Session sponsored by New Relic. AWS Competency Partner

AWS re:Invent 2016: Cloud Monitoring - Understanding, Preparing, and Troubles...

Amazon Web Services

Now that you have earmarked workloads for migration, it's time to look at the various tools and methodologies that are available to help customers shift applications to AWS. This session highlights some of the key AWS tools, services and approaches that organisations are using to successfully migrate to the cloud. AWS Speaker: Sven Hansen, Solution Architect - Amazon Web Services Customer Speaker: Pieter Breed – Core Platform Engineer Zoona

Migration Recipes for Success - AWS Summit Cape Town 2017

Amazon Web Services

AWS re:Invent 2016: Storage State of the Union (STG201)

Amazon Web Services

How can you use Big Data to grow your business and discover new opportunities? When organizations effectively capture, analyze, visualize and apply big data insights to their business goals, they differentiate themselves from their competitors and outperform them in terms of operational efficiency and the bottom line. With Amazon Web Services, businesses and researchers can easily fulfill their high performance computing (HPC) requirements with the added benefit of ad-hoc provisioning, pay-as-you-go pricing and faster time-to-results. Join this session to understand how to run HPC applications in AWS cloud, and about different AWS Big Data and Analytics services such as Amazon Elastic MapReduce (Hadoop), Amazon Redshift (Data Warehouse) and Amazon Kinesis (Streaming), when to use them and how they work together.

Getting Started with Big Data and HPC in the Cloud - August 2015

Amazon Web Services

How Can I Plan for Security, Risk, & Compliance Before Migrating to AWS? | A...

Amazon Web Services

Building Serverless Web Applications - DevDay Los Angeles 2017

Amazon Web Services

What's hot (20)

Demystifying Storage on AWS | AWS Public Sector Summit 2017

AWS re:Invent 2016: Case Study: How Startups Like Smartsheet and Quantcast Ac...

AWS re:Invent 2016: JustGiving: Serverless Data Pipelines, Event-Driven ETL, ...

AWS re:Invent 2016: Billions of Rows Transformed in Record Time Using Matilli...

AWS re:Invent 2016: Streaming ETL for RDS and DynamoDB (DAT315)

AWS re:Invent 2016: Large-scale AWS Migrations (ENT204)

Optimizing Data Management Using AWS Storage and Data Migration Products | AW...

AWS re:Invent 2016: Visualizing Big Data Insights with Amazon QuickSight (BDM...

AWS re:Invent 2016: How Toyota Racing Development Makes Racing Decisions in R...

AWS re:Invent 2016: Migrating Your Data Warehouse to Amazon Redshift (DAT202)

Migrating Your Databases to AWS: Deep Dive on Amazon RDS and AWS Database Mig...

Integrating Infrastructure as Code into a Continuous Delivery Pipeline | AWS ...

Real-time Streaming and Querying with Amazon Kinesis and Amazon Elastic MapRe...

Introduction to the AWS Cloud – Russell Hall

AWS re:Invent 2016: Cloud Monitoring - Understanding, Preparing, and Troubles...

Migration Recipes for Success - AWS Summit Cape Town 2017

AWS re:Invent 2016: Storage State of the Union (STG201)

Getting Started with Big Data and HPC in the Cloud - August 2015

How Can I Plan for Security, Risk, & Compliance Before Migrating to AWS? | A...

Building Serverless Web Applications - DevDay Los Angeles 2017

Viewers also liked

This first session in a series of six ‘Packed Lunch’ webinars provides an overview of Data Virtualization technology, its applications and how it is adding business value to organizations around the world. More information and FREE registrations to this webinar: http://goo.gl/z7mq2S Landing page for the entire Packed Lunch webinar series: http://goo.gl/NATMHw Attend & get unique insights into: What Data Virtualization is and what sets it apart from traditional integration tools How it both complements and leverages existing enterprise architectures The Denodo Data Virtualization platform and its capabilities

Introduction to Data Virtualization (session 1 from Packed Lunch Webinar Series)

Denodo

Data Virtualization Journey: How to Grow from Single Project and to Enterpris...

Denodo

Watch the full session: Denodo DataFest 2016 sessions: https://goo.gl/DOrhiA Connected use cases are gaining momentum! Data integration is the foundation for enabling these connections. In this session, you will experience first-hand our customer case studies and implementation architectures of IoT solutions. In this session, you will learn: • The role of data virtualization in enabling IoT use cases • How our customers have successfully implemented IoT solutions using data virtualization • How our product complements other IoT technologies This session is part of the Denodo DataFest 2016 event. You can also watch more Denodo DataFest sessions on demand here: https://goo.gl/VXb6M6

Denodo DataFest 2016: The Role of Data Virtualization in IoT Integration

Denodo

DataOps with Project Amaterasu

DataWorks Summit/Hadoop Summit

Workshop : Wild Rydes Takes Off - The Dawn of a New Unicorn

Amazon Web Services

Introduction to Amazon EC2 Spot Instances

Amazon Web Services

Data integration is paramount, in this presentation you will find three different paradigms: using client-side tools, creating traditional data warehouses and the data virtualization solution - the logical data warehouse, comparing each other and positioning data virtualization as an integral part of any future-proof IT infrastructure. This presentation is part of the Fast Data Strategy Conference, and you can watch the video here goo.gl/1q94Ka.

Data Integration Alternatives: When to use Data Virtualization, ETL, and ESB

Denodo

Enterprises rely on messaging to integrate services and applications and to exchange information critical to running their business. However, managing and operating dedicated message-oriented middleware and underlying infrastructure creates costly overhead and can compromise reliability. In this session, enterprise architects and developers learn how to improve scalability, availability, and operational efficiency by migrating on-premises messaging middleware to a managed cloud service using Amazon SQS. Hear how Capital One is using SQS to migrate several core banking applications to the cloud to ensure high availability and cost efficiency. We also share some exciting new SQS features that allow even more workloads to take advantage of the cloud.

AWS re:Invent 2016: Migrating Enterprise Messaging to the Cloud (ENT217)

Amazon Web Services

Strata+hadoop data kitchen-seven-steps-to-high-velocity-data-analytics-with d...

DataKitchen

In 2004, approximately 400 billion fax pages were sent. Twelve years later, it’s about 4% of that number. The pace of technological change is rapid, but most devices live in the field for 10 to 15 years. It’s hard to maintain competitive value in the face of constant technology improvement, but IoT is changing that. We’ll examine the architectures that allows AWS IoT customers like Pitney Bowes to connect devices to the cloud and enrich the client experience though personalized analytics and recommendations, automated supplies replenishment, and just-in-time self-service.

AWS re:Invent 2016: Innovation After Installation: Establishing a Digital Rel...

Amazon Web Services

“Attribution" is the marketing term of art for allocating full or partial credit to individual advertisements that eventually lead to a purchase, sign up, download, or other desired consumer interaction. We'll share how we use DynamoDB at the core of our attribution system to store terabytes of advertising history data. The system is cost effective and dynamically scales from 0 to 300K requests per second on demand with predictable performance and low operational overhead.

AWS re:Invent 2016: How DataXu scaled its Attribution System to handle billio...

Amazon Web Services

As customers build and run production microservices architectures based on containers, having powerful tools to manage the placement and scheduling of these workloads is critical. Amazon ECS allows customers to focus on building their application and removes the need for managing the cluster management software entirely. This session will focus on the capabilities of the new ECS task placement engine, the built-in ECS schedulers, and explore how customers can build custom schedulers using the ECS event stream and Blox, a newly announced open source scheduling framework that gives developers additional choice and control in how they manage containers in production.

NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox

Amazon Web Services

High performance computing in the cloud is enabling high scale compute- and graphics-intensive workloads across industries, ranging from aerospace, automotive, and manufacturing to life sciences, financial services, and energy. AWS provides application developers and end users with unprecedented computational power for massively parallel applications, in areas such as large-scale fluid and materials simulations, 3D content rendering, financial computing, and deep learning. This session provides an overview of HPC capabilities on AWS, describes the newest generations of accelerated computing instances (including P2), as well as highlighting customer and partner use-cases across industries. Attendees learn about best practices for running HPC workflows in the cloud, including graphical pre- and post-processing, workflow automation, and optimization. Attendees also learn about new and emerging HPC use cases: in particular, deep learning training and inference, large-scale simulations, and high performance data analytics.

AWS re:Invent 2016: High Performance Computing on AWS (CMP207)

Amazon Web Services

Neural networks have a long and rich history in automatic speech recognition. In this talk, we present a brief primer on the origin of deep learning in spoken language, and then explore today’s world of Alexa. Alexa is the AWS service that understands spoken language and powers Amazon Echo. Alexa relies heavily on machine learning and deep neural networks for speech recognition, text-to-speech, language understanding, and more. We also discuss the Alexa Skills Kit, which lets any developer teach Alexa new skills.

AWS re:Invent 2016: Deep Learning in Alexa (MAC202)

Amazon Web Services

What's New in Amazon Aurora

Amazon Web Services

Review ways to manage the lifecycle of your Dev, test, and production infrastructure using CloudFormation. Learn how to architect your infrastructure through loosely coupled stacks using cross-stack references, tightly coupled nested stacks and other best practices. Learn how to use CloudFormation to provision and manage a continuous deployment pipeline for your infrastructure-as-code. Automate deployment of new development environments as your infrastructure evolves, promote your new architecture for testing, and deploy changes to production.

Infrastructure Continuous Delivery Using AWS CloudFormation

Amazon Web Services

Microservices architectures are changing the way that organizations build their applications and infrastructure. Companies can now achieve new levels of scale and efficiency by disaggregating their large, monolithic applications into small, independent “micro services”, each of which perform different functions. In this session, we’ll introduce the concept of microservices, help you evaluate whether your organization is ready for microservices, and discuss methods for implementing these architectures.

Introduction to Microservices

Amazon Web Services

Organizations face significant challenges moving their applications to the cloud when they require a standard file system interface for accessing their cloud data. In this technical session, we will explore the world’s first cloud-scale file system and its targeted use cases. Attendees will learn about the Amazon Elastic File System (EFS) features and benefits, how to identify applications that are appropriate for use with Amazon EFS, and details about its performance and security models. We will highlight and demonstrate how to deploy Amazon EFS in one of our most common use cases and will share tips for success throughout. Learning Objectives: • Recognize why and when to use Amazon EFS • Understand key technical/security concepts • Learn how to leverage EFS’s performance • See a demo of EFS in action • Review EFS’s economics

Deep Dive on Elastic File System - February 2017 AWS Online Tech Talks

Amazon Web Services

Building Distributed Applications with AWS Step Functions

Amazon Web Services

AWS re:Invent 2016: Big Data Mini Con State of the Union (BDM205)

Amazon Web Services

Viewers also liked (20)

Introduction to Data Virtualization (session 1 from Packed Lunch Webinar Series)

Data Virtualization Journey: How to Grow from Single Project and to Enterpris...

Denodo DataFest 2016: The Role of Data Virtualization in IoT Integration

DataOps with Project Amaterasu

Workshop : Wild Rydes Takes Off - The Dawn of a New Unicorn

Introduction to Amazon EC2 Spot Instances

Data Integration Alternatives: When to use Data Virtualization, ETL, and ESB

AWS re:Invent 2016: Migrating Enterprise Messaging to the Cloud (ENT217)

Strata+hadoop data kitchen-seven-steps-to-high-velocity-data-analytics-with d...

AWS re:Invent 2016: Innovation After Installation: Establishing a Digital Rel...

AWS re:Invent 2016: How DataXu scaled its Attribution System to handle billio...

NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox

AWS re:Invent 2016: High Performance Computing on AWS (CMP207)

AWS re:Invent 2016: Deep Learning in Alexa (MAC202)

What's New in Amazon Aurora

Infrastructure Continuous Delivery Using AWS CloudFormation

Introduction to Microservices

Deep Dive on Elastic File System - February 2017 AWS Online Tech Talks

Building Distributed Applications with AWS Step Functions

AWS re:Invent 2016: Big Data Mini Con State of the Union (BDM205)

Similar to AWS re:Invent 2016: Taking Data to the Extreme (MBL202)

What’s important about a technology is what you can use it to do. I’ve looked at what a number of groups are doing with Apache Hadoop and NoSQL in production, and I will relay what worked well for them and what did not. Drawing from real world use cases, I show how people who understand these new approaches can employ them well in conjunction with traditional approaches and existing applications. Thread Detection, Datawarehouse optimization, Marketing Efficiency, Biometric Database are some examples exposed during this presentation.

Tugdual Grall - Real World Use Cases: Hadoop and NoSQL in Production

Codemotion

Prestogres, ODBC & JDBC connectivity for Presto

Sadayuki Furuhashi

The communities for Apache Atlas and Apache Ranger, which are foundational components for Security and Governance across the Hadoop stack, have spawned a robust industry ecosystem of tools and platforms. Such industry solutions build upon the extensibility offered via open and robust APIs and integration patterns to provide innovative “better together” capabilities. In this talk, we will showcase how the ecosystem of solutions being built by different vendors provide value-added capabilities to address the key aspects of securing and governing your data lakes based on Apache Ranger and Apache Atlas frameworks. The talk will showcase multiple ecosystem demonstrations that will include how to identify, map, and classify personal data, harvest and maintain metadata, track and map the movement of data through your enterprise, and enforce appropriate controls to monitor access and usage of personal data. Come hear from community partners: -Balaji Ganesan from Privacera will showcase how Privacera integrates with and leverages Apache Ranger and Apache Atlas features to help with GDPR compliance -Greg Goldsmith and Jordan Martz from Attunity will showcase how Attunity’s solutions integrate into Apache Atlas to provide robust chain of custody and classifications required for GDPR -Somil Kulkarni from IBM will demonstrate how IBM Information Governance Catalog integrates with Apache Atlas to exchange metadata to build a connected solution for GDPR compliance that harnesses both open source community enhancements and IBM’s innovations in governance space. Speakers Ali Bajwa, Principal Solutions Engineer, Hortonworks Srikanth Venkat, Senior Director Product Management, Hortonworks

GDPR Community Showcase for Apache Ranger and Apache Atlas

DataWorks Summit

How do you go from a strictly typed object-based streaming pipeline with simple operations to a structured streaming pipeline with higher order complex relational operations? This is what the Data Engineering team did at GoPro to scale up the development of streaming pipelines for the rapidly growing number of devices and applications. When big data frameworks such as Hadoop first came to exist, developers were happy because we could finally process large amounts of data without writing complex multi-threaded code or worse yet writing complicated distributed code. Unfortunately, only very simple operations were available such as map and reduce. Almost immediately, higher level operations were desired similar to relational operations. And so Hive and dozens (hundreds?) of SQL-based big data tools became available for more developer-efficient batch processing of massive amounts of data. In recent years, big data has moved from batch processing to stream-based processing since no one wants to wait hours or days to gain insights. Dozens of stream processing frameworks exist today and the same trend that occurred in the batch-based big data processing realm has taken place in the streaming world, so that nearly every streaming framework now supports higher level relational operations. In this talk, we will discuss in a very hands-on manner how the streaming data pipelines for GoPro devices and apps have moved from the original Spark streaming with its simple RDD-based operations in Spark 1.x to Spark's structured streaming with its higher level relational operations in Spark 2.x. We will talk about the differences, advantages, and necessary pain points that must be addressed in order to scale relational-based streaming pipelines for massive IoT streams. We will also talk about moving from “hand built” Hadoop/Spark clusters running in the cloud to using a Spark-based cloud service. DAVID WINTERS, Big Data Architect, GoPro and HAO ZOU, Senior Software Engineer, GoPro

Adding structure to your streaming pipelines: moving from Spark streaming to ...

DataWorks Summit

Trend Micro Big Data Platform and Apache Bigtop

Evans Ye

Building Data Warehouses and Data Lakes in the Cloud - DevDay Austin 2017 Day 2

Amazon Web Services

HDInsight for Architects

Ashish Thapliyal

Challenges in Data Analytics: Different application scenarios need different storage solutions: HBASE is ideal for point query scenarios but unsuitable for multi-dimensional queries. MPP is suitable for data warehouse scenarios but engine and data are coupled together which hampers scalability. OLAP stores used in BI applications perform best for Aggregate queries but full scan queries perform at a sub-optimal performance. Moreover, they are not suitable for real-time analysis. These distinct systems lead to low resource sharing and need different pipelines for data and application management.

Apache CarbonData+Spark to realize data convergence and Unified high performa...

Tech Triveni

Progressive Insurance is well known for its innovative use of data to better serve its customers, and the important role that Hortonworks Data Platform has played in that transformation. However, as with most things worth doing, the path to the Data Lake was not without its challenges. In this session, I’ll share our top use cases for Hadoop – including telematics and display ads, how a skills shortage turned supporting these applications into a nightmare, and how – and why – we now use Syncsort DMX-h to accelerate enterprise adoption by making it quick and easy (or faster and easier) to populate the data lake – and keep it up to date – with data from across the enterprise. I’ll discuss the different approaches we tried, the benefits of using a tool vs. open source, and how we created our Hadoop Ingestor app using Syncsort DMX-h.

Journey to the Data Lake: How Progressive Paved a Faster, Smoother Path to In...

DataWorks Summit

Big data tools such as Hadoop and Spark allow you to process data at unprecedented scale, but keeping your processing engine fed can be a challenge. Metadata in upstream sources can ‘drift’ due to infrastructure, OS and application changes, causing ETL tools and hand-coded solutions to fail. StreamSets Data Collector (SDC) is an Apache 2.0 licensed open source platform for building big data ingest pipelines that allows you to design, execute and monitor robust data flows. In this session we’ll look at how SDC’s “intent-driven” approach keeps the data flowing, with a particular focus on clustered deployment with Spark and other exciting Spark integrations in the works.

Building Data Pipelines with Spark and StreamSets

Pat Patterson

Talk 1 : Evolution of the GoPro's data platform In this talk, we will share GoPro’s experiences in building Data Analytics Cluster in Cloud. We will discuss: evolution of data platform from fixed-size Hadoop clusters to Cloud-based Spark Cluster with Centralized Hive Metastore +S3: Cost Benefits and DevOp Impact; Configurable, spark-based batch Ingestion/ETL framework; Migration Streaming framework to Cloud + S3; Analytics metrics delivery with Slack integration; BedRock: Data Platform Management, Visualization & Self-Service Portal Visualizing Machine learning Features via Google Facets + Spark Speakers: Chester Chen Chester Chen is the Head of Data Science & Engineering, GoPro. Previously, he was the Director of Engineering at Alpine Data Lab. David Winters David is an Architect in the Data Science and Engineering team at GoPro and the creator of their Spark-Kafka data ingestion pipeline. Previously He worked at Apple & Splice Machines. Hao Zou Hao is a Senior big data engineer at Data Science and Engineering team. Previously He worked as Alpine Data Labs and Pivotal

Sf big analytics_2018_04_18: Evolution of the GoPro's data platform

Chester Chen

Intelligent Integration OOW2017 - Jeff Pollock

Jeffrey T. Pollock

Stream processing on mobile networks

pbelko82

Big data and lynda_Subash_DSouza.com

Data Con LA

Apache Druid ingests and enables instant query on many billions of events in real-time. But how? In this talk, each of the components of an Apache Druid cluster is described – along with the data and query optimisations at its core – that unlock fresh, fast data for all. Bio: Peter Marshall (https://linkedin.com/in/amillionbytes/) leads outreach and engineering across Europe for Imply (http://imply.io/), a company founded by the original developers of Apache Druid. He has 20 years architecture experience in CRM, EDRM, ERP, EIP, Digital Services, Security, BI, Analytics, and MDM. He is TOGAF certified and has a BA (hons) degree in Theology and Computer Studies from the University of Birmingham in the United Kingdom.

Druid: Under the Covers (Virtual Meetup)

Imply

Azure Cafe Marketplace with Hortonworks March 31 2016

Joan Novino

Big Data Day LA 2016/ Use Case Driven track - Hydrator: Open Source, Code-Fre...

Data Con LA

Innovation Track AWS Cloud Experience Argentina - Data Lakes & Analytics en AWS

Amazon Web Services LATAM

2024 Feb AI Meetup NYC GenAI_LLMs_ML_Data Codeless Generative AI Pipelines https://www.aicamp.ai/event/eventdetails/W2024022214 apache nifi llm generative ai gen ai ml dl machine learning apache kafka apache flink postgresql python AI Meetup (NYC): GenAI, LLMs, ML and Data Feb 22, 05:30 PM EST Welcome to the monthly in-person AI meetup in New York City, in collaboration with Microsoft. Join us for deep dive tech talks on AI, GenAI, LLMs and machine learning, food/drink, networking with speakers and fellow developers Agenda: * 5:30pm~6:00pm: Checkin, Food/drink and networking * 6:00pm~6:10pm: Welcome/community update * 6:10pm~8:30pm: Tech talks * 8:30pm: Q&A, Open discussion Tech Talk: Searching and Reasoning Over Multimedia Data with Vector Databases and LMMs Speaker: Zain Hasan (Weaviate LinkedIn) Abstract: In this talk, Zain Hasan will discuss how we can use open-source multimodal embedding models in conjunction with large generative multimodal models that can that can see, hear, read, and feel data(!), to perform cross-modal search(searching audio with images, videos with text etc.) and multimodal retrieval augmented generation (MM-RAG) at the billion-object scale with the help of open source vector databases. I will also demonstrate, with live code demos, how being able to perform this cross-modal retrieval in real-time can enables users to use LLMs that can reason over their enterprise multimodal data. This talk will revolve around how we can scale the usage of multimodal embedding and generative models in production. Tech Talk: Codeless Generative AI Pipelines Speaker: Timothy Spann (Cloudera LinkedIn) Abstract: Join us for an insightful talk on leveraging the power of real-time streaming tools, specifically Apache NiFi, to revolutionize GenAI data engineering. In this session, we’ll explore how the integration of Apache NiFi can automate the entire process of prompt building, making it a seamless and efficient task. Speakers/Topics: Stay tuned as we are updating speakers and schedules. If you have a keen interest in speaking to our community, we invite you to submit topics for consideration: Submit Topics Sponsors: We are actively seeking sponsors to support our community. Whether it is by offering venue spaces, providing food/drink, or cash sponsorship. Sponsors will have the chance to speak at the meetups, receive prominent recognition, and gain exposure to our extensive membership base of 20,000+ local or 300K+ developers worldwide. Venue: Microsoft NYC - Times Square, 11 Times Square, New York, NY 10036 Room Name: Central Park West 6501 Community on Slack/Discord - Event chat: chat and connect with speakers and attendees - Sharing blogs, events, job openings, projects collaborations Join Slack (search and join the #newyork channel) | Join Discord

2024 Feb AI Meetup NYC GenAI_LLMs_ML_Data Codeless Generative AI Pipelines

Timothy Spann

Build Deep Learning Applications for Big Data Platforms (CVPR 2018 tutorial)

Jason Dai

Similar to AWS re:Invent 2016: Taking Data to the Extreme (MBL202) (20)

Tugdual Grall - Real World Use Cases: Hadoop and NoSQL in Production

Prestogres, ODBC & JDBC connectivity for Presto

GDPR Community Showcase for Apache Ranger and Apache Atlas

Adding structure to your streaming pipelines: moving from Spark streaming to ...

Trend Micro Big Data Platform and Apache Bigtop

Building Data Warehouses and Data Lakes in the Cloud - DevDay Austin 2017 Day 2

HDInsight for Architects

Apache CarbonData+Spark to realize data convergence and Unified high performa...

Journey to the Data Lake: How Progressive Paved a Faster, Smoother Path to In...

Building Data Pipelines with Spark and StreamSets

Sf big analytics_2018_04_18: Evolution of the GoPro's data platform

Intelligent Integration OOW2017 - Jeff Pollock

Stream processing on mobile networks

Big data and lynda_Subash_DSouza.com

Druid: Under the Covers (Virtual Meetup)

Azure Cafe Marketplace with Hortonworks March 31 2016

Big Data Day LA 2016/ Use Case Driven track - Hydrator: Open Source, Code-Fre...

Innovation Track AWS Cloud Experience Argentina - Data Lakes & Analytics en AWS

2024 Feb AI Meetup NYC GenAI_LLMs_ML_Data Codeless Generative AI Pipelines

Build Deep Learning Applications for Big Data Platforms (CVPR 2018 tutorial)

More from Amazon Web Services

Il Forecasting è un processo importante per tantissime aziende e viene utilizzato in vari ambiti per cercare di prevedere in modo accurato la crescita e distribuzione di un prodotto, l’utilizzo delle risorse necessarie nelle linee produttive, presentazioni finanziarie e tanto altro. Amazon utilizza delle tecniche avanzate di forecasting, in parte questi servizi sono stati messi a disposizione di tutti i clienti AWS. In questa sessione illustreremo come pre-processare i dati che contengono una componente temporale e successivamente utilizzare un algoritmo che a partire dal tipo di dato analizzato produce un forecasting accurato.

Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...

Amazon Web Services

La varietà e la quantità di dati che si crea ogni giorno accelera sempre più velocemente e rappresenta una opportunità irripetibile per innovare e creare nuove startup. Tuttavia gestire grandi quantità di dati può apparire complesso: creare cluster Big Data su larga scala sembra essere un investimento accessibile solo ad aziende consolidate. Ma l’elasticità del Cloud e, in particolare, i servizi Serverless ci permettono di rompere questi limiti. Vediamo quindi come è possibile sviluppare applicazioni Big Data rapidamente, senza preoccuparci dell’infrastruttura, ma dedicando tutte le risorse allo sviluppo delle nostre le nostre idee per creare prodotti innovativi.

Big Data per le Startup: come creare applicazioni Big Data in modalità Server...

Amazon Web Services

Ora puoi utilizzare Amazon Elastic Kubernetes Service (EKS) per eseguire pod Kubernetes su AWS Fargate, il motore di elaborazione serverless creato per container su AWS. Questo rende più semplice che mai costruire ed eseguire le tue applicazioni Kubernetes nel cloud AWS.In questa sessione presenteremo le caratteristiche principali del servizio e come distribuire la tua applicazione in pochi passaggi

Esegui pod serverless con Amazon EKS e AWS Fargate

Amazon Web Services

Vent'anni fa Amazon ha attraversato una trasformazione radicale con l'obiettivo di aumentare il ritmo dell'innovazione. In questo periodo abbiamo imparato come cambiare il nostro approccio allo sviluppo delle applicazioni ci ha permesso di aumentare notevolmente l'agilità, la velocità di rilascio e, in definitiva, ci ha consentito di creare applicazioni più affidabili e scalabili. In questa sessione illustreremo come definiamo le applicazioni moderne e come la creazione di app moderne influisce non solo sull'architettura dell'applicazione, ma sulla struttura organizzativa, sulle pipeline di rilascio dello sviluppo e persino sul modello operativo. Descriveremo anche approcci comuni alla modernizzazione, compreso l'approccio utilizzato dalla stessa Amazon.com.

Costruire Applicazioni Moderne con AWS

Amazon Web Services

L’utilizzo dei container è in continua crescita. Se correttamente disegnate, le applicazioni basate su Container sono molto spesso stateless e flessibili. I servizi AWS ECS, EKS e Kubernetes su EC2 possono sfruttare le istanze Spot, portando ad un risparmio medio del 70% rispetto alle istanze On Demand. In questa sessione scopriremo insieme quali sono le caratteristiche delle istanze Spot e come possono essere utilizzate facilmente su AWS. Impareremo inoltre come Spreaker sfrutta le istanze spot per eseguire applicazioni di diverso tipo, in produzione, ad una frazione del costo on-demand!

Come spendere fino al 90% in meno con i container e le istanze spot

Amazon Web Services

In recent months, many customers have been asking us the question – how to monetise Open APIs, simplify Fintech integrations and accelerate adoption of various Open Banking business models. Therefore, AWS and FinConecta would like to invite you to Open Finance marketplace presentation on October 20th. Event Agenda : Open banking so far (short recap) • PSD2, OB UK, OB Australia, OB LATAM, OB Israel Intro to Open Finance marketplace • Scope • Features • Tech overview and Demo The role of the Cloud The Future of APIs • Complying with regulation • Monetizing data / APIs • Business models • Time to market One platform for all: a Strategic approach Q&A

Open banking as a service

Amazon Web Services

Per creare valore e costruire una propria offerta differenziante e riconoscibile, le startup di successo sanno come combinare tecnologie consolidate con componenti innovativi creati ad hoc. AWS fornisce servizi pronti all'utilizzo e, allo stesso tempo, permette di personalizzare e creare gli elementi differenzianti della propria offerta. Concentrandoci sulle tecnologie di Machine Learning, vedremo come selezionare i servizi di intelligenza artificiale offerti da AWS e, anche attraverso una demo, come costruire modelli di Machine Learning personalizzati utilizzando SageMaker Studio.

Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...

Amazon Web Services

Con l'approccio tradizionale al mondo IT per molti anni è stato difficile implementare tecniche di DevOps, che finora spesso hanno previsto attività manuali portando di tanto in tanto a dei downtime degli applicativi interrompendo l'operatività dell'utente. Con l'avvento del cloud, le tecniche di DevOps sono ormai a portata di tutti a basso costo per qualsiasi genere di workload, garantendo maggiore affidabilità del sistema e risultando in dei significativi miglioramenti della business continuity. AWS mette a disposizione AWS OpsWork come strumento di Configuration Management che mira ad automatizzare e semplificare la gestione e i deployment delle istanze EC2 per mezzo di workload Chef e Puppet. Scopri come sfruttare AWS OpsWork a garanzia e affidabilità del tuo applicativo installato su Instanze EC2.

OpsWorks Configuration Management: automatizza la gestione e i deployment del...

Amazon Web Services

Vuoi conoscere le opzioni per eseguire Microsoft Active Directory su AWS? Quando si spostano carichi di lavoro Microsoft in AWS, è importante considerare come distribuire Microsoft Active Directory per supportare la gestione, l'autenticazione e l'autorizzazione dei criteri di gruppo. In questa sessione, discuteremo le opzioni per la distribuzione di Microsoft Active Directory su AWS, incluso AWS Directory Service per Microsoft Active Directory e la distribuzione di Active Directory su Windows su Amazon Elastic Compute Cloud (Amazon EC2). Trattiamo argomenti quali l'integrazione del tuo ambiente Microsoft Active Directory locale nel cloud e l'utilizzo di applicazioni SaaS, come Office 365, con AWS Single Sign-On.

Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads

Amazon Web Services

Computer Vision con AWS

Amazon Web Services

Amazon Web Services e VMware organizzano un evento virtuale gratuito il prossimo mercoledì 14 Ottobre dalle 12:00 alle 13:00 dedicato a VMware Cloud ™ on AWS, il servizio on demand che consente di eseguire applicazioni in ambienti cloud basati su VMware vSphere® e di accedere ad una vasta gamma di servizi AWS, sfruttando a pieno le potenzialità del cloud AWS e tutelando gli investimenti VMware esistenti. Molte organizzazioni sfruttano i vantaggi del cloud migrando i propri carichi di lavoro Oracle e assicurandosi notevoli vantaggi in termini di agilità ed efficienza dei costi. La migrazione di questi carichi di lavoro, può creare complessità durante la modernizzazione e il refactoring delle applicazioni e a questo si possono aggiungere rischi di prestazione che possono essere introdotti quando si spostano le applicazioni dai data center locali.

Database Oracle e VMware Cloud on AWS i miti da sfatare

Amazon Web Services

Molte aziende oggi, costruiscono applicazioni con funzionalità di tipo ledger ad esempio per verificare lo storico di accrediti o addebiti nelle transazioni bancarie o ancora per tenere traccia del flusso supply chain dei propri prodotti. Alla base di queste soluzioni ci sono i database ledger che permettono di avere un log delle transazioni trasparente, immutabile e crittograficamente verificabile, ma sono strumenti complessi e onerosi da gestire. Amazon QLDB elimina la necessità di costruire sistemi personalizzati e complessi fornendo un database ledger serverless completamente gestito. In questa sessione scopriremo come realizzare un'applicazione serverless completa che utilizzi le funzionalità di QLDB.

Crea la tua prima serverless ledger-based app con QLDB e NodeJS

Amazon Web Services

Con l’ascesa delle architetture di microservizi e delle ricche applicazioni mobili e Web, le API sono più importanti che mai per offrire agli utenti finali una user experience eccezionale. In questa sessione impareremo come affrontare le moderne sfide di progettazione delle API con GraphQL, un linguaggio di query API open source utilizzato da Facebook, Amazon e altro e come utilizzare AWS AppSync, un servizio GraphQL serverless gestito su AWS. Approfondiremo diversi scenari, comprendendo come AppSync può aiutare a risolvere questi casi d’uso creando API moderne con funzionalità di aggiornamento dati in tempo reale e offline. Inoltre, impareremo come Sky Italia utilizza AWS AppSync per fornire aggiornamenti sportivi in tempo reale agli utenti del proprio portale web.

API moderne real-time per applicazioni mobili e web

Amazon Web Services

Molte organizzazioni sfruttano i vantaggi del cloud migrando i propri carichi di lavoro Oracle e assicurandosi notevoli vantaggi in termini di agilità ed efficienza dei costi. La migrazione di questi carichi di lavoro, può creare complessità durante la modernizzazione e il refactoring delle applicazioni e a questo si possono aggiungere rischi di prestazione che possono essere introdotti quando si spostano le applicazioni dai data center locali. In queste slide, gli esperti AWS e VMware presentano semplici e pratici accorgimenti per facilitare e semplificare la migrazione dei carichi di lavoro Oracle accelerando la trasformazione verso il cloud, approfondiranno l’architettura e dimostreranno come sfruttare a pieno le potenzialità di VMware Cloud ™ on AWS.

Database Oracle e VMware Cloud™ on AWS: i miti da sfatare

Amazon Web Services

Tools for building your MVP on AWS

Amazon Web Services

How to Build a Winning Pitch Deck

Amazon Web Services

Building a web application without servers

Amazon Web Services

Fundraising Essentials

Amazon Web Services

AWS_HK_StartupDay_Building Interactive websites while automating for efficien...

Amazon Web Services

Amazon Elastic Container Service (Amazon ECS) è un servizio di gestione dei container altamente scalabile, che semplifica la gestione dei contenitori Docker attraverso un layer di orchestrazione per il controllo del deployment e del relativo lifecycle. In questa sessione presenteremo le principali caratteristiche del servizio, le architetture di riferimento per i differenti carichi di lavoro e i semplici passi necessari per poter velocemente migrare uno o più dei tuo container.

Introduzione a Amazon Elastic Container Service

Amazon Web Services

More from Amazon Web Services (20)

Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...

Big Data per le Startup: come creare applicazioni Big Data in modalità Server...

Esegui pod serverless con Amazon EKS e AWS Fargate

Costruire Applicazioni Moderne con AWS

Come spendere fino al 90% in meno con i container e le istanze spot

Open banking as a service

Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...

OpsWorks Configuration Management: automatizza la gestione e i deployment del...

Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads

Computer Vision con AWS

Database Oracle e VMware Cloud on AWS i miti da sfatare

Crea la tua prima serverless ledger-based app con QLDB e NodeJS

API moderne real-time per applicazioni mobili e web

Database Oracle e VMware Cloud™ on AWS: i miti da sfatare

Tools for building your MVP on AWS

How to Build a Winning Pitch Deck

Building a web application without servers

Fundraising Essentials

AWS_HK_StartupDay_Building Interactive websites while automating for efficien...

Introduzione a Amazon Elastic Container Service

Recently uploaded

In this session, we will delve into strategic approaches for optimizing knowledge management within Microsoft 365, amidst the evolving landscape of Copilot. From leveraging automatic metadata classification and permission governance with SharePoint Premium, to unlocking Viva Engage for the cultivation of knowledge and communities, you will gain actionable insights to bolster your organization's knowledge-sharing initiatives. In this session, we will also explore how to facilitate solutions to enable your employees to find answers and expertise within Microsoft 365. You will leave equipped with practical techniques and a deeper understanding of how there is more to effective knowledge management than just enabling Copilot, but building actual solutions to prepare the knowledge that Copilot and your employees can use.

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Drew Madelung

BooK Now Call us at +918448380779 to hire a gorgeous and seductive call girl for sex. Take a Delhi Escort Service. The help of our escort agency is mostly meant for men who want sexual Indian Escorts In Delhi NCR. It should be noted that any impersonator will get 100 attention from our Young Girls Escorts in Delhi. They will assume the position of reliable allies. VIP Call Girl With Original Photos Book Tonight +918448380779 Our Cheap Price 1 Hour not available 2 Hours 5000 Full Night 8000 TAG: Call Girls in Delhi, Noida, Gurgaon, Ghaziabad, Connaught Place, Greater Kailash Delhi, Lajpat Nagar Delhi, Mayur Vihar Delhi, Chanakyapuri Delhi, New Friends Colony Delhi, Majnu Ka Tilla, Karol Bagh, Malviya Nagar, Saket, Khan Market, Noida Sector 18, Noida Sector 76, Noida Sector 51, Gurgaon Mg Road, Iffco Chowk Gurgaon, Rajiv Chowk Gurgaon All Delhi Ncr Free Home Deliver

08448380779 Call Girls In Diplomatic Enclave Women Seeking Men

Delhi Call girls

Tech Trends Report 2024 Future Today Institute.pdf

hans926745

What is a good lead in your organisation? Which leads are priority? What happens to leads? When sales and marketing give different answers to these questions, or perhaps aren't sure of the answers at all, frustrations build and opportunities are left on the table. Join us for an illuminating session with Cian McLoughlin, HubSpot Principal Customer Success Manager, as we look at that crucial piece of the customer journey in which leads are transferred from marketing to sales.

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

HampshireHUG

GenAI Risks & Security Meetup 01052024.pdf

lior mazor

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Rafal Los

As privacy and data protection regulations evolve rapidly, organizations operating in multiple jurisdictions face mounting challenges to ensure compliance and safeguard customer data. With state-specific privacy laws coming up in multiple states this year, it is essential to understand what their unique data protection regulations will require clearly. How will data privacy evolve in the US in 2024? How to stay compliant? Our panellists will guide you through the intricacies of these states' specific data privacy laws, clarifying complex legal frameworks and compliance requirements. This webinar will review: - The essential aspects of each state's privacy landscape and the latest updates - Common compliance challenges faced by organizations operating in multiple states and best practices to achieve regulatory adherence - Valuable insights into potential changes to existing regulations and prepare your organization for the evolving landscape

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc

A Domino Admins Adventures (Engage 2024)

Gabriella Davis

Automating Google Workspace (GWS) & more with Apps Script

wesley chun

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

What are drone anti-jamming systems? The drone anti-jamming systems and anti-spoof technology protect against interference, jamming, and spoofing of the UAVs. To protect their security, countries are beginning to research drone anti-jamming systems, also known as drone strike weapons. The anti-jam and anti-spoof technology protects against interference, jamming and spoofing. A drone strike weapon is a drone attack weapon that can attack and destroy enemy drones. So what is so unique about this amazing system?

What Are The Drone Anti-jamming Systems Technology?

Antenna Manufacturer Coco

Abhishek Deb(1), Mr Abdul Kalam(2) M. Des (UX) , School of Design, DIT University , Dehradun. This paper explores the future potential of AI-enabled smartphone processors, aiming to investigate the advancements, capabilities, and implications of integrating artificial intelligence (AI) into smartphone technology. The research study goals consist of evaluating the development of AI in mobile phone processors, analyzing the existing state as well as abilities of AI-enabled cpus determining future patterns as well as chances together with reviewing obstacles as well as factors to consider for more growth.

Exploring the Future Potential of AI-Enabled Smartphone Processors

debabhi2

CNv6 Instructor Chapter 6 Quality of Service

giselly40

Imagine a world where information flows as swiftly as thought itself, making decision-making as fluid as the data driving it. Every moment is critical, and the right tools can significantly boost your organization’s performance. The power of real-time data automation through FME can turn this vision into reality. Aimed at professionals eager to leverage real-time data for enhanced decision-making and efficiency, this webinar will cover the essentials of real-time data and its significance. We’ll explore: FME’s role in real-time event processing, from data intake and analysis to transformation and reporting An overview of leveraging streams vs. automations FME’s impact across various industries highlighted by real-life case studies Live demonstrations on setting up FME workflows for real-time data Practical advice on getting started, best practices, and tips for effective implementation Join us to enhance your skills in real-time data automation with FME, and take your operational capabilities to the next level.

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Safe Software

With more memory available, system performance of three Dell devices increased, which can translate to a better user experience Conclusion When your system has plenty of RAM to meet your needs, you can efficiently access the applications and data you need to finish projects and to-do lists without sacrificing time and focus. Our test results show that with more memory available, three Dell PCs delivered better performance and took less time to complete the Procyon Office Productivity benchmark. These advantages translate to users being able to complete workflows more quickly and multitask more easily. Whether you need the mobility of the Latitude 5440, the creative capabilities of the Precision 3470, or the high performance of the OptiPlex Tower Plus 7010, configuring your system with more RAM can help keep processes running smoothly, enabling you to do more without compromising performance.

Boost PC performance: How more available memory can improve productivity

Principled Technologies

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

sammart93

Scaling API-first – The story of a global engineering organization

Radu Cotescu

Recently uploaded (20)

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

08448380779 Call Girls In Diplomatic Enclave Women Seeking Men

Tech Trends Report 2024 Future Today Institute.pdf

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

GenAI Risks & Security Meetup 01052024.pdf

Strategies for Landing an Oracle DBA Job as a Fresher

The 7 Things I Know About Cyber Security After 25 Years | April 2024

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

A Domino Admins Adventures (Engage 2024)

Automating Google Workspace (GWS) & more with Apps Script

How to Troubleshoot Apps for the Modern Connected Worker

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

What Are The Drone Anti-jamming Systems Technology?

Exploring the Future Potential of AI-Enabled Smartphone Processors

CNv6 Instructor Chapter 6 Quality of Service

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Boost PC performance: How more available memory can improve productivity

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Scaling API-first – The story of a global engineering organization

AWS re:Invent 2016: Taking Data to the Extreme (MBL202)

3. David Winters Big Data Architect, Data Science & Engineering, GoPro Gaurav Kumar Product Lead, Data Science & Engineering, GoPro Christian Lam Analytics Engineer, Data Science & Engineering, GoPro

6. Growing data needs

8.  

9.  

10. Origin Story •Make Friends •Haul Ass •Maintain Balance •No Half-Assery •Integrity. Always •Be a HERO Yes, this comes from the top…

11.

12. High-Level Architecture ETL Cluster • Aggregations and Joins • Hive • Map/Reduce Secure Data Mart Cluster • End User Query • Impala/Sentry • Parquet Analytics Apps •Hue •Tableau •Python •R Streaming Ingest Cluster •Log file streaming •RESTful service •Kafka •Spark Streaming •HBase Batch Induction Framework • Batch files • Scheduled downloads • Pre-processing • Java App Original Cluster JSON JSON Parquet DDL

13. Data Pipeline Streaming Ingest Cluster ELBHTTP Pipeline for processing of streaming logs To ETL Cluster

14. Data Pipeline /path1/… /path2/… /path3/… ToETL Cluster /path4/…

15. Data Pipeline ETL Cluster HDFS Hive Metastore To SDM Cluster From Streaming Ingest Cluster Batch Induction Framework

16. Data Delivery! HDFS Hive Metastore Applications Thrift ODBC Server User Studio Studio - Staging GDA Report SDM Cluster From ETL Cluster

17. Areas for Improvement • Isolation of workloads • Fast ingest • Secure • Fast delivery/queries • Loosely coupled clusters • Multiple copies of data • Tightly coupled storage and compute • Lack of elasticity • Operational overhead of multiple clusters

18. Amazon S3 bucket Future Architecture Streaming Ingest Cluster Batch Induction Framework Hive Metastore Ephemeral ETL Cluster JSON Parquet + DDL Aggregates Events + State Ephemeral Data Mart Cluster #1 Ephemeral Data Mart Cluster #2 Ephemeral Data Mart Cluster #N

19.

20. Visualization Categories 3 categories for all visualizations… OperationsPre ETL Analytics

21. 21 Operations

22. Data Ops Operations Data ops dashboards allow us to monitor the health of data streams and detect anomalies, as well as Tableau Server it self. Operations

23. 23

24. PRE BUILT AGGREGATIONS • Aggregates are important for the successful adoption of Tableau • Example: Karma flight table Operations

25. FEED PRODUCT INSIGHTS • Product insights can result in product design changes • How we observed Camera as A Hub upload behaviors Analytics

26. Reporting Analytics CAMERA CONNECTS… • Help us with figuring out stolen/smuggled cameras • Where we should put our new marketing dollars for new products

27. DASHBOARDS (GoPro Plus SUBSCRIPTIONS) • Real-time subscriber growth • Utilizes Hive External Tables and JSON SERDE Analytics

28.

29. External Tables Pre ETL

30.

31. PG # RC Playbook: Your guide to success at GoPro Questions?

32. Thank you!

33. Remember to complete your evaluations!

AWS re:Invent 2016: Taking Data to the Extreme (MBL202)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to AWS re:Invent 2016: Taking Data to the Extreme (MBL202)

Similar to AWS re:Invent 2016: Taking Data to the Extreme (MBL202) (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Recently uploaded

Recently uploaded (20)

AWS re:Invent 2016: Taking Data to the Extreme (MBL202)