Couchbase Meetup Jan 2016

•Als PPTX, PDF herunterladen•

3 gefällt mir•991 views

LinkedIn’s Big Data Pipeline with Kafka, Hadoop and Couchbase

Michael Kehoe
Senior Site Reliability Engineer
LinkedIn
LinkedIn’s Big Data Pipeline
with Kafka, Hadoop and
Couchbase

3
$ whoami
Michael Kehoe
• Sr Site Reliability Engineer
(SRE)
• Member of CBVT
• B.E. (Electrical Engineering)
from
the University of Queensland,
Australia

4
Kafka @ LinkedIn
• Kafka was created by LinkedIn
• Kafka is a publish-subscribe
system as a distributed commit
log
• Processes 500+ TB/ day (~500
billion messages)

5
LinkedIn’s use of Kafka
• Monitoring
• Pub-Sub Messaging
• Analytics
• Building block for (log) distributed application
• Samza
• Espresso
• Pinot

Kafka to Hadoop (Analytics)
6
Use Case
• LinkedIn tracks data to better understand how members use our
products
• Information such as which page got viewed and which content got
clicked on are sent into a Kafka cluster in each data center
• Some of these events are all centrally collected and pushed onto
our Hadoop grid for analysis and daily report generation

7
Couchbase @ LinkedIn
• About 80 separate services with one or more clusters in multiple data
centers
• Up to ~70 servers in a cluster
• Single & Multi-tenant clusters

8
Hadoop to Couchbase
• Our primary use-case for Hadoop  Couchbase is for building
(warming) / restoring Couchbase buckets
• LinkedIn built it’s own in-house solution to work with our ETL
processes etc

Jobs Cluster
9
Clusters & Numbers
• Used for read-scaling, > 150k QPS, 27 node clusters
• We use Hadoop to pre-build data by partition
• Couchbase average latency is 2-3ms
• 99th percentile is ~8 - 12ms

Questions?
10
Thank You

©2014 LinkedIn Corporation. All Rights Reserved.

Weitere ähnliche Inhalte

Was ist angesagt?

Large networks consist of a diverse range of equipment, across private, public, hybrid clouds and partner networks. A hierarchical network has layers of infrastructure, catering to access, core, or distribution roles, managed by different organizations specialized to architect the right network hardware, software, and features for that network layer. The nature of data generated by each component can vary in type and form, including logs, events, metrics, or alarms. The diversity of data generated by a large network is beyond human scale. Apache Kafka® is a critical hub in large networks, empowering AIOps to enhance decision making, improve analysis and insights by contextualizing large volumes of operational data. Kafka solved the big problem of collecting, processing, storing and normalizing data at scale, allowing us to focus on building the AIOps pipeline. Our platform connects the dots across relevant operations data and provides operations teams with simple and powerful access to insights, from within increasingly popular collaboration environments like Slack and Microsoft teams. The pipeline must also integrate with automation solutions. This session will cover how large volumes of streaming messages can be received by parallel Kafka consumers, and turned into action by network operations teams, dramatically reducing downtime and improving performance.

Kafka at the core of an AIOps pipeline | Sunanda Kommula, Selector.ai and Ala...

Kafka at the core of an AIOps pipeline | Sunanda Kommula, Selector.ai and Ala...

Kafka at the core of an AIOps pipeline | Sunanda Kommula, Selector.ai and Ala...

HostedbyConfluent

As Kafka deployments grow within your organization, so do the challenges around lifecycle management. For instance, do you really know what streams exist, who is producing and consuming them? What is the effect of upstream changes? How is this information kept up to date, so it is relevant and consistent to others looking to reuse these streams? Ever wish you had a way to view and visualize graphically the relationships between schemas, topics and applications? In this talk we will show you how to do that and get more value from your Kafka Streaming infrastructure using an event portal. It’s like an API portal but specialized for event streams and publish/subscribe patterns. Join us to see how you can automatically discover event streams from your Kafka clusters, import them to a catalog and then leverage code gen capabilities to ease development of new applications.

How to Discover, Visualize, Catalog, Share and Reuse your Kafka Streams (Jona...

How to Discover, Visualize, Catalog, Share and Reuse your Kafka Streams (Jona...

How to Discover, Visualize, Catalog, Share and Reuse your Kafka Streams (Jona...

HostedbyConfluent

Kafka Connectors are used extensively in data migration solutions, serving as a middle tier when migrating data across databases. In addition, microservice architectures also use Kafka Connectors heavily when communicating with one another while still operating independently on their own data stores. In this talk, we cover these use cases in more detail along with a deep dive into the architecture of the source and sink Kafka Connectors for Cosmos DB.

Azure Cosmos DB Kafka Connectors | Abinav Rameesh, Microsoft

Azure Cosmos DB Kafka Connectors | Abinav Rameesh, Microsoft

Azure Cosmos DB Kafka Connectors | Abinav Rameesh, Microsoft

HostedbyConfluent

One of the great things about running applications in the cloud is that you only pay for the resources that you use. But that also makes it more important than ever for our applications to be resource-efficient. This becomes even more critical when we use serverless functions. Micronaut is an application framework that provides dependency injection, developer productivity features, and excellent support for Apache Kafka. By performing dependency injection, AOP, and other productivity-enhancing magic at compile time, Micronaut allows us to build smaller, more efficient microservices and serverless functions. In this session, we'll explore the ways that Apache Kafka and Micronaut work together to enable us to build fast, efficient, event-driven applications. Then we'll see it in action, using the AWS Lambda Sink Connector for Confluent Cloud.

Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...

Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...

Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...

HostedbyConfluent

Speaker: Jun Rao, Co-founder, Confluent In 2010, LinkedIn began developing Apache Kafka®. In 2011, Kafka was released an Apache open source project. Since then, the use of Kafka has grown rapidly in a variety of businesses. Now more than 30% of Fortune 500 companies are already using Kafka. In this 60-minute online talk, Confluent Co-founder Jun Rao will: -Explain how Kafka became the predominant publish/subscribe messaging system that it is today -Introduce Kafka's most recent additions to its set of enterprise-level features -Demonstrate how to evolve your Kafka implementation into a complete real-time streaming data platform that functions as the central nervous system for your organization Watch the recording: https://cnfl.io/kafka-past-present-future-on-demand

Apache Kafka: Past, Present and Future

Apache Kafka: Past, Present and Future

Apache Kafka: Past, Present and Future

Are you looking for a cloud-based architecture that includes the best of breed streaming and database technologies? In this session you will learn how to setup and configure the Confluent Cloud with MongoDB Atlas. We'll start the journey learning about the basic connectivity between the two cloud services and end with a brief discovery of what you can do with data once it is in MongoDB Atlas. By the end of this session you will know how to securely setup and configure the MongoDB Atlas connectors in the Confluent Cloud in both a source and sink configuration.

Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...

Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...

Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...

HostedbyConfluent

Legacy migration is a journey. Mainframes cannot be replaced in a single project. A big bang will fail. This has to be planned long-term. Mainframe offloading and replacement with Apache Kafka and its ecosystem can be used to keep a more modern data store in real-time sync with the mainframe, while at the same time persisting the event data on the bus to enable microservices, and deliver the data to other systems such as data warehouses and search indexes. This session walks through the different steps some companies are already gone through. Technical options like Change Data Capture (CDC), MQ, and third-party tools for mainframe integration, offloading and replacement are explored.

Mainframe Integration, Offloading and Replacement with Apache Kafka | Kai Wae...

Mainframe Integration, Offloading and Replacement with Apache Kafka | Kai Wae...

Mainframe Integration, Offloading and Replacement with Apache Kafka | Kai Wae...

HostedbyConfluent

Guru Sattanathan, Confluent, Senior Solutions Engineer Enterprise Integration technologies (aka Middleware) are the key enablers when it comes to Real-time data flows or Event Driven Architecture. Starting from real-time payments, e-commerce, travel booking systems, etc, everything is powered by a middleware underneath. It did transform a lot of things but with caveats! Are ESB’s & MQ’s enough for today’s integration needs? Do you know their technical debts? If you are someone looking at integrating your applications or an Integration Architect this session is for you. It's time to refresh yourself and see how organizations are building integrations today. In this session, we will go in this order: -Recap on Enterprise Integration technologies -What are the key flaws & What needs improvement? -What is Apache Kafka? -Rethinking Integration using Apache Kafka https://www.meetup.com/KafkaMelbourne/events/280590162/

Death of the dumb pipes: Using Apache Kafka® for Integration projects

Death of the dumb pipes: Using Apache Kafka® for Integration projects

Death of the dumb pipes: Using Apache Kafka® for Integration projects

HostedbyConfluent

Compacted topics grow over time and are often utilizing high performance, low latency and relatively expensive storage solutions. Reducing duplicated data plays a critical role in the size of compacted topics. with less data on the topics, the Kafka cluster consumes less disk space which in turn it leads to lower operation cost. in this use case-driven talk, we are going to demonstrate how our team at UnitedHealth Group leveraged existing transformers to extract data from the message metadata in the topic as well as how we developed our customized transformers to minimize the amount of duplicated data in each message in the topic.

Developing custom transformation in the Kafka connect to minimize data redund...

Developing custom transformation in the Kafka connect to minimize data redund...

Developing custom transformation in the Kafka connect to minimize data redund...

HostedbyConfluent

The auto industry is midst a data revolution that is transforming how companies do business. Once a scarce resource, data has now become abundant and cheap. What are the new technologies that change the way we produce, collect, process, store, and analyze data. What new streams of data are being created with Industry 4.0 and the Internet of Things on the horizon, is there significant value to taking a strategic approach to Fast Data. How is Porsche building the next level Data Streaming Platform with open source technologies and how we are using CI/CD pipelines amongst others in order to serve our use cases.

0-330km/h: Porsche's Data Streaming Journey | Sridhar Mamella, Porsche

0-330km/h: Porsche's Data Streaming Journey | Sridhar Mamella, Porsche

0-330km/h: Porsche's Data Streaming Journey | Sridhar Mamella, Porsche

HostedbyConfluent

Kafka Summit NYC 2017 - Achieving Predictability and Compliance with BNY Mell...

Kafka Summit NYC 2017 - Achieving Predictability and Compliance with BNY Mell...

Kafka Summit NYC 2017 - Achieving Predictability and Compliance with BNY Mell...

Bootstrap SaaS startup using Open Source Tools

Bootstrap SaaS startup using Open Source Tools

Bootstrap SaaS startup using Open Source Tools

At Gloo.us, we face a challenge in providing platform data to heterogeneous applications in a way that eliminates access contention, avoids high latency ETLs, and ensures consistency for many teams. We're solving this problem by adopting Data Mesh principles and leveraging Kafka, Kafka Connect, and Kafka streams to build an event driven architecture to connect applications to the data they need. A domain driven design keeps the boundaries between specialized process domains and singularly focused data domains clear, distinct, and disciplined. Applying the principles of a Data Mesh, process domains assume the responsibility of transforming, enriching, or aggregating data rather than relying on these changes at the source of truth -- the data domains. Architecturally, we've broken centralized big data lakes into smaller data stores that can be consumed into storage managed by process domains. This session covers how we’re applying Kafka tools to enable our data mesh architecture. This includes how we interpret and apply the data mesh paradigm, the role of Kafka as the backbone for a mesh of connectivity, the role of Kafka Connect to generate and consume data events, and the use of KSQL to perform minor transformations for consumers.

How a Data Mesh is Driving our Platform | Trey Hicks, Gloo

How a Data Mesh is Driving our Platform | Trey Hicks, Gloo

How a Data Mesh is Driving our Platform | Trey Hicks, Gloo

HostedbyConfluent

Flipp is an e-commerce company that promotes weekly shopping opportunities. We began our migration to event-driven microservices in November 2016, and have since moved to nearly 300 Kafka-powered microservices. In this presentation we will explore the major strategies we have used in our migration from distributed monoliths to event-driven microservices. There have been a number of painful learnings and pitfalls along the way that we will share with you. Lastly, we will provide recommendations for each step of the way on your journey from monoliths to effective event-driven microservices. The first major section of this presentation deals with the liberation of data from monolithic services. In this section we will cover: Kafka Connect vs System Production, Event Schematization, Entities and Events, The importance of the Single Source of Truth, Consumption patterns and Event update verbosity. The second major section of this presentation discusses the usage of liberated event data in conjunction with other event streams.In this section we will cover common access patterns, handling (lots) of relational data, Stateful Foreign-Key Joins in Kafka Streams (See Kafka KIP-213), High frequency updates (price, stock) vs static properties and how to handle too many data streams. The third major section details how to abstract event complexity away, leverage the single source of truth and the usage of Core Events across a company. In this section we cover abstracting data streams, Core Events as detailed by the Single Source of Truth, Core Events in relation to bounded contexts and using Core Events successfully as a business.

The Migration to Event-Driven Microservices (Adam Bellemare, Flipp) Kafka Sum...

The Migration to Event-Driven Microservices (Adam Bellemare, Flipp) Kafka Sum...

The Migration to Event-Driven Microservices (Adam Bellemare, Flipp) Kafka Sum...

The Oak Ridge Leadership Facility (OLCF) in the National Center for Computational Sciences (NCCS) division at Oak Ridge National Laboratory (ORNL) houses world-class high-performance computing (HPC) resources and has a history of operating top-ranked supercomputers on the TOP500 list, including the world's current fastest, Summit, an IBM AC922 machine with a peak of 200 petaFLOPS. With the exascale era rapidly approaching, the need for a robust and scalable big data platform for operations data is more important than ever. In the past when a new HPC resource was added to the facility, pipelines from data sources spanned multiple data sinks which oftentimes resulted in data silos, slow operational data onboarding, and non-scalable data pipelines for batch processing. Using Apache Kafka as the message bus of the division's new big data platform has allowed for easier decoupling of scalable data pipelines, faster data onboarding, and stream processing with the goal to continuously improve insight into the HPC resources and their supporting systems. This talk will focus on the NCCS division's transition to Apache Kafka over the past few years to enhance the OLCF's current capabilities and prepare for Frontier, OLCF's future exascale system; including the development and deployment of a full big data platform in a Kubernetes environment from both a technical and cultural shift perspective. This talk will also cover the mission of the OLCF, the operational data insights related to high-performance computing that the organization strives for, and several use-cases that exist in production today.

Enabling Insight to Support World-Class Supercomputing (Stefan Ceballos, Oak ...

Enabling Insight to Support World-Class Supercomputing (Stefan Ceballos, Oak ...

Enabling Insight to Support World-Class Supercomputing (Stefan Ceballos, Oak ...

Are you looking for a cloud-based architecture that includes the best of breed streaming and database technologies? In this session you will learn how to setup and configure the Confluent Cloud with MongoDB Atlas. We'll start the journey learning about the basic connectivity between the two cloud services and end with a brief discovery of what you can do with data once it is in MongoDB Atlas. By the end of this session you will know how to securely setup and configure the MongoDB Atlas connectors in the Confluent Cloud in both a source and sink configuration.

Streaming data in the cloud with Confluent and MongoDB Atlas | Robert Waters,...

Streaming data in the cloud with Confluent and MongoDB Atlas | Robert Waters,...

Streaming data in the cloud with Confluent and MongoDB Atlas | Robert Waters,...

HostedbyConfluent

Introduction to Apache Kafka and why it matters - Madrid

Introduction to Apache Kafka and why it matters - Madrid

Introduction to Apache Kafka and why it matters - Madrid

(Preston Thompson, Braze) Kafka Summit SF 2018 If you collect billions of data points every day and create billions more sending and tracking messages, then you know you need to get your infrastructure right. Our clients use Braze to engage their users over their lifecycle via push notifications, emails, in-app messages and more. Using our Currents product, clients can enable multiple configurable integrations to export this event data in real time to a variety of third-party systems, allowing them to tightly integrate with the rest of their operations and understand the impacts of their engagement strategy. We use Kafka and the Kafka ecosystem to power this high volume real-time export. As you’d expect in a big data environment, we take data collected from a variety of sources—our SDKs, email partner APIs, our own systems—and produce it to Kafka, with topics for each type of event (about 30 types). Kafka Streams filters and transforms this data according to the configurations set by our clients. Clients can choose which types of events should be sent to which third-party systems. Kafka Connect helps to export the data to third-party systems in real time using custom developed connectors. We run a connector instance for each integration for each customer that consumes from the integration-specific topic. On top of it all, we built a service to manage the pipeline. The service provides configurations to the Streams application and also creates topics for new integrations and uses the Connect REST API to create and manage connectors. In this talk, I will discuss: -How we started our journey in designing this large-scale streaming architecture -Why streaming technologies were necessary to solve our technology and business issues -The lessons we learned along the way that can help you with your Kafka-based architecture

Real-Time Dynamic Data Export Using the Kafka Ecosystem

Real-Time Dynamic Data Export Using the Kafka Ecosystem

Real-Time Dynamic Data Export Using the Kafka Ecosystem

The challenge with today’s “data explosion” is finding the most appropriate answer to the question, “So where do I put my data?” while avoiding the longer-term problem: data warehouses, data lakes, cloud storage, NoSQL databases, … are often the places where “big” data goes to die. Enter Physics 101, and my corollary to Newton’s First Law of Motion: Data in motion tends to stay in motion until it comes rest on disk. Similarly, if data is at rest, it will remain at rest until an external “force” puts it in motion again. Data inevitably comes to rest at some point. Without “external forces”, data often gets lost or becomes stale where it lands. “Modern” architectures tend to involve data pipelines where downstream consumers of data make use of data generated upstream, often with built-for-purpose repositories at each stage. This session will explore how data that has come to rest can be put in motion again; how Kafka can keep it in motion longer; and how pipelined architectures might be created to make use of that data.

Data in Motion: Building Stream-Based Architectures with Qlik Replicate & Kaf...

Data in Motion: Building Stream-Based Architectures with Qlik Replicate & Kaf...

Data in Motion: Building Stream-Based Architectures with Qlik Replicate & Kaf...

HostedbyConfluent

Data volumes continue to grow, demanding new, more scalable solutions for low-latency data processing. Previously, the default approach to deploying such systems was to throw a ton of hardware at the problem. However, that is no longer necessary, as newer technologies showcase a level of efficiency that enables smaller, more manageable clusters while handling extreme workloads. Processing billions of events per second on Kafka can now be done with a modest investment in compute resources. In this session, you will learn how to architect and build the fastest data processing applications that scale linearly, and combine streaming data and reference data data-in-motion and data-at-rest with machine learning. We will take you through the end-to-end framework and example application, built on the Hazelcast Platform, an open source software engine designed for ultra-fast performance. We will also show how you can leverage SQL to further explore the operational data in the solution including querying Kafka topics and key-value data on the in-memory data store. Attendees will also get access to the Github sample application shown.

Low-latency real-time data processing at giga-scale with Kafka | John DesJard...

Low-latency real-time data processing at giga-scale with Kafka | John DesJard...

Low-latency real-time data processing at giga-scale with Kafka | John DesJard...

HostedbyConfluent

Was ist angesagt? (20)

Kafka at the core of an AIOps pipeline | Sunanda Kommula, Selector.ai and Ala...

Kafka at the core of an AIOps pipeline | Sunanda Kommula, Selector.ai and Ala...

Kafka at the core of an AIOps pipeline | Sunanda Kommula, Selector.ai and Ala...

How to Discover, Visualize, Catalog, Share and Reuse your Kafka Streams (Jona...

How to Discover, Visualize, Catalog, Share and Reuse your Kafka Streams (Jona...

How to Discover, Visualize, Catalog, Share and Reuse your Kafka Streams (Jona...

Azure Cosmos DB Kafka Connectors | Abinav Rameesh, Microsoft

Azure Cosmos DB Kafka Connectors | Abinav Rameesh, Microsoft

Azure Cosmos DB Kafka Connectors | Abinav Rameesh, Microsoft

Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...

Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...

Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...

Apache Kafka: Past, Present and Future

Apache Kafka: Past, Present and Future

Apache Kafka: Past, Present and Future

Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...

Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...

Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...

Mainframe Integration, Offloading and Replacement with Apache Kafka | Kai Wae...

Mainframe Integration, Offloading and Replacement with Apache Kafka | Kai Wae...

Mainframe Integration, Offloading and Replacement with Apache Kafka | Kai Wae...

Death of the dumb pipes: Using Apache Kafka® for Integration projects

Death of the dumb pipes: Using Apache Kafka® for Integration projects

Death of the dumb pipes: Using Apache Kafka® for Integration projects

Developing custom transformation in the Kafka connect to minimize data redund...

Developing custom transformation in the Kafka connect to minimize data redund...

Developing custom transformation in the Kafka connect to minimize data redund...

0-330km/h: Porsche's Data Streaming Journey | Sridhar Mamella, Porsche

0-330km/h: Porsche's Data Streaming Journey | Sridhar Mamella, Porsche

0-330km/h: Porsche's Data Streaming Journey | Sridhar Mamella, Porsche

Kafka Summit NYC 2017 - Achieving Predictability and Compliance with BNY Mell...

Kafka Summit NYC 2017 - Achieving Predictability and Compliance with BNY Mell...

Kafka Summit NYC 2017 - Achieving Predictability and Compliance with BNY Mell...

Bootstrap SaaS startup using Open Source Tools

Bootstrap SaaS startup using Open Source Tools

Bootstrap SaaS startup using Open Source Tools

How a Data Mesh is Driving our Platform | Trey Hicks, Gloo

How a Data Mesh is Driving our Platform | Trey Hicks, Gloo

How a Data Mesh is Driving our Platform | Trey Hicks, Gloo

The Migration to Event-Driven Microservices (Adam Bellemare, Flipp) Kafka Sum...

The Migration to Event-Driven Microservices (Adam Bellemare, Flipp) Kafka Sum...

The Migration to Event-Driven Microservices (Adam Bellemare, Flipp) Kafka Sum...

Enabling Insight to Support World-Class Supercomputing (Stefan Ceballos, Oak ...

Enabling Insight to Support World-Class Supercomputing (Stefan Ceballos, Oak ...

Enabling Insight to Support World-Class Supercomputing (Stefan Ceballos, Oak ...

Streaming data in the cloud with Confluent and MongoDB Atlas | Robert Waters,...

Streaming data in the cloud with Confluent and MongoDB Atlas | Robert Waters,...

Streaming data in the cloud with Confluent and MongoDB Atlas | Robert Waters,...

Introduction to Apache Kafka and why it matters - Madrid

Introduction to Apache Kafka and why it matters - Madrid

Introduction to Apache Kafka and why it matters - Madrid

Real-Time Dynamic Data Export Using the Kafka Ecosystem

Real-Time Dynamic Data Export Using the Kafka Ecosystem

Real-Time Dynamic Data Export Using the Kafka Ecosystem

Data in Motion: Building Stream-Based Architectures with Qlik Replicate & Kaf...

Data in Motion: Building Stream-Based Architectures with Qlik Replicate & Kaf...

Data in Motion: Building Stream-Based Architectures with Qlik Replicate & Kaf...

Low-latency real-time data processing at giga-scale with Kafka | John DesJard...

Low-latency real-time data processing at giga-scale with Kafka | John DesJard...

Low-latency real-time data processing at giga-scale with Kafka | John DesJard...

Andere mochten auch

Just over two years ago, I completed college, moved countries and started as a SRE at LinkedIn. Over those two years, I’ve gone from a Junior engineer to a Senior Engineer, however, I wouldn’t have been able to do it without a great onboarding experience and the mentors that have guided me This session will take the perspective of myself as a New College Graduate and Nina Mushiana (SRE manager) on how to onboard new SREs and help them reach their full potential. We will deep dive into onboarding, mentoring and training topics and reflect on lessons learnt over my two year experience. Finally I’ll present advice for ELTs on how to become an important and successful part of their organization.

SRECon USA 2016: Growing your Entry Level Talent

SRECon USA 2016: Growing your Entry Level Talent

SRECon USA 2016: Growing your Entry Level Talent

Good monitoring can be the difference between a great night's sleep or hearing your phone go off at 2:37 a.m. because of a production outage. Couchbase Server provides a large number of metrics which can be overwhelming if you do not know the critical things to focus on or how to expose that information to your monitoring system. In this talk we will look at example production incidents, going in depth around specific things to monitor, and how this information can be used to find issues, work out root cause, and discover trends.

Couchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedIn

Couchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedIn

Couchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedIn

SouthBay SRE Meetup Jan 2016

SouthBay SRE Meetup Jan 2016

SouthBay SRE Meetup Jan 2016

Couchbase Connect 2016

Couchbase Connect 2016

Couchbase Connect 2016

LinkedIn serves traffic for its 467 million members from four data centers and multiple PoPs spread geographically around the world. Serving live traffic from from many places at the same time has taken us from a disaster recovery model to a disaster avoidance model where we can take an unhealthy data center or PoP out of rotation and redistribute its traffic to a healthy one within minutes, with virtually no visible impact to users. The geographical distribution of our infrastructure also allows us to optimize the end-user's experience by geo routing users to the best possible PoP and datacenter. This talk provide details on how LinkedIn shifts traffic between its PoPs and data centers to provide the best possible performance and availability for its members. We will also touch on the complexities of performance in APAC, how IPv6 is helping our members and how LinkedIn stress tests data centers verify its disaster recovery capabilities.

APRICOT 2017: Trafficshifting: Avoiding Disasters & Improving Performance at ...

APRICOT 2017: Trafficshifting: Avoiding Disasters & Improving Performance at ...

APRICOT 2017: Trafficshifting: Avoiding Disasters & Improving Performance at ...

Using SaltStack to Auto Triage and Remediate Production Systems

Using SaltStack to Auto Triage and Remediate Production Systems

Using SaltStack to Auto Triage and Remediate Production Systems

LinkedIn’s production stack is made up of over 900 applications and over 2200 internal API’s. With any given application having many interconnected pieces, it is difficult to escalate to the right person in a timely manner. In order to combat this, LinkedIn built an Event Correlation Engine that monitors service health and maps dependencies between services to correctly escalate to the SRE’s who own the unhealthy service. We’ll discuss the approach we used in building a correlation engine and how it has been used at LinkedIn to reduce incident impact and provide better quality of life to LinkedIn’s oncall engineers.

Reducing MTTR and False Escalations: Event Correlation at LinkedIn

Reducing MTTR and False Escalations: Event Correlation at LinkedIn

Reducing MTTR and False Escalations: Event Correlation at LinkedIn

Netflix Keystone - How Netflix Handles Data Streams up to 11M Events/Sec

Netflix Keystone - How Netflix Handles Data Streams up to 11M Events/Sec

Netflix Keystone - How Netflix Handles Data Streams up to 11M Events/Sec

Software reliability tools and common software errors

Software reliability tools and common software errors

Software reliability tools and common software errors

How TPM saves the day

How TPM saves the day

How TPM saves the day

Keystone - Processing over Half a Trillion events per day with 8 million events & 17 GB per second peaks, and at-least once processing semantics. We will explore in detail how we employ Kafka, Samza, and Docker at scale to implement a multi-tenant pipeline. We will also look at the evolution to its current state and where the pipeline is headed next in offering a self-service stream processing infrastructure atop the Kafka based pipeline and support Spark Streaming.

Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015

Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015

Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015

Cloud Native, Microservices and SRE/Chaos Engineering: The new Rules of The G...

Cloud Native, Microservices and SRE/Chaos Engineering: The new Rules of The G...

Cloud Native, Microservices and SRE/Chaos Engineering: The new Rules of The G...

Software Reliability Engineering

Software Reliability Engineering

Software Reliability Engineering

Software reliability growth model

Software reliability growth model

Software reliability growth model

Kafka is a scalable, distributed publish subscribe messaging system that's used as a data transmission backbone in many data intensive digital businesses. Couchbase Server is a scalable, flexible document database that's fast, agile, and elastic. Because they both appeal to the same type of customers, Couchbase and Kafka are often used together. This presentation from a meetup in Mountain View describes Kafka's design and why people use it, Couchbase Server and its uses, and the use cases for both together. Also covered is a description and demo of Couchbase Server writing documents to a Kafka topic and consuming messages from a Kafka topic. using the Couchbase Kafka Connector.

Real time Messages at Scale with Apache Kafka and Couchbase

Real time Messages at Scale with Apache Kafka and Couchbase

Real time Messages at Scale with Apache Kafka and Couchbase

Feedback loops: How SREs benefit and what is needed to realize their potential

Feedback loops: How SREs benefit and what is needed to realize their potential

Feedback loops: How SREs benefit and what is needed to realize their potential

This is a lightning talk topic I presented at Triangle DevOps meetup to share ideas about operating load balancer by following SRE practices. It highlighted continuous delivery process and test driven development when we develop this feature. There're lots of online cloud-based services we used to achieve our goal including AWS, Google Cloud, Librato, Logentries, SignalSciences, BlueBoxCloud, Quay.io etc. And lots of open source software we love and use every day. Disclaimer: It represents personal opinions only, not represent any points of view of my employer.

Load balancing in the SRE way

Load balancing in the SRE way

Load balancing in the SRE way

The need for gleaning answers from data in real-time is moving from nicety to a necessity. There are few options to analyze the never-ending stream of unbounded data at scale. Let’s compare and contrast the core principles and technologies the different open source solutions available to help with this endeavor, and where in the future processing engines need to evolve to solve processing needs at scale. These findings are based on the experience of continuing to build a scalable solution in the cloud to process over 700 billion events at Netflix, and how we are embarking on the next journey to evolve unbounded data processing engines.

Unbounded bounded-data-strangeloop-2016-monal-daxini

Unbounded bounded-data-strangeloop-2016-monal-daxini

Unbounded bounded-data-strangeloop-2016-monal-daxini

In this session, Netflix provides an overview of Keystone, their new data pipeline. The session covers how Netflix migrated from Suro to Keystone, including the reasons behind the transition and the challenges of zero loss while processing over 400 billion events daily. The session covers in detail how they deploy, operate, and scale Kafka, Samza, Docker, and Apache Mesos in AWS to manage 8 million events & 17 GB per second during peak.

(BDT318) How Netflix Handles Up To 8 Million Events Per Second

(BDT318) How Netflix Handles Up To 8 Million Events Per Second

(BDT318) How Netflix Handles Up To 8 Million Events Per Second

Amazon Web Services

Andere mochten auch (19)

SRECon USA 2016: Growing your Entry Level Talent

SRECon USA 2016: Growing your Entry Level Talent

SRECon USA 2016: Growing your Entry Level Talent

Couchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedIn

Couchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedIn

Couchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedIn

SouthBay SRE Meetup Jan 2016

SouthBay SRE Meetup Jan 2016

SouthBay SRE Meetup Jan 2016

Couchbase Connect 2016

Couchbase Connect 2016

Couchbase Connect 2016

APRICOT 2017: Trafficshifting: Avoiding Disasters & Improving Performance at ...

APRICOT 2017: Trafficshifting: Avoiding Disasters & Improving Performance at ...

APRICOT 2017: Trafficshifting: Avoiding Disasters & Improving Performance at ...

Using SaltStack to Auto Triage and Remediate Production Systems

Using SaltStack to Auto Triage and Remediate Production Systems

Using SaltStack to Auto Triage and Remediate Production Systems

Reducing MTTR and False Escalations: Event Correlation at LinkedIn

Reducing MTTR and False Escalations: Event Correlation at LinkedIn

Reducing MTTR and False Escalations: Event Correlation at LinkedIn

Netflix Keystone - How Netflix Handles Data Streams up to 11M Events/Sec

Netflix Keystone - How Netflix Handles Data Streams up to 11M Events/Sec

Netflix Keystone - How Netflix Handles Data Streams up to 11M Events/Sec

Software reliability tools and common software errors

Software reliability tools and common software errors

Software reliability tools and common software errors

How TPM saves the day

How TPM saves the day

How TPM saves the day

Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015

Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015

Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015

Cloud Native, Microservices and SRE/Chaos Engineering: The new Rules of The G...

Cloud Native, Microservices and SRE/Chaos Engineering: The new Rules of The G...

Cloud Native, Microservices and SRE/Chaos Engineering: The new Rules of The G...

Software Reliability Engineering

Software Reliability Engineering

Software Reliability Engineering

Software reliability growth model

Software reliability growth model

Software reliability growth model

Real time Messages at Scale with Apache Kafka and Couchbase

Real time Messages at Scale with Apache Kafka and Couchbase

Real time Messages at Scale with Apache Kafka and Couchbase

Feedback loops: How SREs benefit and what is needed to realize their potential

Feedback loops: How SREs benefit and what is needed to realize their potential

Feedback loops: How SREs benefit and what is needed to realize their potential

Load balancing in the SRE way

Load balancing in the SRE way

Load balancing in the SRE way

Unbounded bounded-data-strangeloop-2016-monal-daxini

Unbounded bounded-data-strangeloop-2016-monal-daxini

Unbounded bounded-data-strangeloop-2016-monal-daxini

(BDT318) How Netflix Handles Up To 8 Million Events Per Second

(BDT318) How Netflix Handles Up To 8 Million Events Per Second

(BDT318) How Netflix Handles Up To 8 Million Events Per Second

Ähnlich wie Couchbase Meetup Jan 2016

CA Technologies Customer Presentation

CA Technologies Customer Presentation

CA Technologies Customer Presentation

Data Stream Processing for Beginners with Kafka and CDC

Data Stream Processing for Beginners with Kafka and CDC

Data Stream Processing for Beginners with Kafka and CDC

Apache Kafka at LinkedIn

Apache Kafka at LinkedIn

Apache Kafka at LinkedIn

Ruslan Belkin And Sean Dawson on LinkedIn's Network Updates Uncovered

Ruslan Belkin And Sean Dawson on LinkedIn's Network Updates Uncovered

Ruslan Belkin And Sean Dawson on LinkedIn's Network Updates Uncovered

Building Scalable Big Data Infrastructure Using Open Source Software Presenta...

Building Scalable Big Data Infrastructure Using Open Source Software Presenta...

Building Scalable Big Data Infrastructure Using Open Source Software Presenta...

Unlocking big data with Hadoop + MySQL

Unlocking big data with Hadoop + MySQL

Unlocking big data with Hadoop + MySQL

___________________________________________ Meetup#7 | Session 2 | 21/03/2018 | Taboola _____________________________________________ In this talk, we will present our multi-DC Kafka architecture, and discuss how we tackle sending and handling 10B+ messages per day, with maximum availability and no tolerance for data loss. Our architecture includes technologies such as Cassandra, Spark, HDFS, and Vertica - with Kafka as the backbone that feeds them all.

Distributed Kafka Architecture Taboola Scale

Distributed Kafka Architecture Taboola Scale

Distributed Kafka Architecture Taboola Scale

Apache Kafka TLV

This presentation will describe how to go beyond a "Hello world" stream application and build a real-time data-driven product. We will present architectural patterns, go through tradeoffs and considerations when deciding on technology and implementation strategy, and describe how to put the pieces together. We will also cover necessary practical pieces for building real products: testing streaming applications, and how to evolve products over time. Presented at highloadstrategy.com 2016 by Øyvind Løkling (Schibsted Products & Technology), joint work with Lars Albertsson (independent, www.mapflat.com).

Building real time data-driven products

Building real time data-driven products

Building real time data-driven products

Lars Albertsson

Modernizing Global Shared Data Analytics Platform and our Alluxio Journey

Modernizing Global Shared Data Analytics Platform and our Alluxio Journey

Modernizing Global Shared Data Analytics Platform and our Alluxio Journey

Future of Data Engineering

Future of Data Engineering

Future of Data Engineering

Designing a social network offers some exciting challenges to engineers. The system needs to operate at scale, to provide a responsive user experience and to be able to inspect user activity in order to both generate new content and improve how the existing content is delivered. Event Driven Architectures are particularly suitable to handle these kind of challenges and highly scalable messaging systems such as Apache Kafka have been designed specifically to support the requirements of modern high volume applications. In this talk we are describing how the Crowdmix back-end has been designed as an Event Based system running on top of Kafka. We are going to present the overall system architecture and discuss in more detail some of the different sub components processing those events in different fashions, from streaming based processing to batch processing passing through a lambda-style batch and stream cooperation. We are going to conclude describing some lessons learned from our one-year journey in implementing and operating the system

Music streams

Stefano Galarraga

Customer Feedback Analytics for Starbucks

Customer Feedback Analytics for Starbucks

Customer Feedback Analytics for Starbucks

We consider a microservices architecture to achieve an end goal, not because it's "the cool thing to do". Every organization looking to adopt this architecture must realize (and adhere) to a set of foundational principles. Guided by those principles, we can correctly choose the technology to help support a microservices architecture and meet our end goals. This talk explains those core principles and gives you the tools needed for your microservices journey.

Microservices Journey Summer 2017

Microservices Journey Summer 2017

Microservices Journey Summer 2017

Christian Posta

Modern ETL: Azure Data Factory, Data Lake, and SQL Database

Modern ETL: Azure Data Factory, Data Lake, and SQL Database

Modern ETL: Azure Data Factory, Data Lake, and SQL Database

LinkedIn Infrastructure (analytics@webscale, at fb 2013)

LinkedIn Infrastructure (analytics@webscale, at fb 2013)

LinkedIn Infrastructure (analytics@webscale, at fb 2013)

(Bruno Simic, Solutions Engineer, Couchbase) Breakout during Confluent’s streaming event in Munich. This three-day hands-on course focused on how to build, manage, and monitor clusters using industry best-practices developed by the world’s foremost Apache Kafka™ experts. The sessions focused on how Kafka and the Confluent Platform work, how their main subsystems interact, and how to set up, manage, monitor, and tune your cluster.

Billions of Messages in Real Time: Why Paypal & LinkedIn Trust an Engagement ...

Billions of Messages in Real Time: Why Paypal & LinkedIn Trust an Engagement ...

Billions of Messages in Real Time: Why Paypal & LinkedIn Trust an Engagement ...

LINE's messaging service architecture underlying more than 200 million monthl...

LINE's messaging service architecture underlying more than 200 million monthl...

LINE's messaging service architecture underlying more than 200 million monthl...

Facing Open Banking regulation, rapidly increasing transaction volumes and increasing customer expectations, Nationwide took the decision to take load off their back-end systems through real-time streaming of data changes into Kafka. Hear about how Nationwide started their journey with Kafka, from their initial use case of creating a real-time data cache using Change Data Capture, Kafka and Microservices to how Kafka allowed them to build a stream processing backbone used to reengineer the entire banking experience including online banking, payment processing and mortgage applications. See a working demo of the system and what happens to the system when the underlying infrastructure breaks. Technologies covered include: Change Data Capture, Kafka (Avro, partitioning and replication) and using KSQL and Kafka Streams Framework to join topics and process data.

Introducing Events and Stream Processing into Nationwide Building Society (Ro...

Introducing Events and Stream Processing into Nationwide Building Society (Ro...

Introducing Events and Stream Processing into Nationwide Building Society (Ro...

Tras la compra de MySQL por el gigante de las bases de datos se ha experimentado una inevitable evolución. Exposición de algunos de los principales cambios acontecidos por este hecho. La llegada de más inversión, nuevos productos, las relaciones con la comunidad, la colaboración con openstack son algunos de los puntos clave. Además, explicación de algunos de los principales casos de éxito mundiales en diferentes sectores con Wikipedia, NASA Jet Propulsion Laboratory, BBC News, Bank of Finland, Sahlgrenska University Hospital o Virgin América. Autor: Abel Flórez Loranca (Oracle MySQL). Librecon.io

MySQL en el mundo real. Evolución desde la compra por Oracle

MySQL en el mundo real. Evolución desde la compra por Oracle

MySQL en el mundo real. Evolución desde la compra por Oracle

Cloud-native Semantic Layer on Data Lake

Cloud-native Semantic Layer on Data Lake

Cloud-native Semantic Layer on Data Lake

Ähnlich wie Couchbase Meetup Jan 2016 (20)

CA Technologies Customer Presentation

CA Technologies Customer Presentation

CA Technologies Customer Presentation

Data Stream Processing for Beginners with Kafka and CDC

Data Stream Processing for Beginners with Kafka and CDC

Data Stream Processing for Beginners with Kafka and CDC

Apache Kafka at LinkedIn

Apache Kafka at LinkedIn

Apache Kafka at LinkedIn

Ruslan Belkin And Sean Dawson on LinkedIn's Network Updates Uncovered

Ruslan Belkin And Sean Dawson on LinkedIn's Network Updates Uncovered

Ruslan Belkin And Sean Dawson on LinkedIn's Network Updates Uncovered

Building Scalable Big Data Infrastructure Using Open Source Software Presenta...

Building Scalable Big Data Infrastructure Using Open Source Software Presenta...

Building Scalable Big Data Infrastructure Using Open Source Software Presenta...

Unlocking big data with Hadoop + MySQL

Unlocking big data with Hadoop + MySQL

Unlocking big data with Hadoop + MySQL

Distributed Kafka Architecture Taboola Scale

Distributed Kafka Architecture Taboola Scale

Distributed Kafka Architecture Taboola Scale

Building real time data-driven products

Building real time data-driven products

Building real time data-driven products

Modernizing Global Shared Data Analytics Platform and our Alluxio Journey

Modernizing Global Shared Data Analytics Platform and our Alluxio Journey

Modernizing Global Shared Data Analytics Platform and our Alluxio Journey

Future of Data Engineering

Future of Data Engineering

Future of Data Engineering

Music streams

Customer Feedback Analytics for Starbucks

Customer Feedback Analytics for Starbucks

Customer Feedback Analytics for Starbucks

Microservices Journey Summer 2017

Microservices Journey Summer 2017

Microservices Journey Summer 2017

Modern ETL: Azure Data Factory, Data Lake, and SQL Database

Modern ETL: Azure Data Factory, Data Lake, and SQL Database

Modern ETL: Azure Data Factory, Data Lake, and SQL Database

LinkedIn Infrastructure (analytics@webscale, at fb 2013)

LinkedIn Infrastructure (analytics@webscale, at fb 2013)

LinkedIn Infrastructure (analytics@webscale, at fb 2013)

Billions of Messages in Real Time: Why Paypal & LinkedIn Trust an Engagement ...

Billions of Messages in Real Time: Why Paypal & LinkedIn Trust an Engagement ...

Billions of Messages in Real Time: Why Paypal & LinkedIn Trust an Engagement ...

LINE's messaging service architecture underlying more than 200 million monthl...

LINE's messaging service architecture underlying more than 200 million monthl...

LINE's messaging service architecture underlying more than 200 million monthl...

Introducing Events and Stream Processing into Nationwide Building Society (Ro...

Introducing Events and Stream Processing into Nationwide Building Society (Ro...

Introducing Events and Stream Processing into Nationwide Building Society (Ro...

MySQL en el mundo real. Evolución desde la compra por Oracle

MySQL en el mundo real. Evolución desde la compra por Oracle

MySQL en el mundo real. Evolución desde la compra por Oracle

Cloud-native Semantic Layer on Data Lake

Cloud-native Semantic Layer on Data Lake

Cloud-native Semantic Layer on Data Lake

Mehr von Michael Kehoe

eBPF Workshop

eBPF Basics

Code Yellow: Helping operations top-heavy teams the smart way

Code Yellow: Helping operations top-heavy teams the smart way

Code Yellow: Helping operations top-heavy teams the smart way

In 2016, Susan Fowler released the 'Production Ready Microservices' book. This book sets an industry benchmark on explaining how microservices should be conceived, all the way through to documentation. So how does this translate actionable items? This session will explore how to expertly deploy your microservice to production. The audience will learn best practice for designing, deploying, monitoring & documenting application. By the end of the session, attendees should feel confident that they have the knowledge to deploy a service that will be reliable and scalable.

QConSF 2018: Building Production-Ready Applications

QConSF 2018: Building Production-Ready Applications

QConSF 2018: Building Production-Ready Applications

All engineering teams run into trouble from time to time. Alert fatigue, caused by technical debt or a failure to plan for growth, can quickly burn out SREs, overloading both development and operations with reactive work. Layer in the potential for communication problems between teams, and we can find ourselves in a place so troublesome we cannot easily see a path out. At times like this, our natural instinct as reliability engineers is to double down and fight through the issues. Often, however, we need to step back, assess the situation, and ask for help to put the team back on the road to success. We will look at the process for Code Yellow, the term we use for this process of “righting the ship”, and discuss how to identify teams that are struggling. Through a look at three separate experiences, we will examine some of the root causes, what steps were taken, and how the engineering organization as a whole supports the process.

Helping operations top-heavy teams the smart way

Helping operations top-heavy teams the smart way

Helping operations top-heavy teams the smart way

The National Transport Safety Bureau is one of the most widely known Government bodies in the world. It’s their role to run into an incident, secure the scene and understand everything that happened. Given the important and unpredictable nature of their work, they have an extensive manual that sets out how incidents should be attended to and how the investigation should progress. This session will detail how the NTSB’s approach to its work and the procedure that drives it, is transferable to us as incident responders. We’ll talk about the NTSB’s pre-incident preparation, incident notification, attending it, collecting information from the field and writing up a report and holding hearings. We’ll consistently draw parallels to IT incident management and how to create applicable process and procedures that mimic those of the NTSB.

AllDayDevops: What the NTSB teaches us about incident management & postmortems

AllDayDevops: What the NTSB teaches us about incident management & postmortems

AllDayDevops: What the NTSB teaches us about incident management & postmortems

Linux Container Basics

Linux Container Basics

Linux Container Basics

Network failures continue to plague datacenter operators as their symptoms may not have direct correlation with where or why they occur. We introduce 007, a lightweight, always-on diagnosis application that can find problematic links and also pinpoint problems for each TCP connection. 007 is completely contained within the end host. During its two month deployment in a tier-1 datacenter, it detected every problem found by previously deployed monitoring tools while also finding the sources of other problems previously undetected.

Papers We Love Sept. 2018: 007: Democratically Finding The Cause of Packet Drops

Papers We Love Sept. 2018: 007: Democratically Finding The Cause of Packet Drops

Papers We Love Sept. 2018: 007: Democratically Finding The Cause of Packet Drops

The National Transport Safety Bureau is one of the most widely known Government bodies in the world. It’s their role to run into an incident, secure the scene and understand everything that happened. Given the important and unpredictable nature of their work, they have an extensive manual that sets out how incidents should be attended to and how the investigation should progress. This session will detail how the NTSB’s approach to its work and the procedure that drives it, is transferable to us as incident responders. We’ll talk about the NTSB’s pre-incident preparation, incident notification, attending it, collecting information from the field and writing up a report and holding hearings. We’ll consistently draw parallels to IT incident management and how to create applicable process and procedures that mimic those of the NTSB.

What the NTSB teaches us about incident management & postmortems

What the NTSB teaches us about incident management & postmortems

What the NTSB teaches us about incident management & postmortems

PyBay 2018: Production-Ready Python Applications

PyBay 2018: Production-Ready Python Applications

PyBay 2018: Production-Ready Python Applications

SRE teams can sometimes run into periods of time where they have staff burnout, technical debt or poor reliability. As SRE’s, we’re programmed to keep fighting through the issues, when sometimes it’s best to step back, assess the situation; and ask for help to put the team back on a successful pathway. This talk will discuss three separate experiences where teams needed some extra help to stabilize their services and oncall. We’ll discuss how to identify struggling teams; get the right assistance; and build a strategy for the team to succeed.

Helping operations top-heavy teams the smart way

Helping operations top-heavy teams the smart way

Helping operations top-heavy teams the smart way

In 2018, Site Reliability Engineering (SRE) will turn 15 years old. Since Google's inception of the term SRE, companies across the world have adopted a new operations mindset along with automation, deployment and monitoring principals. Most of what SRE does now is well established throughout the industry, so what is the next-wave of reliability principals and automation frameworks? This session will dive into what the future holds for reliability engineering as a field and what will be the next areas of investment and improvement for reliability teams.

The Next Wave of Reliability Engineering

The Next Wave of Reliability Engineering

The Next Wave of Reliability Engineering

Building Production-Ready Microservices: DevopsExchangeSF

Building Production-Ready Microservices: DevopsExchangeSF

Building Production-Ready Microservices: DevopsExchangeSF

How often have you heard stories where someone thought they had a disaster strategy, never tested it and it fails when you need it the most? LinkedIn has evolved from serving live traffic out of one data center to four data centers spread geographically. Serving live traffic from four data centers at the same time has taken the company from a disaster recovery model to a disaster avoidance model, where an unhealthy data center can be taken out of rotation and its traffic redistributed to the healthy data centers within minutes, with virtually no visible impact to users. As LinkedIn transitioned from big monolithic applications to microservices, it was difficult to determine capacity constraints of individual services to handle extra load during disaster scenarios. Stress testing individual services using artificial load in a complex microservices architecture wasn’t sufficient to provide enough confidence in data center’s capacity. To solve this problem, LinkedIn moves live traffic to services site-wide by shifting traffic between datacenters to simulate a disaster every business day!

SF Chaos Engineering Meetup: Building Disaster Recovery via Resilience Engine...

SF Chaos Engineering Meetup: Building Disaster Recovery via Resilience Engine...

SF Chaos Engineering Meetup: Building Disaster Recovery via Resilience Engine...

LinkedIn’s production stack is made up of over 900 applications, 2200 internal API’s and hundreds of databases. With any given application having many interconnected pieces, it is difficult to escalate to the right person in a timely manner. In order to combat this, LinkedIn built an Event Correlation Engine that monitors service health and maps dependencies between services to correctly escalate to the SRE’s who own the unhealthy service. We’ll discuss the approach we used in building a correlation engine and how it has been used at LinkedIn to reduce incident impact and provide better quality of life to LinkedIn’s oncall engineers.

SRECon-Europe-2017: Reducing MTTR and False Escalations: Event Correlation at...

SRECon-Europe-2017: Reducing MTTR and False Escalations: Event Correlation at...

SRECon-Europe-2017: Reducing MTTR and False Escalations: Event Correlation at...

All of us depend on the underlying network to be stable whether in the datacenter or in the cloud. We all have a basic knowledge of how traditional networks run, however in the past 10 years, we’ve moved to building redundant physical topologies in our networks, optimized the routing methodologies accordingly, moved into the cloud and gotten greater visibility and tuneables in the Linux kernel network stack. A lot has changed! However, the way we troubleshoot the network in relation to the applications we support hasn’t adapted. In this session, we’ll review the progress that network infrastructure has made look at specific examples where traditional troubleshooting responses fail us and demonstrate our need to rethink our approach to making applications and the network interact harmoniously.

SRECon-Europe-2017: Networks for SREs

SRECon-Europe-2017: Networks for SREs

SRECon-Europe-2017: Networks for SREs

LinkedIn has evolved from serving live traffic out of one data center to four data centers spread geographically. Serving live traffic from four data centers at the same time has taken the company from a disaster recovery model to a disaster avoidance model, where an unhealthy data center can be taken out of rotation and its traffic redistributed to the healthy data centers within minutes, with virtually no visible impact to users. As LinkedIn transitioned from big monolithic applications to microservices, it was difficult to determine capacity constraints of individual services to handle extra load during disaster scenarios. Stress testing individual services using artificial load in a complex microservices architecture wasn’t sufficient to provide enough confidence in data center’s capacity. To solve this problem, LinkedIn leverages live traffic to stress services site-wide by shifting traffic to simulate a disaster load. Michael Kehoe and Anil Mallapur discuss how LinkedIn uses traffic shifts to mitigate user impact by migrating live traffic between its data centers and stress test site-wide services for improved capacity handling and member experience.

Velocity San Jose 2017: Traffic shifts: Avoiding disasters at scale

Velocity San Jose 2017: Traffic shifts: Avoiding disasters at scale

Velocity San Jose 2017: Traffic shifts: Avoiding disasters at scale

Mehr von Michael Kehoe (17)

eBPF Workshop

eBPF Basics

Code Yellow: Helping operations top-heavy teams the smart way

Code Yellow: Helping operations top-heavy teams the smart way

Code Yellow: Helping operations top-heavy teams the smart way

QConSF 2018: Building Production-Ready Applications

QConSF 2018: Building Production-Ready Applications

QConSF 2018: Building Production-Ready Applications

Helping operations top-heavy teams the smart way

Helping operations top-heavy teams the smart way

Helping operations top-heavy teams the smart way

AllDayDevops: What the NTSB teaches us about incident management & postmortems

AllDayDevops: What the NTSB teaches us about incident management & postmortems

AllDayDevops: What the NTSB teaches us about incident management & postmortems

Linux Container Basics

Linux Container Basics

Linux Container Basics

Papers We Love Sept. 2018: 007: Democratically Finding The Cause of Packet Drops

Papers We Love Sept. 2018: 007: Democratically Finding The Cause of Packet Drops

Papers We Love Sept. 2018: 007: Democratically Finding The Cause of Packet Drops

What the NTSB teaches us about incident management & postmortems

What the NTSB teaches us about incident management & postmortems

What the NTSB teaches us about incident management & postmortems

PyBay 2018: Production-Ready Python Applications

PyBay 2018: Production-Ready Python Applications

PyBay 2018: Production-Ready Python Applications

Helping operations top-heavy teams the smart way

Helping operations top-heavy teams the smart way

Helping operations top-heavy teams the smart way

The Next Wave of Reliability Engineering

The Next Wave of Reliability Engineering

The Next Wave of Reliability Engineering

Building Production-Ready Microservices: DevopsExchangeSF

Building Production-Ready Microservices: DevopsExchangeSF

Building Production-Ready Microservices: DevopsExchangeSF

SF Chaos Engineering Meetup: Building Disaster Recovery via Resilience Engine...

SF Chaos Engineering Meetup: Building Disaster Recovery via Resilience Engine...

SF Chaos Engineering Meetup: Building Disaster Recovery via Resilience Engine...

SRECon-Europe-2017: Reducing MTTR and False Escalations: Event Correlation at...

SRECon-Europe-2017: Reducing MTTR and False Escalations: Event Correlation at...

SRECon-Europe-2017: Reducing MTTR and False Escalations: Event Correlation at...

SRECon-Europe-2017: Networks for SREs

SRECon-Europe-2017: Networks for SREs

SRECon-Europe-2017: Networks for SREs

Velocity San Jose 2017: Traffic shifts: Avoiding disasters at scale

Velocity San Jose 2017: Traffic shifts: Avoiding disasters at scale

Velocity San Jose 2017: Traffic shifts: Avoiding disasters at scale

Kürzlich hochgeladen

A Domino Admins Adventures (Engage 2024)

A Domino Admins Adventures (Engage 2024)

A Domino Admins Adventures (Engage 2024)

Gabriella Davis

The presentation explores the development and application of artificial intelligence (AI) from its inception to its current status in the modern world. The term "artificial intelligence" was first coined by John McCarthy in 1956 to describe efforts to develop computer programs capable of performing tasks that typically require human intelligence. This concept was first introduced at a conference held at Dartmouth College, where programs demonstrated capabilities such as playing chess, proving theorems, and interpreting texts. In the early stages, Alan Turing contributed to the field by defining intelligence as the ability of a being to respond to certain questions intelligently, proposing what is now known as the Turing Test to evaluate the presence of intelligent behavior in machines. As the decades progressed, AI evolved significantly. The 1980s focused on machine learning, teaching computers to learn from data, leading to the development of models that could improve their performance based on their experiences. The 1990s and 2000s saw further advances in algorithms and computational power, which allowed for more sophisticated data analysis techniques, including data mining. By the 2010s, the proliferation of big data and the refinement of deep learning techniques enabled AI to become mainstream. Notable milestones included the success of Google's AlphaGo and advancements in autonomous vehicles by companies like Tesla and Waymo. A major theme of the presentation is the application of generative AI, which has been used for tasks such as natural language text generation, translation, and question answering. Generative AI uses large datasets to train models that can then produce new, coherent pieces of text or other media. The presentation also discusses the ethical implications and the need for regulation in AI, highlighting issues such as privacy, bias, and the potential for misuse. These concerns have prompted calls for comprehensive regulations to ensure the safe and equitable use of AI technologies. Artificial intelligence has also played a significant role in healthcare, particularly highlighted during the COVID-19 pandemic, where it was used in drug discovery, vaccine development, and analyzing the spread of the virus. The capabilities of AI in healthcare are vast, ranging from medical diagnostics to personalized medicine, demonstrating the technology's potential to revolutionize fields beyond just technical or consumer applications. In conclusion, AI continues to be a rapidly evolving field with significant implications for various aspects of society. The development from theoretical concepts to real-world applications illustrates both the potential benefits and the challenges that come with integrating advanced technologies into everyday life. The ongoing discussion about AI ethics and regulation underscores the importance of managing these technologies responsibly to maximize their their benefits while minimizing potential harms.

Artificial Intelligence: Facts and Myths

Artificial Intelligence: Facts and Myths

Artificial Intelligence: Facts and Myths

Automating Google Workspace (GWS) & more with Apps Script

Automating Google Workspace (GWS) & more with Apps Script

Automating Google Workspace (GWS) & more with Apps Script

Manulife - Insurer Innovation Award 2024

Manulife - Insurer Innovation Award 2024

Manulife - Insurer Innovation Award 2024

The Digital Insurer

Scaling API-first – The story of a global engineering organization Ian Reasor, Senior Computer Scientist - Adobe Radu Cotescu, Senior Computer Scientist - Adobe Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Discord is a free app offering voice, video, and text chat functionalities, primarily catering to the gaming community. It serves as a hub for users to create and join servers tailored to their interests. Discord’s ecosystem comprises servers, each functioning as a distinct online community with its own channels dedicated to specific topics or activities. Users can engage in text-based discussions, voice calls, or video chats within these channels. Understanding Discord Servers Discord servers are virtual spaces where users congregate to interact, share content, and build communities. Servers may revolve around gaming, hobbies, interests, or fandoms, providing a platform for like-minded individuals to connect. Communication Features Discord offers a range of communication tools, including text channels for messaging, voice channels for real-time audio conversations, and video channels for face-to-face interactions. These features facilitate seamless communication and collaboration. What Does NSFW Mean? The acronym NSFW stands for “Not Safe For Work,” indicating content that may be inappropriate for professional or public settings. NSFW Content NSFW content encompasses material that is sexually explicit, violent, or otherwise graphic in nature. It often includes nudity, profanity, or depictions of sensitive topics.

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

AWS Community Day CPH - Three problems of Terraform

AWS Community Day CPH - Three problems of Terraform

AWS Community Day CPH - Three problems of Terraform

Andrey Devyatkin

GenAI Risks & Security Meetup 01052024.pdf

GenAI Risks & Security Meetup 01052024.pdf

GenAI Risks & Security Meetup 01052024.pdf

Tata AIG General Insurance Company - Insurer Innovation Award 2024

Tata AIG General Insurance Company - Insurer Innovation Award 2024

Tata AIG General Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

The Good, the Bad and the Governed - Why is governance a dirty word? David O'Neill, Chief Operating Officer - APIContext Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

Partners Life - Insurer Innovation Award 2024

Partners Life - Insurer Innovation Award 2024

Partners Life - Insurer Innovation Award 2024

The Digital Insurer

How to Troubleshoot Apps for the Modern Connected Worker

How to Troubleshoot Apps for the Modern Connected Worker

How to Troubleshoot Apps for the Modern Connected Worker

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

Axa Assurance Maroc - Insurer Innovation Award 2024

Axa Assurance Maroc - Insurer Innovation Award 2024

Axa Assurance Maroc - Insurer Innovation Award 2024

The Digital Insurer

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

🐬 The future of MySQL is Postgres 🐘

🐬 The future of MySQL is Postgres 🐘

🐬 The future of MySQL is Postgres 🐘

As privacy and data protection regulations evolve rapidly, organizations operating in multiple jurisdictions face mounting challenges to ensure compliance and safeguard customer data. With state-specific privacy laws coming up in multiple states this year, it is essential to understand what their unique data protection regulations will require clearly. How will data privacy evolve in the US in 2024? How to stay compliant? Our panellists will guide you through the intricacies of these states' specific data privacy laws, clarifying complex legal frameworks and compliance requirements. This webinar will review: - The essential aspects of each state's privacy landscape and the latest updates - Common compliance challenges faced by organizations operating in multiple states and best practices to achieve regulatory adherence - Valuable insights into potential changes to existing regulations and prepare your organization for the evolving landscape

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

Imagine a world where information flows as swiftly as thought itself, making decision-making as fluid as the data driving it. Every moment is critical, and the right tools can significantly boost your organization’s performance. The power of real-time data automation through FME can turn this vision into reality. Aimed at professionals eager to leverage real-time data for enhanced decision-making and efficiency, this webinar will cover the essentials of real-time data and its significance. We’ll explore: FME’s role in real-time event processing, from data intake and analysis to transformation and reporting An overview of leveraging streams vs. automations FME’s impact across various industries highlighted by real-life case studies Live demonstrations on setting up FME workflows for real-time data Practical advice on getting started, best practices, and tips for effective implementation Join us to enhance your skills in real-time data automation with FME, and take your operational capabilities to the next level.

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Kürzlich hochgeladen (20)

A Domino Admins Adventures (Engage 2024)

A Domino Admins Adventures (Engage 2024)

A Domino Admins Adventures (Engage 2024)

Artificial Intelligence: Facts and Myths

Artificial Intelligence: Facts and Myths

Artificial Intelligence: Facts and Myths

Automating Google Workspace (GWS) & more with Apps Script

Automating Google Workspace (GWS) & more with Apps Script

Automating Google Workspace (GWS) & more with Apps Script

Manulife - Insurer Innovation Award 2024

Manulife - Insurer Innovation Award 2024

Manulife - Insurer Innovation Award 2024

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

AWS Community Day CPH - Three problems of Terraform

AWS Community Day CPH - Three problems of Terraform

AWS Community Day CPH - Three problems of Terraform

GenAI Risks & Security Meetup 01052024.pdf

GenAI Risks & Security Meetup 01052024.pdf

GenAI Risks & Security Meetup 01052024.pdf

Tata AIG General Insurance Company - Insurer Innovation Award 2024

Tata AIG General Insurance Company - Insurer Innovation Award 2024

Tata AIG General Insurance Company - Insurer Innovation Award 2024

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

Partners Life - Insurer Innovation Award 2024

Partners Life - Insurer Innovation Award 2024

Partners Life - Insurer Innovation Award 2024

How to Troubleshoot Apps for the Modern Connected Worker

How to Troubleshoot Apps for the Modern Connected Worker

How to Troubleshoot Apps for the Modern Connected Worker

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Axa Assurance Maroc - Insurer Innovation Award 2024

Axa Assurance Maroc - Insurer Innovation Award 2024

Axa Assurance Maroc - Insurer Innovation Award 2024

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

🐬 The future of MySQL is Postgres 🐘

🐬 The future of MySQL is Postgres 🐘

🐬 The future of MySQL is Postgres 🐘

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Couchbase Meetup Jan 2016

1.

2. Michael Kehoe Senior Site Reliability Engineer LinkedIn LinkedIn’s Big Data Pipeline with Kafka, Hadoop and Couchbase

3. 3 $ whoami Michael Kehoe • Sr Site Reliability Engineer (SRE) • Member of CBVT • B.E. (Electrical Engineering) from the University of Queensland, Australia

4. 4 Kafka @ LinkedIn • Kafka was created by LinkedIn • Kafka is a publish-subscribe system as a distributed commit log • Processes 500+ TB/ day (~500 billion messages)

5. 5 LinkedIn’s use of Kafka • Monitoring • Pub-Sub Messaging • Analytics • Building block for (log) distributed application • Samza • Espresso • Pinot

6. Kafka to Hadoop (Analytics) 6 Use Case • LinkedIn tracks data to better understand how members use our products • Information such as which page got viewed and which content got clicked on are sent into a Kafka cluster in each data center • Some of these events are all centrally collected and pushed onto our Hadoop grid for analysis and daily report generation

7. 7 Couchbase @ LinkedIn • About 80 separate services with one or more clusters in multiple data centers • Up to ~70 servers in a cluster • Single & Multi-tenant clusters

8. 8 Hadoop to Couchbase • Our primary use-case for Hadoop  Couchbase is for building (warming) / restoring Couchbase buckets • LinkedIn built it’s own in-house solution to work with our ETL processes etc

9. Jobs Cluster 9 Clusters & Numbers • Used for read-scaling, > 150k QPS, 27 node clusters • We use Hadoop to pre-build data by partition • Couchbase average latency is 2-3ms • 99th percentile is ~8 - 12ms

10. Questions? 10 Thank You

11. ©2014 LinkedIn Corporation. All Rights Reserved.