How PubSub Helped Build Vox Media’s Data Applications With Moses Musaelian | Current 2022

•

0 gefällt mir•336 views

The PubSub model enables organizations to effectively create systems modeled on the concept of event publishers and subscribers. Leveraging a messaging service like Kafka or GCP PubSub is a method of developing and enabling the data application lifecycle given high throughput conditions. This talk will discuss practical tips for architecting and productionalizing scalable and latent data applications that leverage the PubSub model. Attendees will learn about common data messaging capabilities found through the PubSub model and how to leverage PubSub to optimize the performance of data systems with low-latency requirements.

Technologie

How PubSub helped build Vox
Media’s multi-component,
high-volume data applications
Moses Musaelian | Vox Media
Current 2022

Introduction
● ML Engineer at Vox Media for over a year
● Working in Revenue Data Team

Google Pub/Sub - “message passing system for distributed applications in GCP”

Use Case #1: ML Service Design : “Latency Sensitive” Approach
Expect 800 bids per second!
Time out at 250ms!
“Lighten the load” off of our
front-facing API and let the
other components subscribe to
our message service

Use Case #2: DataFlow Logging (Viewability Data Application)
API Layer
Memorystore
Pubsub
Dataﬂow
BigQuery
Request Response

Use Case #3: Real Time Metrics from Pub/Sub (GCP Cloud Monitoring API v3)
Requirement → Don't write data to a
single time series faster than one point
each 5 seconds

Concluding Thoughts
API Layer
→ Metric send
→ Logging send
→ Worker send
API Layer
→ PubSub Send

Weitere ähnliche Inhalte

Ähnlich wie How PubSub Helped Build Vox Media’s Data Applications With Moses Musaelian | Current 2022

Faster, Simpler, Better - MongoDB to the rescue

MongoDB

AWS Lambda is a powerful and flexible tool for solving diverse business problems, from traditional grid computing to scheduled batch processing workflows. Cloud native solutions using AWS Lambda enable architectures that depart from traditional enterprise application design. These new design patterns can provide substantially increased performance and reduced costs. In this session, learn how Fannie Mae re-architected one of their mission-critical traditional grid computing applications to a modern serverless solution using AWS Lambda. Learn More: https://aws.amazon.com/government-education/

Serverless Design Patterns for Rethinking Traditional Enterprise Application ...

Amazon Web Services

IBM Interconnect 2016. This session outlines the offerings and initiatives that IBM provides around cloud and "as-a-service" messaging. We explain their roles and how they work together to deliver agility to business, while retaining the mission-critical reliability that enterprises have come to expect of IBM messaging. Topics include the work we are doing in IBM MQ Enterprise messaging to facilitate its deployment in public and private IaaS clouds, the use of MQ in Docker and how we are making it easier to build self-service deployments on-premise, the new MQ Light API and how it can be exploited from IBM Bluemix and "fast-speed of IT" systems of engagement, the MQ Light Service for IBM Bluemix and the work we are doing with the Apache Kafka project.

IBM Interconnect 2016 - Hybrid Cloud Messaging

Robert Nicholson

Google for モバイルアプリ 16:00: モバイル kpi 分析の新標準 fluentd + google big query

Google Cloud Platform - Japan

Watch this talk here: https://www.confluent.io/online-talks/introducing-events-and-stream-processing-nationwide-building-society Open Banking regulations compel the UK’s largest banks, and building societies to enable their customers to share personal information with other regulated companies securely. As a result companies such as Nationwide Building Society are re-architecting their processes and infrastructure around customer needs to reduce the risk of losing relevance and the ability to innovate. In this online talk, you will learn why, when facing Open Banking regulation and rapidly increasing transaction volumes, Nationwide decided to take load off their back-end systems through real-time streaming of data changes into Apache Kafka®. You will hear how Nationwide started their journey with Apache Kafka®, beginning with the initial use case of creating a real-time data cache using Change Data Capture, Confluent Platform and Microservices. Rob Jackson, Head of Application Architecture, will also cover how Confluent enabled Nationwide to build the stream processing backbone that is being used to re-engineer the entire banking experience including online banking, payment processing and mortgage applications. View now to: -Explore the technologies used by Nationwide to meet the challenges of Open Banking -Understand how Nationwide is using KSQL and Kafka Streams Framework to join topics and process data. -Learn how Confluent Platform can enable enterprises such as Nationwide to embrace the event streaming paradigm -See a working demo of the Nationwide system and what happens when the underlying infrastructure breaks.

Introducing Events and Stream Processing into Nationwide Building Society

confluent

Cloud computing infrastructure

sinhhn

Refactoring to Microservices

Jacinto Limjap

Wso2 con 2014 event driven architecture Publish/Subscribe Pubsub

John Mathon

Beeswax, which provides real time Bidder as a Service for programmatic digital advertising solutions, will talk about how they built a feature-rich, real-time streaming data solution on AWS using Amazon Kinesis, Amazon Redshift, Amazon S3, Amazon Data Pipeline. Beeswax will discuss key components of their solution including scalable data capture, messaging hub for archival, data warehousing, near real-time analytics, and real-time alerting.

Real-Time Streaming Data Solution on AWS with Beeswax

Amazon Web Services

This talk begins with a short recap of how we created systems over the past 20 years, up to the current idea of building systems, using a Microservices architecture. What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional REST APIs to integrate services with each eachother in a Microservices Architecture? Or is it better to use a more loosely-coupled protocol? Answers to these and many other questions are provided. The talk will show how a distributed log (event hub) can help to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk shows the difference between a request-driven and event-driven communication and answers when to use which. It highlights how a modern stream processing systems can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.

Building Event-Driven (Micro) Services with Apache Kafka

Guido Schmutz

Facing Open Banking regulation, rapidly increasing transaction volumes and increasing customer expectations, Nationwide took the decision to take load off their back-end systems through real-time streaming of data changes into Kafka. Hear about how Nationwide started their journey with Kafka, from their initial use case of creating a real-time data cache using Change Data Capture, Kafka and Microservices to how Kafka allowed them to build a stream processing backbone used to reengineer the entire banking experience including online banking, payment processing and mortgage applications. See a working demo of the system and what happens to the system when the underlying infrastructure breaks. Technologies covered include: Change Data Capture, Kafka (Avro, partitioning and replication) and using KSQL and Kafka Streams Framework to join topics and process data.

Introducing Events and Stream Processing into Nationwide Building Society (Ro...

confluent

Kochi mulesoft meetup 02

sumitahuja94

How can business processes be improved to use innovative new APIs quickly and easily? How can existing processes be published as APIs to easily be included as part of new cloud solutions? This session will answer these questions and present common business use cases. We will also discuss the Business Operations Connect and Process Connect aspects of IBM Connect family of solutions. This session goes into details about how a process designer would consume APIs using IBM BPM Process Designer and the REST and OpenAPI support in Process Connect. Come learn how to expose process APIs for reuse in innovative cloud solutions, and how to manage them in API Connect.

InterConnect 2017 HBP-3394-Enable innovative cloud solutions with IBM BPM and...

Brian Petrini

This session will begin with a short recap of how we created systems over the past 20 years, up to the current idea of building systems, using a Microservices architecture. What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional REST APIs to integrate services with each eachother in a Microservcies Architecture? Or is it better to use a more loosely-coupled protocol? Answers to these and many other questions are provided. The talk will show how a distributed log (event hub) can help to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk shows the difference between a request-driven and event-driven communication and answers when to use which. It highlights how a modern stream processing systems can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.

Microservices with Kafka Ecosystem

Guido Schmutz

Schnellere Digitalisierung mit einer cloudbasierten Datenstrategie

MongoDB

Sophisticated customers wanted to take truck positions, run algorithms to optimize routes, and change the driver’s job orders based on location and skill. To do this, Lucid Logistics needed to open access to the raw data so that companies could run their own business rules. With RabbitMQ, the development team is now in a position to connect incoming data to all different types of micro-services, push data into Hadoop, or integrate with other clouds like Amazon Web Services. To learn more, visit pivotal.io/platform-as-a-service/pivotal-cf.

Lucid logistics case study

VMware Tanzu

How the economist with cloud BI and Looker have improved data-driven decision...

Looker

Openbiz Cubi Case Study - Cymap

Zhaoyang Sun

Go Cloud Native with IBM Bluemix Developer Console - GIDS17

Vidyasagar Machupalli

Introduction to Google's Cloud Technologies

Chris Schalk

Ähnlich wie How PubSub Helped Build Vox Media’s Data Applications With Moses Musaelian | Current 2022 (20)

Faster, Simpler, Better - MongoDB to the rescue

Serverless Design Patterns for Rethinking Traditional Enterprise Application ...

IBM Interconnect 2016 - Hybrid Cloud Messaging

Google for モバイルアプリ 16:00: モバイル kpi 分析の新標準 fluentd + google big query

Introducing Events and Stream Processing into Nationwide Building Society

Cloud computing infrastructure

Refactoring to Microservices

Wso2 con 2014 event driven architecture Publish/Subscribe Pubsub

Real-Time Streaming Data Solution on AWS with Beeswax

Building Event-Driven (Micro) Services with Apache Kafka

Introducing Events and Stream Processing into Nationwide Building Society (Ro...

Kochi mulesoft meetup 02

InterConnect 2017 HBP-3394-Enable innovative cloud solutions with IBM BPM and...

Microservices with Kafka Ecosystem

Schnellere Digitalisierung mit einer cloudbasierten Datenstrategie

Lucid logistics case study

How the economist with cloud BI and Looker have improved data-driven decision...

Openbiz Cubi Case Study - Cymap

Go Cloud Native with IBM Bluemix Developer Console - GIDS17

Introduction to Google's Cloud Technologies

Mehr von HostedbyConfluent

"In this talk, attendees will be provided with an introduction to Kafka Connect and the basics of Single Message Transforms (SMTs) and how they can be used to transform data streams in a simple and efficient way. SMTs are a powerful feature of Kafka Connect that allow custom logic to be applied to individual messages as they pass through the data pipeline. The session will explain how SMTs work, the types of transformations they can be used for, and how they can be applied in a modular and composable way. Further, the session will discuss where SMTs fit in with Kafka Connect and when they should be used. Examples will be provided of how SMTs can be used to solve common data integration challenges, such as data enrichment, filtering, and restructuring. Attendees will also learn about the limitations of SMTs and when it might be more appropriate to use other tools or frameworks. Additionally, an overview of the alternatives to SMTs, such as Kafka Streams and KSQL, will be provided. This will help attendees make an informed decision about which approach is best for their specific use case. Whether attendees are developers, data engineers, or data scientists, this talk will provide valuable insights into how Kafka Connect and SMTs can help streamline data processing workflows. Attendees will come away with a better understanding of how these tools work and how they can be used to solve common data integration challenges."

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

How PubSub Helped Build Vox Media’s Data Applications With Moses Musaelian | Current 2022

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Ähnlich wie How PubSub Helped Build Vox Media’s Data Applications With Moses Musaelian | Current 2022

Ähnlich wie How PubSub Helped Build Vox Media’s Data Applications With Moses Musaelian | Current 2022 (20)

Mehr von HostedbyConfluent

Mehr von HostedbyConfluent (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

How PubSub Helped Build Vox Media’s Data Applications With Moses Musaelian | Current 2022