Дмитрий Лавриненко "Big & Fast Data for Identity & Telemetry services"

•

2 gefällt mir•260 views

- Business goal - What is Fast Data for us - What is Fast & Big Data solution - Reference Architecture - Data Science for Big Data - Technology Stack - Solution Architecture - Identity & Telemetry Data Processing Facts - Continuous Deployment - Quality Control

Technologie

Big & Fast Data
for Identity & Telemetry services

What is Fast Data for us?
• Continuous data loading
• Massively parallel processing
• Data consolidation
• Dimensional processing
• Data normalization & denormalization (depends on tech stack)
• Structured & dimensional data models
• Hybrid distributed warehouses

What is Fast & Big Data Solution
• Data WH
• Processing
• Analytics
• Visualisation
• Machine Learning
• Data Virtualization
• Data Ingestion
As a Service

Portal
Solution Architecture
Event
broker 
(Kafka)
data split
cache
retention
storage
scenarios
Master
Storage
(AVRO)
Batch layer
Speed layer
Serving layer
Consumer
Dashboard
Storage
(JSON)
Stream
Data
Warehouse
Hadoop
cluster
Spark
cluster
Ad-Hoc  
Queries
BI
Analytics
Visualization
engine
Telemetry
Identity
ML
ML

Reference Architecture
All Data
Real-time Data Processing
Data Acquisition and Storing
DataIntegration
Data Warehousing
Data Management
(Governance, Security, Quality, MDM)
Analytics
Reporting and
Analysis
Predictive
Modeling
Data Mining
Data Lake
(Landing, Exploration
and Archiving)
UX and
Visualization
Applications
Application data
Media data
Social data
Enterprise
content data
Telemetry
Other data
Customer
Analytics
Marketing
Analytics
Web/Mobile/Social
Analytics
IT Operational
Analytics
Fraud and Risk
Analytics
Complex Event
Processing
Real-time Query and
Search

Data Science for Big Data
Artificial
Intelligence
Machine Learning
High-Dimensional Data
Big Data
Apache Hadoop
Infrastructure
Data Collection
Data Augmentation
Real-time
Data Processing
Predictive
Analytics
Risk Analysis
Direct Marketing
Decision Support
Systems
Learning and
Intelligent Optimization
Data
Analysis
Data Exploration
Data Visualization
Business Intelligence

• ~ 100,000 Metrics & Events
• ~100,000 events per min
• 3TB per day
• JSON
• Independent Kafka clusters in scale
• Independent Spark Streaming in scale
• StreamSets
• Distributed HDFS
• Custom Analytics platform, cloud based.
• Machine learning on flight
Identity and Telemetry Data Processing: Facts

Continuous World
• Continuous Deployment is the actual delivery of features
and fixes to the customer as soon as they are ready
• Continuous Delivery represents a philosophy and a
commitment to ensuring that your code is always in a
release-ready state
• Continuous Integration allows automatically build and test
your software on a regular basis

Continuous Deployment
• Verify source
code
• Build
artifacts
• Run artifacts
• Run
automated
testing
• Publish
package
• Deploy
production

Quality control areas
• Application code
• Automated tests code
• Docker images
• Functional requirements
• Non-functional requirements
• Integrations

Thank you
Dmitry Lavrinenko, DevOps Solutions Architect

Weitere ähnliche Inhalte

Was ist angesagt?

Eugene Polonichko "Architecture of modern data warehouse"

Lviv Startup Club

The volume of available data is growing by the second (to an estimated 175 zetabytes by 2025), and it is becoming increasingly granular in its information. With that change every organization is moving towards building a data driven culture. We at Northwestern Mutual share similar story of driving towards making data driven decisions to improve both efficiency and effectiveness. Legacy system analysis revealed bottlenecks, excesses, duplications etc. Based on ever growing need to analyze more data our BI Team decided to make a move to more modern, scalable, cost effective data platform. As a financial company, data security is as important as ingestion of data. In addition to fast ingestion and compute we would need a solution that can support column level encryption, Role based access to different teams from our datalake. In this talk we describe our journey to move 100’s of ELT jobs from current MSBI stack to Databricks and building a datalake (using Lakehouse). How we reduced our daily data load time from 7 hours to 2 hours with capability to ingest more data. Share our experience, challenges, learning, architecture and design patterns used while undertaking this huge migration effort. Different sets of tools/frameworks built by our engineers to help ease the learning curve that our non-Apache Spark engineers would have to go through during this migration. You will leave this session with more understand on what it would mean for you and your organization if you are thinking about migrating to Apache Spark/Databricks.

Northwestern Mutual Journey – Transform BI Space to Cloud

Databricks

Data migration services

Nitai Partners Inc

Converging Database Transactions and Analytics

SingleStore

Snaplogic Live: Big Data in Motion

SnapLogic

2016 Spark Summit East Keynote: Ali Ghodsi and Databricks Community Edition demo

Databricks

Using Premium Data - for Business Analysts

Lynn Langit

Big data ingest frameworks ship with an array of connectors for common data origins and destinations, such as flat files, S3, HDFS, Kafka etc, but sometimes, you need to send data to, or receive data from a system that's not on the list. StreamSets includes template code for building your own connectors and processors; we'll walk through the process of building a simple destination that sends data to a REST web service, and show how it can be extended to target more sophisticated systems such as Salesforce Wave Analytics.

Building Custom Big Data Integrations

Pat Patterson

Narasimhan Sampath and Avinash Ramineni share how Choice Hotels International used Spark Streaming, Kafka, Spark, and Spark SQL to create an advanced analytics platform that enables business users to be self-reliant by accessing the data they need from a variety of sources to generate customer insights and property dashboards and enable data-driven decisions with minimal IT engagement. Narasimhan and Avinash highlight the architecture, lessons learned, and the challenges that were overcome on both the business and technology fronts. The analytics platform is designed as a framework to enable self-service data intake, data processing, and report/model generation by the business users. The data-driven framework consists of a distributed hybrid-cloud data ingestor for data intake and a Cloudera CDH cluster with Spark as the distributed compute engine. The solution is built in such a way that storage and compute have been decoupled and encourages the concept of BYOC (bring your own compute). The platform uses EC2 instances to run CDH and leverages Amazon S3 as a data warehouse storage layer (data lake), Spark as an ETL engine, and Spark SQL as a distributed query engine. Results (computations/derived tables) are exposed to the end users via Spark SQL and are discovered via Tableau. The platform supports both batch and streaming use cases and is built on the following technology stack: AWS (S3, EC2, SQS, SNS), Cloudera CDH (YARN, Navigator, Sentry), Spark, Kafka, Spark SQL, and Spark Streaming.

Strata+Hadoop World NY 2016 - Avinash Ramineni

Avinash Ramineni

In the online gaming industry we receive a vast amount of transactions that need to be handled in real time. Our customers get to choose from hundreds or even thousand options, and providing a seamless experience is crucial in our industry. Recommendation systems can be the answer in such cases but require handling loads of data and need to utilize large amounts of processing power. Towards this goal, in the last two years we have taken down the road of machine learning and AI in order to transform our customer’s daily experience and upgrade our internal services.

Personalization Journey: From Single Node to Cloud Streaming

Databricks

Big data is the new big thing where storing the data is the easy part. Gaining insights in your pile of data is something different. Based on a data dump of the well-known StackExchange websites, we will store & analyse 150+ GB of data with Azure Data Lake Store & Analytics to gain some insights about their users. After that we will use Power BI to give an at a glance overview of our learnings. If you are a developer that is interested in big data, this is your time to shine! We will use our existing SQL & C# skills to analyse everything without having to worry about running clusters.

Integration Monday - Analysing StackExchange data with Azure Data Lake

Tom Kerkhove

Community day ppt_kinesisv1.0

Sridevi Murugayen

SnapLogic Live: Big Data Integration

SnapLogic

Modern Data architecture Design

Kujambu Murugesan

Scaling to Infinity - Open Source meets Big Data

Treasure Data, Inc.

Data platform architecture

Sudheer Kondla

As big data and data warehousing scale-up and move into the cloud, they’re increasingly likely to be delivered as services using distributed cloud query engines such as Google BigQuery, loaded using streaming data pipelines and queried using BI tools such as Looker. In this session the presenter will walk through how data modelling and query processing works when storing petabytes of customer event-level activity in a distributed data store and query engine like BigQuery, how data ingestion and processing works in an always-on streaming data pipeline, how additional services such as Google Natural Language API can be used to classify for sentiment and extract entity nouns from incoming unstructured data, and how BI tools such as Looker and Google Data Studio bring data discovery and business metadata layers to cloud big data analytics

Budapest Data Forum 2017 - BigQuery, Looker And Big Data Analytics At Petabyt...

Rittman Analytics

Real-Time Analytics with Confluent and MemSQL

SingleStore

Data Architecture Brief Overview

Hal Kalechofsky

LogStash: Concept Run-Through

Manuj Aggarwal

Was ist angesagt? (20)

Eugene Polonichko "Architecture of modern data warehouse"

Northwestern Mutual Journey – Transform BI Space to Cloud

Data migration services

Converging Database Transactions and Analytics

Snaplogic Live: Big Data in Motion

2016 Spark Summit East Keynote: Ali Ghodsi and Databricks Community Edition demo

Using Premium Data - for Business Analysts

Building Custom Big Data Integrations

Strata+Hadoop World NY 2016 - Avinash Ramineni

Personalization Journey: From Single Node to Cloud Streaming

Integration Monday - Analysing StackExchange data with Azure Data Lake

Community day ppt_kinesisv1.0

SnapLogic Live: Big Data Integration

Modern Data architecture Design

Scaling to Infinity - Open Source meets Big Data

Data platform architecture

Budapest Data Forum 2017 - BigQuery, Looker And Big Data Analytics At Petabyt...

Real-Time Analytics with Confluent and MemSQL

Data Architecture Brief Overview

LogStash: Concept Run-Through

Ähnlich wie Дмитрий Лавриненко "Big & Fast Data for Identity & Telemetry services"

StreamCentral Technical Overview

Raheel Retiwalla

BDaas- BigData as a service

Agile Testing Alliance

Дмитрий Лавриненко "Blockchain for Identity Management, based on Fast Big Data"

Fwdays

So you got a handle on what Big Data is and how you can use it to find business value in your data. Now you need an understanding of the Microsoft products that can be used to create a Big Data solution. Microsoft has many pieces of the puzzle and in this presentation I will show how they fit together. How does Microsoft enhance and add value to Big Data? From collecting data, transforming it, storing it, to visualizing it, I will show you Microsoft’s solutions for every step of the way

How does Microsoft solve Big Data?

James Serra

Bringing the Power of Big Data Computation to Salesforce

Salesforce Developers

Maxim Tereschenko (BigData Lead, Provectus) with the talk "Analytical Systems Evolution - From Excel to Big Data Platforms and Data Lakes". Description: For the last ten years, analytical systems have changed dramatically. From Excel and Data Warehouses, we came to Big Data platforms and Data Lakes. It is no longer fantasy to communicate with the analytical system by voice or to wander in 3D glasses among the visualizations of the data. In scope of the speech, I want to follow this evolution, identify its main trends and fantasize about the future.

Analytical Systems Evolution: From Excel to Big Data Platforms and Data Lakes

Provectus

Prague data management meetup 2017-01-23

Martin Bém

Tapdata Product Intro

Tapdata

How to govern and secure a Data Mesh?

confluent

The introductory morning session will discuss big data challenges and provide an overview of the AWS Big Data Platform. We will also cover: • How AWS customers leverage the platform to manage massive volumes of data from a variety of sources while containing costs. • Reference architectures for popular use cases, including: connected devices (IoT), log streaming, real-time intelligence, and analytics. • The AWS big data portfolio of services, including Amazon S3, Kinesis, DynamoDB, Elastic MapReduce (EMR) and Redshift. • The latest relational database engine, Amazon Aurora - a MySQL-compatible, highly-available relational database engine which provides up to five times better performance than MySQL at a price one-tenth the cost of a commercial database. • Amazon Machine Learning – the latest big data service from AWS provides visualization tools and wizards that guide you through the process of creating machine learning (ML) models without having to learn complex ML algorithms and technology.

The AWS Big Data Platform – Overview

Amazon Web Services

Financial Services Analytics on AWS

Amazon Web Services

AzureDay - Introduction Big Data Analytics.

Łukasz Grala

From Data to Services at the Speed of Business

Ali Hodroj

This overview presentation discusses big data challenges and provides an overview of the AWS Big Data Platform by covering: - How AWS customers leverage the platform to manage massive volumes of data from a variety of sources while containing costs. - Reference architectures for popular use cases, including, connected devices (IoT), log streaming, real-time intelligence, and analytics. - The AWS big data portfolio of services, including, Amazon S3, Kinesis, DynamoDB, Elastic MapReduce (EMR), and Redshift. - The latest relational database engine, Amazon Aurora— a MySQL-compatible, highly-available relational database engine, which provides up to five times better performance than MySQL at one-tenth the cost of a commercial database. Created by: Rahul Pathak, Sr. Manager of Software Development

AWS Big Data Platform

Amazon Web Services

Data Platform Overview

Hamid J. Fard

Data Lake allows an organisation to store all of their data, structured and unstructured, in one, centralised repository. Since data can be stored as-is, there is no need to convert it to a predefined schema and you no longer need to know what questions you want to ask of your data beforehand. In this session we will explore the architecture of a Data Lake on AWS and cover topics such as storage, processing and security.

Building your Datalake on AWS

Amazon Web Services

Building IoT and Big Data Solutions on Azure

Ido Flatow

Real-time Microservices and In-Memory Data Grids

Ali Hodroj

Big data and the cloud are perfect partners for companies who want to unlock maximum value from all of their unstructured, semi-structured, and structured data. The challenge has been how to create and manage a reliable end-to-end solution that spans data ingestion, storage and analysis in the face of the volume, velocity and variety of big data sources. In this webinar, we will show you how to achieve big data bliss by combining StreamSets Data Collector, which specializes in creating and running complex any-to-any dataflows, with Microsoft's Azure Data Lake and Azure analytic solutions. We will walk through an example of how a major bank is using StreamSets to transport their on-premise data to the Azure Cloud Computing Platform and Azure Data Lake to take advantage of analytics tools with unprecedented scale and performance.

Enabling Next Gen Analytics with Azure Data Lake and StreamSets

Streamsets Inc.

Big Data Meetup: Analytical Systems Evolution

Provectus

Ähnlich wie Дмитрий Лавриненко "Big & Fast Data for Identity & Telemetry services" (20)

StreamCentral Technical Overview

BDaas- BigData as a service

Дмитрий Лавриненко "Blockchain for Identity Management, based on Fast Big Data"

How does Microsoft solve Big Data?

Bringing the Power of Big Data Computation to Salesforce

Analytical Systems Evolution: From Excel to Big Data Platforms and Data Lakes

Prague data management meetup 2017-01-23

Tapdata Product Intro

How to govern and secure a Data Mesh?

The AWS Big Data Platform – Overview

Financial Services Analytics on AWS

AzureDay - Introduction Big Data Analytics.

From Data to Services at the Speed of Business

AWS Big Data Platform

Data Platform Overview

Building your Datalake on AWS

Building IoT and Big Data Solutions on Azure

Real-time Microservices and In-Memory Data Grids

Enabling Next Gen Analytics with Azure Data Lake and StreamSets

Big Data Meetup: Analytical Systems Evolution

Mehr von Fwdays

"How Preply reduced ML model development time from 1 month to 1 day",Yevhen Y...

Fwdays

In my talk, I will tell about the world of GenAI services beyond GPT-wrappers and how we developed and scaled GenAI-centric applications. I'll share personal experiences about the obstacles, lessons, and strategic tools and methodologies that were key in taking GenAI applications from 0 to 1. I'll talk about the challenges we faced when launching LLM-based and image generative applications and delivering them to end users, and what conclusions and solutions were made.

"GenAI Apps: Our Journey from Ideas to Production Excellence",Danil Topchii

Fwdays

Python engineers are introduced to the transformative potential of Large Language Models (LLMs) in the realm of advanced data analysis and the application of Semantic Kernel techniques. We will talk about how LLMs like ChatGPT can be integrated into Python environments to automate data processing, enhance predictive modeling, and unlock deeper insights from complex datasets. The session will delve into practical strategies for embedding Semantic Kernel methods within Python projects, illustrating how these advanced techniques can refine the accuracy of machine learning models by embedding domain-specific knowledge directly into the analysis process. Attendees will leave with a clear roadmap for leveraging the combined power of LLMs and Semantic Kernels, equipped with actionable knowledge to drive innovation in their data analysis projects and beyond, marking a significant leap forward in the evolution of Python engineering practices.

"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...

Fwdays

"Federated learning: out of reach no matter how close",Oleksandr Lapshyn

Fwdays

"What is a RAG system and how to build it",Dmytro Spodarets

Fwdays

"Debugging python applications inside k8s environment", Andrii Soldatenko

Fwdays

"ML in Production",Oleksandr Bagan

Fwdays

Ever seen a code base where understanding a simple method meant jumping through tangled class hierarchies? We all have! And while "Favor composition over inheritance!" is almost as old as object-oriented programming, strictly avoiding all types of subclassing leads to verbose, un-Pythonic code. So, what to do? The discussion on composition vs. inheritance is so frustrating because far-reaching design decisions like this can only be made with the ecosystem in mind – and because there's more than one type of subclassing! Let's take a dogma-free stroll through the types of subclassing through a Pythonic lens and untangle some patterns and trade-offs together. By the end, you'll be more confident in deciding when subclassing will make your code more Pythonic and when composition will improve its clarity.

"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack

Fwdays

The current architecture of Prom.ua is built on microservices and GraphQL API, but it was not always like that. In this talk, I'll tell you how far we've come and how we've made using graphs in a microservice architecture convenient and simple. I will talk about the problems we faced and how we overcame them, made our development process more accessible, deployments faster, and the remains of the monolith less loaded.

"Distributed graphs and microservices in Prom.ua", Maksym Kindritskyi

Fwdays

ETL stands for extract, transform, load. It's a process that combines data from different sources into a single repository for further processing, analysis, and utilization. This talk provides an example of how pandas can be used to solve ETL tasks as a stage in the evolution of the data intake component. This involves preliminary validation, filtering, and conversion of data according to a set of business rules and internal representation, with intermediate combination with other sources.

"Rethinking the existing data loading and processing process as an ETL exampl...

Fwdays

I’m confident that many IT professionals are currently facing the same situation I was in a few months ago. Mobilization, uncertainty. How can I be maximally beneficial to the country with my experience and continue professional development in such circumstances? Since the onset of the full-scale invasion, I've been actively volunteering and assisting the army. Mobilization became the next logical step. I want to share: My journey in IT, volunteering, and the beginning of my service in the Armed Forces Impressions from the first few months Which Soft Skills are helpful in this context I aim to dispel myths about the mobilization process and projects of the Armed Forces. Address your questions And yes, military personnel can travel abroad during their leave.

"How Ukrainian IT specialist can go on vacation abroad without crossing the T...

Fwdays

The leader must be strong all the time. The leader cannot afford to make mistakes, let alone fail in front of their team. Is that really true? Nick Gicinto, a cybersecurity leader with over 25 years of experience, who has worked for the CIA and has built security systems from scratch at Tesla and Uber, fully hiring teams for these projects, will talk about the importance of being vulnerable to build trust within a team.

"The Strength of Being Vulnerable: the experience from CIA, Tesla and Uber", ...

Fwdays

Sharing open feedback can be difficult because it equals much work on yourself. However, feedback needs attention and a special place in the corporate culture. It helps to grow dynamically, build a team of like-minded people and achieve powerful results. In the presentation, I will talk about: The ability to work with feedback as a soft, solid skill in developing technical specialists. A list of difficulties that prevent quality work with feedback. The 4A Framework is a tool for successful giving and receiving feedback. I will also help specialists learn the following: Form constructive feedback and understand how and when to give it. Work analytically with the received feedback. Feel free to share your thoughts and be heard.

"[QUICK TALK] Radical candor: how to achieve results faster thanks to a cultu...

Fwdays

Will discuss: Current communication challenges, including mishaps and toxic versus productive interactions. Ever wondered about PDP? It’s likely because its relevance to career planning, even outside your current company, hasn’t been fully spotlighted. Exploring how PDP functions within career planning, applicable even if you’re eyeing an exit. “Who do I aspire to become?” Summarizing key points with a reference to a practical form you can download to use.

"[QUICK TALK] PDP Plan, the only one door to raise your salary and boost care...

Fwdays

This talk will reveal four destructive communication patterns that can undermine team spirit, reduce productivity and cause conflict, and offer effective strategies for neutralizing them. Let's start with exciting storytelling about a fictional team of developers working on Scrum. You will learn about situations that their team member noticed during team meetings. Next, we will analyze "The Gottman Four Horsemen" model, which describes the four "horsemen of the apocalypse" of work relationships: criticism, defensiveness, contempt, and stonewalling. For each of these patterns, specific "antidotes" will be offered that allow you to build healthier and more productive relationships in the team. Finally, we'll look at why this topic is critical to team productivity, drawing on Google's "Project Aristotle" research. Special attention will be paid to the concept of psychological safety, which is a key factor in the success of high-performance teams. This talk will not only provide valuable insights and tools for improving communication and management in Tech teams, but will also help each member better understand their own contribution to the overall success of the team.

"4 horsemen of the apocalypse of working relationships (+ antidotes to them)"...

Fwdays

We are all living in exceptionally turbulent times. Ukrainians are facing numerous crises and challenges, ranging from war to lay-offs. Work provides us with a sense of stability, which many of us deeply value and strive to maintain. However, there are times when we may not realize that our workload becomes overwhelming or that the stress we experience in our jobs becomes highly toxic. This often leads to burnout. During our discussion, we will cover: - The factors that contribute to burnout; - How to identify it; - Short-term strategies for addressing it; - Long-term approaches to finding fulfillment and satisfaction in our work; I will also share my personal experiences and insights, as well as those of my colleagues in the IT field. - Burnout - Stress

"Reconnecting with Purpose: Rediscovering Job Interest after Burnout", Anast...

Fwdays

As you grow in experience as a tech specialist and close the needs of Maslow's hierarchy, you realize that you can help beginners in your specialization. But how to do it effectively? Where to look for future mentees? How to avoid mistakes that demotivate both sides? And, in general, how can you benefit from investing your time in the success of a mentee? I'll discuss this based on my 1.5 years of experience mentoring beginners. It will be useful for professionals with at least six months of experience in IT and a willingness to share their knowledge with the community.

"Mentoring 101: How to effectively invest experience in the success of others...

Fwdays

"Mission (im) possible: How to get an offer in 2024?", Oleksandra Myronova

Fwdays

Historically, Ukrainians are not good at presenting themselves (read: selling) because they are not self-confident. But we were never taught to be confident or to understand our strengths, moreover to pitch ourselves well in 3 minutes at an interview. Therefore, my task is to teach you in a few minutes how to: Create your personal pitch correctly (sales points, structure) Understand, depending on the type of company, what managers want to hear about you at the interview Be able to package irrelevant or no experience Understand your strengths Create an example of a personal pitch No, there will be no information here on how to get more $ at the interview, but it's not sure 😂

"Why have we learned how to package products, but not how to 'package ourselv...

Fwdays

Imposter syndrome is a psychological phenomenon that most of us suffer from to one degree or another. I'm no exception and have faced this for most of my IT career, which has been around 10 years. I did not immediately understand that the impostor syndrome is one of the biggest blockers for my personal and career development. When I understood it, I gradually started, with small and large steps, fighting with a syndrome in all available ways. In my talk, I will share my story and how it helped me grow from an ordinary developer to a team leader and, eventually, the leader of the entire department.

"How to tame the dragon, or leadership with imposter syndrome", Oleksandr Zin...

Fwdays

Mehr von Fwdays (20)