Apache Hudi: Community-Driven Development

•

0 gefällt mir•61 views

Alluxio Day x APAC Modern Data Stack September 22, 2022 For more on Alluxio Day: https://www.alluxio.io/alluxio-day/ For more Alluxio events: https://alluxio.io/events/ Speaker: Shiyan Xu (Founding Member, OneHouse & Apache Hudi PMC) Apache Hudi’s open-source community is very active and healthy. In this talk, an overview of community-driven major features will be presented, followed by a deep-dive into two of those features, metastore and table management service, driven by Bytedance to illustrate Hudi’s platform vision.

Software

Apache Hudi:
Community-Driven
Development
Shiyan Xu
Engineering Manager @ Onehouse, Apache Hudi PMC

Agenda
● Hudi: quick overview - 1 min
● Community-driven features - 5 min
● MetaServer & Table Management Service - 18 min
● Closing - 1 min

1800+
Slack Members
225+
Contributors
1000+
GH Engagers
20+
Committers
Pre-installed on 5 cloud providers
Diverse PMC/Committers
1M DLs/month
(400% YoY)
800B+
Records/Day
(from even just 1
customer!)
Marquee OSS Users
Community Stats

- Slack Community
Notable newcomers in Q1 - 2022 include:
Total Slack Members:
Engagement Rate:
1.8k
12%
221
Active Members:
40
61
7.7%
8.5%
519
712
+157% YoY
+3.5pts YoY
+262% YoY

● RFC-51: Support Change-Data-Capture (Alibaba)
○ Capture and produce all changes made to the table
○ Implement debezium-like semantics
● RFC-42: Consistent hashing index (Alibaba)
○ Overcome throttling issues for large tables
● RFC-8: Record-level indexing mechanisms (Uber)
○ Improve indexing by direct mapping record keys to ﬁles
● RFC-48: Reduce write ampliﬁcation with Log-Compaction (Uber)
○ Balance write amp and I/O eﬃciency
Community-Driven RFCs

Community-Driven RFCs (cont’d)
● RFC-61: Snapshot view management (Shopee)
● RFC-52: Secondary index to improve query performance (Alibaba)
● RFC-60: Optimized storage layout for cloud object stores (AWS)
● RFC-12: Eﬃcient migration of large parquet tables to Hudi
(AWS/Onehouse)

MetaServer &
Table Management Service
MetaServer (RFC-36)
● Centralize the metadata management for tables
● Atomic sync: storage <-> metastore
TMS (RFC-43)
● Centralize the orchestration for running table services
● Decouple compute from the writer cluster

MetaServer: Storage
● Table
○ Path, Schema, properties, format, table type, etc
● Partition
○ Path, parameters
● Timeline
○ Instant: timestamp, action type, state
○ Instant metadata: updated partitions, ﬁles, write
statistics
● Snapshot
○ File name, size, creation timestamp

MetaServer: Other features
● Lock-provider
○ Storage supports Check-and-Set (CAS)
○ Supports multi-writer
● Schema evolution
○ Server stores schemas with versions and bind to
commits
○ Detect conﬂicts during commit
● Concurrency management
○ Conﬂict detection strategies: timestamp, table,
partition, ﬁlegroup, ﬁle

MetaServer &
Table Management Service
Proposers: Xiaoyu Geng (ByteDance), Zhaojing Yu (ByteDance)
Reviewers: Vinoth (Onehouse), Prasanna (OneHouse), Shiyan Xu (OneHouse)
Community engagements
● Submit RFCs
● Weekly sync meetings
● RFC reviews
● Implementation discussions & planning

Engage With Our Community
User Docs : https://hudi.apache.org
Technical Wiki : https://cwiki.apache.org/conﬂuence/display/HUDI
Github : https://github.com/apache/hudi/
Twitter : https://twitter.com/apachehudi
Mailing list(s) : dev-subscribe@hudi.apache.org (send an empty email to subscribe)
dev@hudi.apache.org (actual mailing list)
Slack : https://join.slack.com/t/apache-hudi/signup

Weitere ähnliche Inhalte

Ähnlich wie Apache Hudi: Community-Driven Development

During this webinar, we will review best practices and lessons learned from working with large and mid-size companies on their deployment of PostgreSQL. We will explore the practices that helped industry leaders move through these stages quickly, and get as much value out of PostgreSQL as possible without incurring undue risk. We have identified a set of levers that companies can use to accelerate their success with PostgreSQL: - Application Tiering - Collaboration between DBAs and Development Teams - Evangelizing - Standardization and Automation - Balance of Migration and New Development

PostgreSQL as a Strategic Tool

EDB

MyHeritage BackEnd group was built to scale to support 77 million users, 27 million family trees containing over 1.6 billion individuals, and over 6 billion historical documents. With big data comes big challenges and this presentation explains the structure, the methodology and the technologies that support scaling up. The presentation covers: • How cross R&D continuous deployment and R&D structure supports scalability • Sharding techniques • Cassandra usage at MyHeritage • Our search engine scaling structure

MyHeritage backend group - build to scale

Ran Levy

___________________________________________ Meetup#7 | Session 2 | 21/03/2018 | Taboola _____________________________________________ In this talk, we will present our multi-DC Kafka architecture, and discuss how we tackle sending and handling 10B+ messages per day, with maximum availability and no tolerance for data loss. Our architecture includes technologies such as Cassandra, Spark, HDFS, and Vertica - with Kafka as the backbone that feeds them all.

Distributed Kafka Architecture Taboola Scale

Apache Kafka TLV

Dutch Lotus User Group 2009 - Domino Tuning Presentation

Vladislav Tatarincev

Alluxio 2.0 & Near Real-time Big Data Platform w/ Spark & Alluxio

Alluxio, Inc.

Presto Strata Hadoop SJ 2016 short talk

kbajda

Mongo db admin_20110329

radiocats

Update on OpenTSDB and AsyncHBase

HBaseCon

YugaByte DB Internals - Storage Engine and Transactions

Yugabyte

Machbase Neo is an innovative iot data processing solution that integrates various features into an #all_in_one timeseries database. In the past, development organizations had to invest a lot of time and resources to build a single service or solution. Moreover, they had to navigate complex and challenging processes for data collection and processing. But now, with the introduction of Machbase Neo, these problems have been solved. You can now set up everything using just one Machbase Neo server, allowing developers to focus on their core tasks. This product can save developers over 90% of their time by eliminating unnecessary tasks.

MACHBASE_NEO

MACHBASE

Discover the evolution of Apache Hudi within the open-source realm - a community and project pushing the boundaries of data lake possibilities. This presentation delves into Apache Hudi 1.0, a pivotal release reimagining its transactional database layer while honoring its foundational principles. Join us in this transformative journey! Join the Apache Hudi Community https://join.slack.com/t/apache-hudi/shared_invite/zt-20r833rxh-627NWYDUyR8jRtMa2mZ~gg. Follow us on LinkedIn and Twitter https://www.linkedin.com/company/apache-hudi/ https://twitter.com/apachehudi

A Hudi Live Event: Shaping a Database Experience within the Data Lake with Ap...

nadine39280

Bquery Reporting & Analytics Architecture

Carst Vaartjes

WhereHows: Taming Metadata for 150K Datasets Over 9 Data Platforms

Mars Lan

Apache Impala is a modern, open-source MPP SQL engine architected from the ground up for the Hadoop data processing environment. Impala provides low latency and high concurrency for BI/analytic read-mostly queries on Hadoop, not delivered by batch frameworks such as Hive or SPARK. Impala is written from the ground up in C++ and Java. It maintains Hadoop’s flexibility by utilizing standard components (HDFS, HBase, Metastore, Sentry) and is able to read the majority of the widely-used file formats (e.g. Parquet, Avro, RCFile). To reduce latency, such as that incurred from utilizing MapReduce or by reading data remotely, Impala implements a distributed architecture based on daemon processes that are responsible for all aspects of query execution and that run on the same machines as the rest of the Hadoop infrastructure. Impala employs runtime code generation using LLVM in order to improve execution times and uses static and dynamic partition pruning to significantly reduce the amount of data accessed. The result is performance that is on par or exceeds that of commercial MPP analytic DBMSs, depending on the particular workload. Although initially designed for running on-premises against HDFS-stored data, Impala can also run on public clouds and access data stored in various storage engines such as object stores (e.g. AWS S3), Apache Kudu and HBase. In this talk, we present Impala's architecture in detail and discuss the integration with different storage engines and the cloud.

Performance Optimizations in Apache Impala

Cloudera, Inc.

Born at Facebook, Presto is an open source high performance, distributed SQL query engine. With the disaggregation of storage and compute, Presto was created to simplify querying of all data lakes - cloud data lakes like S3 and on premise data lakes like HDFS. Presto's high performance and flexibility has made it a very popular choice for interactive query workloads on large Hadoop-based clusters as well as AWS S3, Google Cloud Storage and Azure blob store. Today it has grown to support many users and use cases including ad hoc query, data lake house analytics, and federated querying. In this session, we will give an overview on Presto including architecture and how it works, the problems it solves, and most common use cases. We'll also share the latest innovation in the project as well as the future roadmap.

Presto – Today and Beyond – The Open Source SQL Engine for Querying all Data...

Dipti Borkar

Slack in the Age of Prometheus

George Luong

Smartsheet’s Transition to Snowflake and Databricks: The Why and Immediate Im...

Databricks

Speakers: Ravi Dubey, Senior Manager, Software Engineering, Capital One + Jeff Sharpe, Software Engineer, Capital One Capital One supports interactions with real-time streaming transactional data using Apache Kafka®. Kafka helps deliver information to internal operation teams and bank tellers to assist with assessing risk and protect customers in a myriad of ways. Inside the bank, Kafka allows Capital One to build a real-time system that takes advantage of modern data and cloud technologies without exposing customers to unnecessary data breaches, or violating privacy regulations. These examples demonstrate how a streaming platform enables Capital One to act on their visions faster and in a more scalable way through the Kafka solution, helping establish Capital One as an innovator in the banking space. Join us for this online talk on lessons learned, best practices and technical patterns of Capital One’s deployment of Apache Kafka. -Find out how Kafka delivers on a 5-second service-level agreement (SLA) for inside branch tellers. -Learn how to combine and host data in-memory and prevent personally identifiable information (PII) violations of in-flight transactions. -Understand how Capital One manages Kafka Docker containers using Kubernetes. Watch the recording: https://videos.confluent.io/watch/6e6ukQNnmASwkf9Gkdhh69?.

Capital One Delivers Risk Insights in Real Time with Stream Processing

confluent

Migration to ClickHouse. Practical guide, by Alexander Zaitsev

Altinity Ltd

Introducing TiDB [Delivered: 09/27/18 at NYC SQL Meetup]

Kevin Xu

Ähnlich wie Apache Hudi: Community-Driven Development (20)

PostgreSQL as a Strategic Tool

MyHeritage backend group - build to scale

Distributed Kafka Architecture Taboola Scale

Dutch Lotus User Group 2009 - Domino Tuning Presentation

Alluxio 2.0 & Near Real-time Big Data Platform w/ Spark & Alluxio

Presto Strata Hadoop SJ 2016 short talk

Mongo db admin_20110329

Update on OpenTSDB and AsyncHBase

YugaByte DB Internals - Storage Engine and Transactions

MACHBASE_NEO

A Hudi Live Event: Shaping a Database Experience within the Data Lake with Ap...

Bquery Reporting & Analytics Architecture

WhereHows: Taming Metadata for 150K Datasets Over 9 Data Platforms

Performance Optimizations in Apache Impala

Presto – Today and Beyond – The Open Source SQL Engine for Querying all Data...

Slack in the Age of Prometheus

Smartsheet’s Transition to Snowflake and Databricks: The Why and Immediate Im...

Capital One Delivers Risk Insights in Real Time with Stream Processing

Migration to ClickHouse. Practical guide, by Alexander Zaitsev

Introducing TiDB [Delivered: 09/27/18 at NYC SQL Meetup]

Mehr von Alluxio, Inc.

As GenAI and AI continue to transform businesses, scaling these workloads requires optimized underlying infrastructure. A multi-cloud architecture allows organizations to leverage different cloud services to meet diverse workload demands while maximizing efficiency, reducing costs, and avoiding vendor lock-in. However, achieving a multi-cloud vision can be challenging. In this webinar, Tarik will share how an agonistic data layer, like Alluxio, allows you to embrace the separation of storage from compute and simplify the adoption of multi-cloud for AI. - Learn why leveraging multiple cloud providers is critical for balancing performance, scalability, and cost of your AI platform - Discover how an agnostic data layer like Alluxio provides seamless data access in multi-cloud that bridges storage and compute without data replication - Gain insights into real-world examples and best practices for deploying AI across on-prem, hybrid, and multi-cloud environments

Alluxio Monthly Webinar | Cloud-Native Model Training on Distributed Data

Alluxio, Inc.

Optimizing Data Access for Analytics And AI with Alluxio

Alluxio, Inc.

Speed Up Presto at Uber with Alluxio Caching

Alluxio, Inc.

Alluxio x Tobiko - ETL Happy Hour April 16, 2024 For more Alluxio events: https://alluxio.io/events/ Speaker: Toby Mao (CTO @ Tobiko Data) Writing efficient and correct incremental pipelines is challenging. Data practitioners who take on this challenge are viewed as performing an "advanced" function, which discourages broader teams from adopting incremental loads. In this lightning talk, CTO of Tobiko Data, Toby Mao, will demystify incremental loading data at scale.

Correctly Loading Incremental Data at Scale

Alluxio, Inc.

Big Data Bellevue Meetup March 21, 2024 For more Alluxio events: https://alluxio.io/events/ Speakers: Bin Fan (VP of Open Source, Alluxio) In this presentation, Bin Fan (VP of Open Source @ Alluxio) will address a critical challenge of optimizing data loading for distributed Python applications within AI/ML workloads in the cloud, focusing on popular frameworks like Ray and Hugging Face. Integration of Alluxio’s distributed caching for Python applications is accomplished using the fsspec interface, thus greatly improving data access speeds. This is particularly useful in machine learning workflows, where repeated data reloading across slow, unstable or congested networks can severely affect GPU efficiency and escalate operational costs. Attendees can look forward to practical, hands-on demonstrations showcasing the tangible benefits of Alluxio’s caching mechanism across various real-world scenarios. These demos will highlight the enhancements in data efficiency and overall performance of data-intensive Python applications. This presentation is tailored for developers and data scientists eager to optimize their AI/ML workloads. Discover strategies to accelerate your data processing tasks, making them not only faster but also more cost-efficient.

Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML

Alluxio, Inc.

Alluxio Monthly Webinar Feb. 27, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Tarik Bennett (Senior Solutions Engineer, Alluxio) As GenAI and AI continue to transform businesses, scaling these workloads requires optimized underlying infrastructure. A multi-cloud architecture allows organizations to leverage different cloud services to meet diverse workload demands while maximizing efficiency, reducing costs, and avoiding vendor lock-in. However, achieving a multi-cloud vision can be challenging. In this webinar, Tarik will share how an agonistic data layer, like Alluxio, allows you to embrace the separation of storage from compute and simplify the adoption of multi-cloud for AI. - Learn why leveraging multiple cloud providers is critical for balancing performance, scalability, and cost of your AI platform - Discover how an agnostic data layer like Alluxio provides seamless data access in multi-cloud that bridges storage and compute without data replication - Gain insights into real-world examples and best practices for deploying AI across on-prem, hybrid, and multi-cloud environments

Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...

Alluxio, Inc.

Alluxio Monthly Webinar Jan. 30, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Kevin Petrie (VP of Research, Eckerson Group) - Omid Razavi (SVP of Customer Success, Alluxio) 2024 is gearing up to be an impactful year for AI and analytics. Join us on January 30, as Kevin Petrie (VP of Research at Eckerson Group) and Omid Razavi (SVP of Customer Success at Alluxio) share key trends that data and AI leaders should know. This event will efficiently guide you with market data and expert insights to drive successful business outcomes. - Assess current and future trends in data and AI with industry experts - Discover valuable insights and practical recommendations - Learn best practices to make your enterprise data more accessible for both analytics and AI applications

Alluxio Monthly Webinar | Five Disruptive Trends that Every Data & AI Leader...

Alluxio, Inc.

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Juncheng Yang(Ph.D Candidate, @CMU) As a cache eviction algorithm, FIFO has a lot of attractive properties, such as simplicity, speed, scalability, and flash-friendliness. The most prominent criticism of FIFO is its low efficiency (high miss ratio). In this talk, I will describe a simple, scalable FIFO-based algorithm with three static queues (S3-FIFO). Evaluated on 6594 cache traces from 14 datasets, we show that S3- FIFO has lower miss ratios than state-of-the-art algorithms across traces. Moreover, S3-FIFO’s efficiency is robust — it has the lowest mean miss ratio on 10 of the 14 datasets. FIFO queues enable S3-FIFO to achieve good scalability with 6× higher throughput compared to optimized LRU at 16 threads. Our insight is that most objects in skewed workloads will only be accessed once in a short window, so it is critical to evict them early (also called quick demotion). The key of S3-FIFO is a small FIFO queue that filters out most objects from entering the main cache, which provides a guaranteed demotion speed and high demotion precision.

Data Infra Meetup | FIFO Queues are All You Need for Cache Eviction

Alluxio, Inc.

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Jingwen Ouyang (Product Manager, @Alluxio) In this session, Jingwen presents an overview of using Alluxio Edge caching to accelerate Trino or Presto queries. She offers practical best practices for using distributed caching with compute engines. In addition, this session also features insights from real-world examples.

Data Infra Meetup | Accelerate Your Trino/Presto Queries - Gain the Alluxio Edge

Alluxio, Inc.

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Siyuan Sheng (Senior Software Engineer, @Alluxio) - Chunxu Tang (Research Scientist, @Alluxio) In this session, cloud optimization specialists Chunxu and Siyuan break down the challenges and present a fresh architecture designed to optimize I/O across the data pipeline, ensuring GPUs function at peak performance. The integrated solution of PyTorch/Ray + Alluxio + S3 offers a promising way forward, and the speakers delve deep into its practical applications. Attendees will not only gain theoretical insights but will also be treated to hands-on instructions and demonstrations of deploying this cutting-edge architecture in Kubernetes, specifically tailored for Tensorflow/PyTorch/Ray workloads in the public cloud.

Data Infra Meetup | Accelerate Distributed PyTorch/Ray Workloads in the Cloud

Alluxio, Inc.

Data Infra Meetup | ByteDance's Native Parquet Reader

Alluxio, Inc.

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Jing Zhao (Principal Engineer, @Uber) Uber builds one of the biggest data lakes in the industry, which stores exabytes of data. In this talk, we will introduce the evolution of our data storage architecture, and delve into multiple key initiatives during the past several years. Specifically, we will introduce: - Our on-prem HDFS cluster scalability challenges and how we solved them - Our efficiency optimizations that significantly reduced the storage overhead and unit cost without compromising reliability and performance - The challenges we are facing during the ongoing Cloud migration and our solutions

Data Infra Meetup | Uber's Data Storage Evolution

Alluxio, Inc.

Alluxio Monthly Webinar Nov. 15, 2023 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Tarik Bennett (Senior Solutions Engineer) - Beinan Wang (Senior Staff Engineer & Architect) Many companies are working with development architectures for AI platforms but have concerns about efficiency at scale as data volumes increase. They use centralized cloud data lakes, like S3, to store training data for AI platforms. However, GPU shortages add more complications. Storage and compute can be separate, or even remote, making data loading slow and expensive: 1) Optimizing a developmental setup can include manual copies, which are slow and error-prone 2) Directly transferring data across regions or from cloud to on-premises can incur expensive egress fees This webinar covers solutions to improve data loading for model training. You will learn: - The data loading challenges with distributed infrastructure - Typical solutions, including NFS/NAS on object storage, and why they are not the best options - Common architectures that can improve data loading and cost efficiency - Using Alluxio to accelerate model training and reduce costs

Alluxio Monthly Webinar | Why NFS/NAS on Object Storage May Not Solve Your AI...

Alluxio, Inc.

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Adit Madan (Director of Product Management, @Alluxio) In this session, Adit Madan, Director of Product Management at Alluxio, presents an overview of using distributed caching to accelerate model training and serving. He explores the requirements of data access patterns in the ML pipeline and offers practical best practices for using distributed caching in the cloud. This session features insights from real-world examples, such as AliPay, Zhihu, and more.

AI Infra Day | Accelerate Your Model Training and Serving with Distributed Ca...

Alluxio, Inc.

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Bin Fan (Cheif Architect, VP of Open Source, @Alluxio) As the AI landscape rapidly evolves, the advancements in generative AI technologies, such as ChatGPT, are driving a need for a robust AI infra stack. This opening keynote will explore the key trends of the AI infra stack in the generative AI era.

AI Infra Day | The AI Infra in the Generative AI Era

Alluxio, Inc.

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Lu Qiu (Machine Learning Engineer, @Alluxio) - Shawn Sun (Software Engineer, @Alluxio) This hands-on session will discuss best practices for using PyTorch and Alluxio during model training on AWS. Chunxu and Lu will provide a step-by-step demonstration of how to use Alluxio on EKS as a distributed cache to accelerate computer vision model training jobs that read datasets from S3. This architecture significantly improves the utilization of GPUs from 30% to 90%+, archives ~5x faster training, and lower cloud storage costs.

AI Infra Day | Hands-on Lab: CV Model Training with PyTorch & Alluxio on Kube...

Alluxio, Inc.

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Jordan Plawner (Global Director of Artificial intelligence Product Management and Strategy, @Intel) ChatGPT and other massive models represents an amazing step forward in AI, yet they do not solve real-world business problems. We will survey how the AI ecosystem has worked non-stop over this last year to take these all-purpose multi-task models and optimize them to they can be used by organizations to address domain specific problems. We will explain these new AI-for-the-real world techniques and methods such as fine tuning and how can be applied to deliver results which are highly performant with state-of-the-art accuracy while also being economical to build and deploy everywhere to enhance products and services.

AI Infra Day | The Generative AI Market And Intel AI Strategy and Product Up...

Alluxio, Inc.

AI Infra Day | Composable PyTorch Distributed with PT2 @ Meta

Alluxio, Inc.

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Sally (Mihyoung) Lee (Senior Staff Engineer, TLM, @Uber) Machine learning models power Uber’s everyday business. However, developing and deploying a model is not a one-time event but a continuous process that requires careful planning, execution, and monitoring. This session will highlight Uber’s practice on the machine learning lifecycle to ensure high model quality.

AI Infra Day | Model Lifecycle Management Quality Assurance at Uber Scale

Alluxio, Inc.

Alluxio Monthly Webinar Oct. 3, 2023 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Greg Palmer (Lead Solutions Engineer) Model training requires extensive computational and GPU resources. When training models on AWS, loading data from S3 often becomes a major bottleneck, wasting valuable GPU cycles. Optimizing data loading can greatly reduce GPU idle time and increase GPU utilization. In this webinar, Greg Palmer will discuss best practices for efficient data loading during model training on AWS. He will demonstrate how to use Alluxio on EKS as a distributed cache to accelerate PyTorch training jobs that read datasets from S3. This architecture significantly improves the utilization of GPUs from 30% to 90%+, archives ~5x faster training, and lower cloud storage costs. What you will learn: - The challenges of feeding data-hungry GPUs in the cloud - How to accelerate model training by optimizing data loading on AWS - The reference architecture for running PyTorch jobs with Alluxio cache on EKS while reading data from S3, with benchmark results of training ResNet50 and BERT - How to use TensorBoard to identify bottlenecks in GPU utilization

Alluxio Monthly Webinar | Efficient Data Loading for Model Training on AWS

Alluxio, Inc.

Mehr von Alluxio, Inc. (20)

Alluxio Monthly Webinar | Cloud-Native Model Training on Distributed Data

Optimizing Data Access for Analytics And AI with Alluxio

Speed Up Presto at Uber with Alluxio Caching

Correctly Loading Incremental Data at Scale

Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML

Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...

Alluxio Monthly Webinar | Five Disruptive Trends that Every Data & AI Leader...

Data Infra Meetup | FIFO Queues are All You Need for Cache Eviction

Data Infra Meetup | Accelerate Your Trino/Presto Queries - Gain the Alluxio Edge

Data Infra Meetup | Accelerate Distributed PyTorch/Ray Workloads in the Cloud

Data Infra Meetup | ByteDance's Native Parquet Reader

Data Infra Meetup | Uber's Data Storage Evolution

Alluxio Monthly Webinar | Why NFS/NAS on Object Storage May Not Solve Your AI...

AI Infra Day | Accelerate Your Model Training and Serving with Distributed Ca...

AI Infra Day | The AI Infra in the Generative AI Era

AI Infra Day | Hands-on Lab: CV Model Training with PyTorch & Alluxio on Kube...

AI Infra Day | The Generative AI Market And Intel AI Strategy and Product Up...

AI Infra Day | Composable PyTorch Distributed with PT2 @ Meta

AI Infra Day | Model Lifecycle Management Quality Assurance at Uber Scale

Alluxio Monthly Webinar | Efficient Data Loading for Model Training on AWS

Kürzlich hochgeladen

OpenChain - The Ramifications of ISO/IEC 5230 and ISO/IEC 18974 for Legal Pro...

Shane Coughlan

%in Midrand+277-882-255-28 abortion pills for sale in midrand

masabamasaba

We specialize in Psychic Readings, Psychic Love Spells, Binding Love Spells, Obsession Spells, Voodoo Spells, Lottery Spells, Marriage Spells, Black Magic Spells, Palm Readings & much more. Are you depressed? We perform this come-to-me love spell that works instantly with the aim of bringing back the victim to the person performing the magic. Have you lost your lover? We perform this come-to-me love spell that works instantly with the aim of bringing back the victim to the person performing the magic. Have you lost your lover? Do u need to solve any relationship problem? Contact the powerful spells caster chief kule with love spells that work overnight and love spells that really work. Have you found yourself infatuated with a special someone you think could be the one? Are you looking for a spell to provide them with a nudge in the right direction? Or maybe the spell you cast didn’t achieve the results you were hoping for? Whether you’re new or versed in the ways of spell casting, we’re here to help. Today we’re going to provide you with a detailed guide on the types of love spells to cast. Not only that but there’s something for those who wish to find outside advice from more advanced spell casters. We’re also going to provide you with the top sites available to help you with your dilemma. Let’s begin our journey by educating ourselves on love magic and what a real love caster looks like. Love Magic and Love Casters Love magic made its first appearance back in Ancient Egypt and has been an active practice since. This type of magic is a branch of traditional magic and can be practiced in various ways. Typically the more common use of love magic is through the work of spells, but other methods look like Charms Rituals-LOVE Potions-Dolls and even Amulets If you are interested in becoming a love caster, be prepared for what’s to come. A genuine love caster knows that the art of love casting is no easy feat and shouldn’t be done casually. You should know that not only does it require you to be gifted spiritually, but you must be ready to serve others. Someone who is considered a real love caster has experience in all manner of spells, no matter the difficulty. Training yourself in attraction, commitment, and marriage spells is an excellent place to start. But this by no means will make you a professional. Practice your craft and expand your knowledge; understand that you will possess the ability to help others in time truly. Types of Love Spells What better way to start broadening your experiences with love spells than by learning more about them? These spells work like just about any other spell. Simply apply your intention, use a medium (sigils, mantras, candles, or charm bags), and top it off with establishing the belief that you will receive what you want. So what kind of spells are available and which ones suit your needs the best? Let’s take a look at the many options you have at your disposal. Attraction Spells

%+27788225528 love spells in Toronto Psychic Readings, Attraction spells,Brin...

masabamasaba

Direct Style Effect Systems -The Print[A] Example- A Comprehension Aid

Philip Schwarz

%in kempton park+277-882-255-28 abortion pills for sale in kempton park

masabamasaba

Artyushina_Guest lecture_YorkU CS May 2024.pptx

AnnaArtyushina1

Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...

Bert Jan Schrijver

WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation

WSO2

%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...

masabamasaba

WSO2CON 2024 - WSO2's Digital Transformation Journey with Choreo: A Platforml...

WSO2

%in tembisa+277-882-255-28 abortion pills for sale in tembisa

masabamasaba

Investing in AI transformation today The modern business advantage: Uncovering deep insights with AI Organizations around the world have come to recognize AI as the transformative technology that enables them to gain real business advantage. AI’s ability to organize vast quantities of data allows those who implement it to uncover deep business insights, augment human expertise, drive operational efficiency, transform their products, and better serve their customers

Microsoft AI Transformation Partner Playbook.pdf

Willy Marroquin (WillyDevNET)

VTU technical seminar 8Th Sem on Scikit-learn

AmarnathKambale

Abortion Pills In Pretoria ](+27832195400*)[ 🏥 Women's Abortion Clinic In Pretoria ● Abortion Pills For Sale In Pretoria ● Pretoria 🏥🚑!! Abortion Clinic Near Me Cost, Price, Women's Clinic Near Me, Abortion Clinic Near, Abortion Doctors Near me, Abortion Services Near Me, Abortion Pills Over The Counter, Abortion Pill Doctors' Offices, Abortion Clinics, Abortion Places Near Me, Cheap Abortion Places Near Me, Medical Abortion & Surgical Abortion, approved cyctotec pills and womb cleaning pills too plus all the instructions needed This Discrete women’s Termination Clinic offers same day services that are safe and pain free, we use approved pills and we clean the womb so that no side effects are present. Our main goal is that of preventing unintended pregnancies and unwanted births every day to enable more women to have children by choice, not chance. We offer Terminations by Pill and The Morning After Pill.” Our Private VIP Abortion Service offers the ultimate in privacy, efficiency and discretion. we do safe and same day termination and we do also womb cleaning as well its done from 1 week up to 28 weeks. We do delivery of our services world wide SAFE ABORTION CLINICS/PILLS ON SALE WE DO DELIVERY OF PILLS ALSO Abortion clinic at very low costs, 100% Guaranteed and it’s safe, pain free and a same day service. It Is A 45 Minutes Procedure, we use tested abortion pills and we do womb cleaning as well. Alternatively the medical abortion pill and womb cleansing !!!

Abortion Pills In Pretoria ](+27832195400*)[ 🏥 Women's Abortion Clinic In Pre...

Medical / Health Care (+971588192166) Mifepristone and Misoprostol tablets 200mg

%in Harare+277-882-255-28 abortion pills for sale in Harare

masabamasaba

This presentation covers the following topics: What is logging? The purpose of logging: Debugging The purpose of logging: Security The purpose of logging: Stats & analytics Traditional logging Traditional logging: Advantages Traditional logging: Disadvantages The solution: Large-scale logging Large-scale logging: Core principles Large-scale logging: Solution types Large-scale logging: Cloud vs on-prem Large-scale logging: Operational complexity Large-scale logging: Security Large-scale logging: Costs Large-scale logging: On-prem comparison - Elasticsearch - Grafana Loki - VictoriaLogs On-prem comparison: Setup and operation On-prem comparison: Costs On-prem comparison: Full-text search support On-prem comparison: How to efficiently query 100TB of logs? On-prem comparison: Integration with CLI tools VictoriaLogs for large-scale logging VictoriaLogs demo instance - Ingestion rate: 3600 messages / minute - The number of log messages: 1.1 billion - Uncompressed log messages’ size: 1.5TB - Compressed log messages’ size: 23GB - Compression ratio: 47x - Memory usage: 150MB VictoriaLogs CLI integration demo - Which errors have occurred in all the apps during the last hour? - How many errors have occurred during the last hour? - Which apps generated the most of errors during the last hour? - The number of per-minute errors for the last 10 minutes - Status codes for the last hour - Non-200 status codes for the last week - Top client IPs for the last 4 weeks with 404 and 500 response status codes - Per-month stats for the given IP across all the logs Large-scale logging solution MUST provide excellent CLI integration VictoriaLogs: (temporary) drawbacks VictoriaLogs: Recap - Easy to setup and operate - The lowest RAM usage and disk space usage (up to 30x less than Elasticsearch and Grafana Loki) - Fast full-text search - Excellent integration with traditional command-line tools for log analysis - Accepts logs from popular log shippers (Filebeat, Fluentbit, Logstash, Vector, Promtail, Grafana Agent) - Open source and free to use!

Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024

VictoriaMetrics

Abortion Pill Prices Tembisa [(+27832195400*)] 🏥 Women's Abortion Clinic in Tembisa ● Abortion Pills For Sale in Tembisa ● Tembisa 🏥🚑!! Abortion Clinic Near Me Cost, Price, Women's Clinic Near Me, Abortion Clinic Near, Abortion Doctors Near me, Abortion Services Near Me, Abortion Pills Over The Counter, Abortion Pill Doctors' Offices, Abortion Clinics, Abortion Places Near Me, Cheap Abortion Places Near Me, Medical Abortion & Surgical Abortion, approved cyctotec pills and womb cleaning pills too plus all the instructions needed This Discrete women’s Termination Clinic offers same day services that are safe and pain free, we use approved pills and we clean the womb so that no side effects are present. Our main goal is that of preventing unintended pregnancies and unwanted births every day to enable more women to have children by choice, not chance. We offer Terminations by Pill and The Morning After Pill.” Our Private VIP Abortion Service offers the ultimate in privacy, efficiency and discretion. we do safe and same day termination and we do also womb cleaning as well its done from 1 week up to 28 weeks. We do delivery of our services world wide SAFE ABORTION CLINICS/PILLS ON SALE WE DO DELIVERY OF PILLS ALSO Abortion clinic at very low costs, 100% Guaranteed and it’s safe, pain free and a same day service. It Is A 45 Minutes Procedure, we use tested abortion pills and we do womb cleaning as well. Alternatively the medical abortion pill and womb cleansing !!!

Abortion Pill Prices Tembisa [(+27832195400*)] 🏥 Women's Abortion Clinic in T...

Medical / Health Care (+971588192166) Mifepristone and Misoprostol tablets 200mg

tonesoftg

lanshi9

We specialize in Psychic Readings, Psychic Love Spells, Binding Love Spells, Obsession Spells, Voodoo Spells, Lottery Spells, Marriage Spells, Black Magic Spells, Palm Readings & much more. Are you depressed? We perform this come-to-me love spell that works instantly with the aim of Winnipeg back the victim to the person performing the magic. Have you lost your lover? We perform this come-to-me love spell that works instantly with the aim of bringing back the victim to the person performing the magic. Have you lost your lover? Do u need to solve any relationship problem? Contact the powerful spells caster chief kule with love spells that work overnight and love spells that really work. Have you found yourself infatuated with a special someone you think could be the one? Are you looking for a spell to provide them with a nudge in the right direction? Or maybe the spell you cast didn’t achieve the results you were hoping for? Whether you’re new or versed in the ways of spell casting, we’re here to help. Today we’re going to provide you with a detailed guide on the types of love spells to cast. Not only that but there’s something for those who wish to find outside advice from more advanced spell casters. We’re also going to provide you with the top sites available to help you with your dilemma. Let’s begin our journey by educating ourselves on love magic and what a real love caster looks like. Love Magic and Love Casters Love magic made its first appearance back in Ancient Egypt and has been an active practice since. This type of magic is a branch of traditional magic and can be practiced in various ways. Typically the more common use of love magic is through the work of spells, but other methods look like Charms Rituals-LOVE Potions-Dolls and even Amulets If you are interested in becoming a love caster, be prepared for what’s to come. A genuine love caster knows that the art of love casting is no easy feat and shouldn’t be done casually. You should know that not only does it require you to be gifted spiritually, but you must be ready to serve others. Someone who is considered a real love caster has experience in all manner of spells, no matter the difficulty. Training yourself in attraction, commitment, and marriage spells is an excellent place to start. But this by no means will make you a professional. Practice your craft and expand your knowledge; understand that you will possess the ability to help others in time truly. Types of Love Spells What better way to start broadening your experiences with love spells than by learning more about them? These spells work like just about any other spell. Simply apply your intention, use a medium (sigils, mantras, candles, or charm bags), and top it off with establishing the belief that you will receive what you want. So what kind of spells are available and which ones suit your needs the best? Let’s take a look at the many options you have at your disposal. Attraction Spells

%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...

masabamasaba

%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein

masabamasaba

Kürzlich hochgeladen (20)

OpenChain - The Ramifications of ISO/IEC 5230 and ISO/IEC 18974 for Legal Pro...

%in Midrand+277-882-255-28 abortion pills for sale in midrand

%+27788225528 love spells in Toronto Psychic Readings, Attraction spells,Brin...

Direct Style Effect Systems -The Print[A] Example- A Comprehension Aid

%in kempton park+277-882-255-28 abortion pills for sale in kempton park

Artyushina_Guest lecture_YorkU CS May 2024.pptx

Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...

WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation

%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...

WSO2CON 2024 - WSO2's Digital Transformation Journey with Choreo: A Platforml...

%in tembisa+277-882-255-28 abortion pills for sale in tembisa

Microsoft AI Transformation Partner Playbook.pdf

VTU technical seminar 8Th Sem on Scikit-learn

Abortion Pills In Pretoria ](+27832195400*)[ 🏥 Women's Abortion Clinic In Pre...

%in Harare+277-882-255-28 abortion pills for sale in Harare

Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024

Abortion Pill Prices Tembisa [(+27832195400*)] 🏥 Women's Abortion Clinic in T...

tonesoftg

%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...

%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein

Apache Hudi: Community-Driven Development

1. Apache Hudi: Community-Driven Development Shiyan Xu Engineering Manager @ Onehouse, Apache Hudi PMC

2. Agenda ● Hudi: quick overview - 1 min ● Community-driven features - 5 min ● MetaServer & Table Management Service - 18 min ● Closing - 1 min

3. Hudi: Typical Use-Cases

4. 1800+ Slack Members 225+ Contributors 1000+ GH Engagers 20+ Committers Pre-installed on 5 cloud providers Diverse PMC/Committers 1M DLs/month (400% YoY) 800B+ Records/Day (from even just 1 customer!) Marquee OSS Users Community Stats

5. - Slack Community Notable newcomers in Q1 - 2022 include: Total Slack Members: Engagement Rate: 1.8k 12% 221 Active Members: 40 61 7.7% 8.5% 519 712 +157% YoY +3.5pts YoY +262% YoY

6. ● RFC-51: Support Change-Data-Capture (Alibaba) ○ Capture and produce all changes made to the table ○ Implement debezium-like semantics ● RFC-42: Consistent hashing index (Alibaba) ○ Overcome throttling issues for large tables ● RFC-8: Record-level indexing mechanisms (Uber) ○ Improve indexing by direct mapping record keys to files ● RFC-48: Reduce write amplification with Log-Compaction (Uber) ○ Balance write amp and I/O efficiency Community-Driven RFCs

7. Community-Driven RFCs (cont’d) ● RFC-61: Snapshot view management (Shopee) ● RFC-52: Secondary index to improve query performance (Alibaba) ● RFC-60: Optimized storage layout for cloud object stores (AWS) ● RFC-12: Eﬃcient migration of large parquet tables to Hudi (AWS/Onehouse)

8. MetaServer & Table Management Service MetaServer (RFC-36) ● Centralize the metadata management for tables ● Atomic sync: storage <-> metastore TMS (RFC-43) ● Centralize the orchestration for running table services ● Decouple compute from the writer cluster

9. MetaServer

10. MetaServer: Storage ● Table ○ Path, Schema, properties, format, table type, etc ● Partition ○ Path, parameters ● Timeline ○ Instant: timestamp, action type, state ○ Instant metadata: updated partitions, ﬁles, write statistics ● Snapshot ○ File name, size, creation timestamp

11. MetaServer: Write

12. MetaServer: Read

13. MetaServer: Event Bus

14. MetaServer: Other features ● Lock-provider ○ Storage supports Check-and-Set (CAS) ○ Supports multi-writer ● Schema evolution ○ Server stores schemas with versions and bind to commits ○ Detect conflicts during commit ● Concurrency management ○ Conflict detection strategies: timestamp, table, partition, filegroup, file

15. Table Management Service (TMS)

16. TMS: Plan Generator

17. TMS: Job Management

18. TMS: Stateless Design

19. MetaServer & Table Management Service Proposers: Xiaoyu Geng (ByteDance), Zhaojing Yu (ByteDance) Reviewers: Vinoth (Onehouse), Prasanna (OneHouse), Shiyan Xu (OneHouse) Community engagements ● Submit RFCs ● Weekly sync meetings ● RFC reviews ● Implementation discussions & planning

20. Engage With Our Community User Docs : https://hudi.apache.org Technical Wiki : https://cwiki.apache.org/conﬂuence/display/HUDI Github : https://github.com/apache/hudi/ Twitter : https://twitter.com/apachehudi Mailing list(s) : dev-subscribe@hudi.apache.org (send an empty email to subscribe) dev@hudi.apache.org (actual mailing list) Slack : https://join.slack.com/t/apache-hudi/signup

21. Who are using

22. Thanks! Questions?

Apache Hudi: Community-Driven Development

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Ähnlich wie Apache Hudi: Community-Driven Development

Ähnlich wie Apache Hudi: Community-Driven Development (20)

Mehr von Alluxio, Inc.

Mehr von Alluxio, Inc. (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Apache Hudi: Community-Driven Development