Java zone 2015 How to make life with kafka easier.

•

2 gefällt mir•1,127 views

You’ve just set up your Kafka cluster and now you are ready to process tens of thousands events per second. You decoupled your architecture and now all the communication goes via pubSub bus and you can focus only on providing business value. It would be great if that can be true. In real life you need to do a lot of tweaks to have your backbone ready to handle all the traffic you want.

Technologie

PubSub++
How to make your life with Kafka easier
Krzysztof Dębski
@DebskiChris
JavaZone 2015

Who am I
@DebskiChris
http://hermes.allegro.tech

Allegro Group
500+ people in IT
50+ independent teams
16 years on market
2 years after technical revolution

Partitioning
Round robin partitioning (default)
Key based partitioning

Rebalancing leaders
Broker 1
P1 P0
Broker 2
P2 P1
Broker 3
P0 P2
Topic: test Partition count: 3 Replication factor: 1
Configs: retention.ms=86400000
Topic: test Partition: 0 Leader: 3 Replicas: 3, 1 ISR: 3, 1
Topic: test Partition: 1 Leader: 1 Replicas: 1, 2 ISR: 1, 2
Topic: test Partition: 2 Leader: 2 Replicas: 2, 3 ISR: 2, 3

ACK levels
0 - don’t wait for response from the leader
1 - only the leader has to respond
-1 - all replicas must be in sync
Speed
Safety

Lost events
ERROR [Replica Manager on Broker 2]: Error when processing
fetch request for partition [test,1] offset 10000 from consumer
with correlation id 0. Possible cause:
Request for offset 10000 but we only have log segments in the
range 8000 to 9000. (kafka.server.ReplicaManager)

Lost events
Broker 1 Broker 2
Producer
ACK = 1
Replication factor = 1
replica.lag.max.messages = 2000
commited offset = 10000 commited offset = 9000
Zookeeper

Slow responses
75%
99%
99,9%
responsetime

Slow responses
messagesize
75%
99%
99,9%
Is response time correlated to message size?

Slow responses
responsetime
75%
99%
99,9%
Same distribution for fixed message size.

Slow responses
responsetime
75%
99%
99,9%
Hermes overhead is just about 1 ms.

Optimize message sizemessagesize
99,9%
all
topics
99,9%
biggest
topic

Optimize message size
JSON human readable
big memory and network footprint
poor support for Hadoop

Optimize message size
JSON
Snappy
ERROR Error when sending message to topic t3 with key: 4 bytes, value: 100
bytes with error: The server experienced an unexpected error when
processing the request (org.apache.kafka.clients.producer.internals.
ErrorLoggingCallback)
java: target/snappy-1.1.1/snappy.cc:423: char* snappy::internal::
CompressFragment(const char*, size_t, char*, snappy::uint16*, int): Assertion
`0 == memcmp(base, candidate, matched)' failed.
errors on publishing large amount of
messages

Optimize message size
JSON
Snappy
Lz4
failed on distributed data
compressionratio
single
topic
multiple
topics

Optimize message size
JSON
Snappy
Lz4
Avro
small network footprint
Hadoop friendly
easy schema verification

Consumer backoff
You can’t have exactly one delivery
http://bravenewgeek.com/you-cannot-have-exactly-once-delivery/

Kafka offsets
<=0.8.1 - Zookeeper
>=0.8.2 - Zookeeper or Kafka
>=0.9(?) - Kafka

Improved security
Authentication and authorization interfaces provided
By Default:
You can create any topic in your group
You can publish everywhere (in progress)
Group owner defines subscriptions

Turn back the time
PUT /groups/{group}/topics/{topic}/subscriptions/{subscription}/retransmission -8h

Blog: http://allegro.tech
Twitter: @allegrotechblog
Twitter: @debskichris

Weitere ähnliche Inhalte

Was ist angesagt?

Learn RabbitMQ with Python in 90mins

Larry Cai

Kube-proxy enables access to Kubernetes services (virtual IPs backed by pods) by configuring client-side load-balancing on nodes. The first implementation relied on a userspace proxy which was not very performant. The second implementation used iptables and is still the one used in most Kubernetes clusters. Recently, the community introduced an alternative based on IPVS. This talk will start with a description of the different modes and how they work. It will then focus on the IPVS implementation, the improvements it brings, the issues we encountered and how we fixed them as well as the remaining challenges and how they could be addressed. Finally, the talk will present alternative solutions based on eBPF such as Cilium.

Evolution of kube-proxy (Brussels, Fosdem 2020)

Laurent Bernaille

Anton Moldovan "Building an efficient replication system for thousands of ter...

Fwdays

During ElixirConf, I had spoken with several people about how we get our Elixir applications into production at Puppet Labs, so I decided to do a lightning talk. Awesome things: - Dependency-free OS packages \o/ - Same OS packages, but different configurations based on server, data center, etc from hiera - Operations folks don't need to know/care about internals of the application or its configuration format

ElixirConf Lightning Talk: Elixir |> Production

Jeff Weiss

DPNHTW

metaldart

Peter Zaitsev "18 ways to fix MySQL bottlenecks"

Fwdays

Percona XtraDB 集群文档

YUCHENG HU

Alternative Infrastucture

Marc Seeger

The Puppet Master on the JVM - PuppetConf 2014

Puppet

Dissecting the rabbit: RabbitMQ Internal Architecture

Alvaro Videla

Implement server push in flask framework

Chi-Chia Huang

Mail server configuration

chacheng oo

Streams are Awesome - (Node.js) TimesOpen Sep 2012

Tom Croucher

Scaling PostgreSQL with Skytools

Gavin Roy

Montreal On Rails 5 : Rails deployment using : Nginx, Mongrel, Mongrel_cluste...

addame

Nats meetup oct 2016 docker 112

Nirmal Mehta

The Docker network overlay driver relies on several technologies: network namespaces, VXLAN, Netlink and a distributed key-value store. This talk will present each of these mechanisms one by one along with their userland tools and show hands-on how they interact together when setting up an overlay to connect containers. The talk will continue with a demo showing how to build your own simple overlay using these technologies. Finally, it will show how we can dynamically distribute IP and MAC information to every hosts in the overlay using BGP EVPN

Deeper dive in Docker Overlay Networks

Laurent Bernaille

Build service with_docker_in_90mins

Larry Cai

Consul - service discovery and others

Walter Liu

Extra credit

metaldart

Was ist angesagt? (20)

Learn RabbitMQ with Python in 90mins

Evolution of kube-proxy (Brussels, Fosdem 2020)

Anton Moldovan "Building an efficient replication system for thousands of ter...

ElixirConf Lightning Talk: Elixir |> Production

DPNHTW

Peter Zaitsev "18 ways to fix MySQL bottlenecks"

Percona XtraDB 集群文档

Alternative Infrastucture

The Puppet Master on the JVM - PuppetConf 2014

Dissecting the rabbit: RabbitMQ Internal Architecture

Implement server push in flask framework

Mail server configuration

Streams are Awesome - (Node.js) TimesOpen Sep 2012

Scaling PostgreSQL with Skytools

Montreal On Rails 5 : Rails deployment using : Nginx, Mongrel, Mongrel_cluste...

Nats meetup oct 2016 docker 112

Deeper dive in Docker Overlay Networks

Build service with_docker_in_90mins

Consul - service discovery and others

Extra credit

Andere mochten auch

Messaging architectures in any environment, from local standalone deployments through to public clouds, must provide the highest reliability yet maximize their performance. This session gives you an insight into IBM MQ and how applications can be made to perform to their absolute best while maintaining the data integrity that IBM MQ is renowned for. We'll see how this can be achieved through a combination of good application design, system tuning and architectural patterns.

3450 - Writing and optimising applications for performance in a hybrid messag...

Timothy McCormick

Apache kafka- Onkar Kadam

Onkar Kadam

Publish/subscribe is the messaging model of choice for a wide range of messaging environments. Ranging from traditional MQ applications, applications running in frameworks like JEE and .NET or applications using the new MQ Light and open source AMQP capabilities, either on premise or in a Hybrid cloud. When MQ is at the heart of the solution all these environments can interoperate over a shared publish/subscribe infrastructure. This session will take you from the fundamentals of publish/subscribe through to how you can join these diverse systems together.

3425 - Using publish/subscribe to integrate applications

Timothy McCormick

Hermes - Greek God

dnelson7

Introducing IBM Message Hub: Cloud-scale messaging based on Apache Kafka

Andrew Schofield

The IBM MQ Light API makes it simple for developers to create responsive applications that are easy to scale without having to become messaging experts. Increasingly, development teams choose from a wide variety of languages, so the MQ Light API is available in a range of popular languages such as Ruby and Python, with the syntax tailored to fit naturally in each. The same API can be used with MQ Light installed on a laptop, with enterprise MQ queue managers, or in the cloud with the Message Hub service, so you can move seamlessly between these environments. Come and see how this API can make your developers more productive.

HHM-3540: The IBM MQ Light API: From Developer Laptop to Enterprise Data Cen...

Matt Leming

#1922 rest-push2 ap-im-v6

Jack Carnes

3429 How to transform your messaging environment to a secure messaging envi...

Robert Parker

Apache Kafka lies at the heart of the largest data pipelines, handling trillions of messages and petabytes of data every day. Learn the right approach for getting the most out of Kafka from the experts at LinkedIn and Confluent. Todd Palino and Gwen Shapira demonstrate how to monitor, optimize, and troubleshoot performance of your data pipelines—from producer to consumer, development to production—as they explore some of the common problems that Kafka developers and administrators encounter when they take Apache Kafka from a proof of concept to production usage. Too often, systems are overprovisioned and underutilized and still have trouble meeting reasonable performance agreements. Topics include: - What latencies and throughputs you should expect from Kafka - How to select hardware and size components - What you should be monitoring - Design patterns and antipatterns for client applications - How to go about diagnosing performance bottlenecks - Which configurations to examine and which ones to avoid

Putting Kafka Into Overdrive

Todd Palino

WhatsNewIBMIntegrationBus10FP4

bthomps1979

Tuning Kafka for Fun and Profit

Todd Palino

ConnectorsForIntegration

bthomps1979

High Performance Processing of Streaming Data

Geoffrey Fox

Presentation from IBM InterConnect 2016 . With growth in the number of business applications and exponential growth in connectivity between applications and systems, it is important to understand not just how to implement security, but why it is important to ensure all parts of the business can appreciate it and apply the right levels of security to their messaging system use. - jointly presented by Leif Davidsen and Rob Parker

IBM Messaging Security - Why securing your environment is important : IBM Int...

Leif Davidsen

Hia 1691-using iib-to_support_api_economy

Andrew Coleman

This is a talk given at ApacheCon 2015 If data is the lifeblood of high technology, Apache Kafka is the circulatory system in use at LinkedIn. It is used for moving every type of data around between systems, and it touches virtually every server, every day. This can only be accomplished with multiple Kafka clusters, installed at several sites, and they must all work together to assure no message loss, and almost no message duplication. In this presentation, we will discuss the architectural choices behind how the clusters are deployed, and the tools and processes that have been developed to manage them. Todd Palino will also discuss some of the challenges of running Kafka at this scale, and how they are being addressed both operationally and in the Kafka development community. Note - there are a significant amount of slide notes on each slide that goes into detail. Please make sure to check out the downloaded file to get the full content!

Kafka at Scale: Multi-Tier Architectures

Todd Palino

Hia 1689-techinical introduction-to_iib

Andrew Coleman

Holistic approach towards risk management

PureValueAdvisors

PT.Jeklindo Consulting adalah perusahaan yang bergerak di bidang jasa konsultan perijinan yang bergerak dibidang layanan pemberian jasa pengurusan perijinan usaha di Indonesia. Disini kami mengundang anda untuk mengetahui lebih banyak tentang kami serta jasa layanan kami. Dengan pengalaman kami dalam mengurus Perijinan usaha dan dokumen Perusahaan maupun Pribadi maka kami menjamin dapat mengurus perijinan dengan cepat dan harga yang terjangkau atas ijin usaha anda . Ijin Usaha ataupun Jasa-jasa Pengurusan yang dapat kami berikan antara lain : 1. URUS IZIN SURAT KETERANGAN PENCANTUMAN LABEL DALAM BAHASA INDONESIA ( SKPLBI) - (PERATURAN TERBARU DARI DIREKTORAT JENDERAL STANDARDISASI DAN PERLINDUNGAN KONSUMEN ) 2. SKT MIGAS ( SURAT KETERANGAN TERDAFTAR MINYAK DAN GAS BUMI) 3. PENGURUSAN PMA ( PENANAMAN MODAL ASING ) 4. PENDIRIAN PT ( PERSEROAN TERBATAS ) 5. JUAL PT 6. API ( ANGKA PENGENAL IMPORTIR ) 7. SURAT REGISTRASI PABEAN ( SRP ) / NOMER INDUK KEPABEANAN ( NIK ) 8. IMPORTIR TERDAFTAR PRODUCT TERTENTU 9. SERTIFIKAT TANAH 10. SURAT IJIN USAHA PERDAGANGAN ( SIUP ) 11. TANDA DAFTAR PERUSAHAAN ( TDP ) 12. SERTIFIKASI BADAN USAHA ( SBU ) 13. SERTIFIKASI 14. KEAGENAN 15. MEREK PATENT 16. NOMER PENGENAL IMPORTIR KHUSUS ( NPIK ) 17. DAN PERIJINAN LAINNYA. Dengan tenaga profesional yang berpengalaman menjadikan kami sebagai mitra bagi Pengusaha dan investor untuk berinvestasi di Indonesia.

PT.JEKLINDO PERSADA JASA PENGURUSAN IZIN DOKUMEN PERUSAHAAN ( 085262245981)

PT.Jeklindo Persada Consulting 085262245981

PT. JEKLINDO CONSULTING ( URUS DOKUMEN PERIJINAN PERUSAHAAN ) 085262245981

PT.Jeklindo Persada Consulting 085262245981

Andere mochten auch (20)

3450 - Writing and optimising applications for performance in a hybrid messag...

Apache kafka- Onkar Kadam

3425 - Using publish/subscribe to integrate applications

Hermes - Greek God

Introducing IBM Message Hub: Cloud-scale messaging based on Apache Kafka

HHM-3540: The IBM MQ Light API: From Developer Laptop to Enterprise Data Cen...

#1922 rest-push2 ap-im-v6

3429 How to transform your messaging environment to a secure messaging envi...

Putting Kafka Into Overdrive

WhatsNewIBMIntegrationBus10FP4

Tuning Kafka for Fun and Profit

ConnectorsForIntegration

High Performance Processing of Streaming Data

IBM Messaging Security - Why securing your environment is important : IBM Int...

Hia 1691-using iib-to_support_api_economy

Kafka at Scale: Multi-Tier Architectures

Hia 1689-techinical introduction-to_iib

Holistic approach towards risk management

PT.JEKLINDO PERSADA JASA PENGURUSAN IZIN DOKUMEN PERUSAHAAN ( 085262245981)

PT. JEKLINDO CONSULTING ( URUS DOKUMEN PERIJINAN PERUSAHAAN ) 085262245981

Ähnlich wie Java zone 2015 How to make life with kafka easier.

MAKE YOUR WORLD EVENT DRIVEN Just after you set up your first microservice you realize that the game has just started. You need to improve latency in your application and reduce unnecessary communication. To make your architecture fully decoupled you need to embrace asynchronous communication. Good way to achieve that is to switch to Event Driven Architecture. We will see how to use Kafka in your microservices. We will also cover some pitfalls you might face during using Kafka and how to deal with them. After the talk you will know the toolset that are need to improve your microservice ecosystem.

JDD2015: Make your world event driven - Krzysztof Dębski

PROIDEA

Apache Kafka is a key part of the Big Data infrastructure at Salesforce, enabling publish/subscribe and data transport in near real-time at enterprise scale handling trillions of messages per day. In this session, hear from the teams at Salesforce that manage Kafka as a service, running over a hundred clusters across on-premise and public cloud environments with over 99.9% availability. Hear about best practices and innovations, including: * How to manage multi-tenant clusters in a hybrid environment * High volume data pipelines with Mirus replicating data to Kafka and blob storage * Kafka Fault Injection Framework built on Trogdor and Kibosh * Automated recovery without data loss * Using Envoy as an SNI-routing Kafka gateway We hope the audience will have practical takeaways for building, deploying, operating, and managing Kafka at scale in the enterprise.

Tales from the four-comma club: Managing Kafka as a service at Salesforce | L...

HostedbyConfluent

Introduction to Apache Kafka

Shiao-An Yuan

Performance Optimization of Rails Applications

Serge Smetana

TDEA 2018 Kafka EOS (Exactly-once)

Erhwen Kuo

Geecon.cz 2015 debski krzysztof

Krzysztof Debski

Apache Kafka has shown that the log is a powerful abstraction for data-intensive applications. It can play a key role in managing data and distributing it across the enterprise efficiently. Vital to any data plane is not just performance, but availability and scalability. In this session, we examine what a distributed log is, how it works, and how it can achieve these goals. Specifically, we'll discuss lessons learned while building NATS Streaming, a reliable messaging layer built on NATS that provides similar semantics. We'll cover core components like leader election, data replication, log persistence, and message delivery. Come learn about distributed systems!

Building a Distributed Message Log from Scratch

Tyler Treat

Kafka is the bedrock of Wix’s distributed Mega Microservices system. Over the years we have learned a lot about how to successfully scale our event-driven architecture to roughly 1400 mostly Scala microservices. In this talk, you will learn about 10 key decisions and steps you can take in order to safely scale-up your Kafka-based system. These Include: * How to increase dev velocity of event-driven style code. * How to optimize working with Kafka in polyglot setting * How to migrate from request-reply to event-driven * How to tackle multiple DCs environment.

10 Lessons Learned from using Kafka in 1000 microservices - ScalaUA

Natan Silnitsky

Citi TechTalk Session 2: Kafka Deep Dive

confluent

In the financial industry, losing data is unacceptable. Financial firms are adopting Kafka for their critical applications. Kafka provides the low latency, high throughput, high availability, and scale that these applications require. But can it also provide complete reliability? As a system architect, when asked “Can you guarantee that we will always get every transaction,” you want to be able to say “Yes” with total confidence. In this session, we will go over everything that happens to a message – from producer to consumer, and pinpoint all the places where data can be lost – if you are not careful. You will learn how developers and operation teams can work together to build a bulletproof data pipeline with Kafka. And if you need proof that you built a reliable system – we’ll show you how you can build the system to prove this too.

When it Absolutely, Positively, Has to be There: Reliability Guarantees in Ka...

confluent

Golang Performance : microbenchmarks, profilers, and a war story

Aerospike

Replication MongoDB Days 2013

Randall Hunt

"It's important that even under load, Apache Kafka ensures user topics are fully replicated in synch. Replication is essential to endure resilience to data loss, so both users and operators care about it. If a topic partition falls out of the ISR (In-Synch-replicas) set, a user experiences unavailability (when producing with the default acknowledgment setting). Users may use non-default acks mode to work around it, but the effect on a Kafka cluster is to make the under-replication worse. Even simple Under replication with no Under Min Isr is to be avoided as a cluster update may cause the dreaded Under Min ISR. There are a number of settings that can be used, from quotas to number of replication threads to more low-level settings. This session wants to show how we successfully measured and evolved our Kafkas configuration, with the goal of giving the best possible user experience (and resilience to their data). Hofstadter's Law applied! ""It always takes longer than you expect, even when you take into account Hofstadter's Law."""

Seek and Destroy Kafka Under Replication

HostedbyConfluent

Apache Kafka Women Who Code Meetup

Snehal Nagmote

RedisConf17- Using Redis at scale @ Twitter

Redis Labs

Performance tweaks and tools for Linux (Joe Damato)

Ontico

We recently learned about “Fault Tree Analysis” and decided to apply the technique to bulletproof our Apache Kafka deployments. In this talk, learn about fault tree analysis and what you should focus on to make your Apache Kafka clusters resilient. This talk should provide a framework for answers the following common questions a Kafka operator or user might have: -What guarantees can I promise my users? -What should my replication factor? -What should the ISR setting be? -Should I use RAID or not? -Should I use external storage such as EBS or local disks?

Bulletproof Kafka with Fault Tree Analysis (Andrey Falko, Lyft) Kafka Summit ...

confluent

Learn the specifics of Amazon RDS for PostgreSQL’s capabilities and extensions that make it powerful. This session begins with a brief overview of the RDS PostgreSQL service, how it provides High Availability & Durability and will then deep dive into the new features that we have released since re:Invent 2014, including major version upgrade and newly added PostgreSQL extensions to RDS PostgreSQL. During the session, we will also discuss lessons learned running a large fleet of PostgreSQL instances, including specific recommendations. In addition we will present benchmarking results looking at differences between the 9.3, 9.4 and 9.5 releases.

(DAT402) Amazon RDS PostgreSQL:Lessons Learned & New Features

Amazon Web Services

Kafka zero to hero

Avi Levi

Apache Kafka - From zero to hero

Apache Kafka TLV

Ähnlich wie Java zone 2015 How to make life with kafka easier. (20)

JDD2015: Make your world event driven - Krzysztof Dębski

Tales from the four-comma club: Managing Kafka as a service at Salesforce | L...

Introduction to Apache Kafka

Performance Optimization of Rails Applications

TDEA 2018 Kafka EOS (Exactly-once)

Geecon.cz 2015 debski krzysztof

Building a Distributed Message Log from Scratch

10 Lessons Learned from using Kafka in 1000 microservices - ScalaUA

Citi TechTalk Session 2: Kafka Deep Dive

When it Absolutely, Positively, Has to be There: Reliability Guarantees in Ka...

Golang Performance : microbenchmarks, profilers, and a war story

Replication MongoDB Days 2013

Seek and Destroy Kafka Under Replication

Apache Kafka Women Who Code Meetup

RedisConf17- Using Redis at scale @ Twitter

Performance tweaks and tools for Linux (Joe Damato)

Bulletproof Kafka with Fault Tree Analysis (Andrey Falko, Lyft) Kafka Summit ...

(DAT402) Amazon RDS PostgreSQL:Lessons Learned & New Features

Kafka zero to hero

Apache Kafka - From zero to hero

Mehr von Krzysztof Debski

Architektura w szybko zmieniających się organizacjach

Krzysztof Debski

Devopsdays.pl 2015 krzysztof_debski (2)

Krzysztof Debski

Centralized or decentralized architecture?

Krzysztof Debski

Open your project

Krzysztof Debski

PubSub++ - Atmosphere 2015

Krzysztof Debski

Technical Product Owner or How to build technical backing for services

Krzysztof Debski

33degree Krzysztof Debski - Let's build a solid base for a scale

Krzysztof Debski

AtmosphereConf - Let's build a solid base for a scale

Krzysztof Debski

Mehr von Krzysztof Debski (8)

Architektura w szybko zmieniających się organizacjach

Devopsdays.pl 2015 krzysztof_debski (2)

Centralized or decentralized architecture?

Open your project

PubSub++ - Atmosphere 2015

Technical Product Owner or How to build technical backing for services

33degree Krzysztof Debski - Let's build a solid base for a scale

AtmosphereConf - Let's build a solid base for a scale

Kürzlich hochgeladen

Corporate and higher education. Two industries that, in the past, have had a clear divide with very little crossover. The difference in goals, learning styles and objectives paved the way for differing learning technologies platforms to evolve. Now, those stark lines are blurring as both sides are discovering they have content that’s relevant to the other. Join Tammy Rutherford as she walks through the pros and cons of corporate and higher ed collaborating. And the challenges of these different technology platforms working together for a brighter future.

Corporate and higher education May webinar.pptx

Rustici Software

ICT role in 21st century education and its challenges

rafiqahmad00786416

Whatsapp Number Escorts Call girls 8617370543 Available 24x7 Navi Mumbai Call Girls Service Offer Genuine VIP Model Escorts Call Girls in Your Budget. Navi Mumbai Call Girls Service Provide Real Call Girls Number. Make Your Sexual Pleasure Memorable with Our Navi Mumbai Call Girls at Affordable Price. Top VIP Escorts Call Girls, High Profile Independent Escorts Call Girls, Housewife Women Escorts Call Girl, College Girls Escorts Call Girls, Russian Escorts Call girls Service in Your Budget.

Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model

Deepika Singh

Building Digital Trust in a Digital Economy Veronica Tan, Director - Cyber Security Agency of Singapore Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

apidays

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

Scalable LLM APIs for AI and Generative AI Application Development Ettikan Karuppiah, Director/Technologist - NVIDIA Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...

apidays

Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...

Zilliz

Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows. We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases. This video focuses on the deployment of external web forms using Jotform for Bonterra Impact Management. This solution can be customized to your organization’s needs and deployed to support the common use cases below: - Intake and consent - Assessments - Surveys - Applications - Program registration Interested in deploying web form automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

Jeffrey Haguewood

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Juan lago vázquez

MINDCTI Revenue Release Quarter One 2024

MIND CTI

Real Time Object Detection Using Open CV

Khem

Abhishek Deb(1), Mr Abdul Kalam(2) M. Des (UX) , School of Design, DIT University , Dehradun. This paper explores the future potential of AI-enabled smartphone processors, aiming to investigate the advancements, capabilities, and implications of integrating artificial intelligence (AI) into smartphone technology. The research study goals consist of evaluating the development of AI in mobile phone processors, analyzing the existing state as well as abilities of AI-enabled cpus determining future patterns as well as chances together with reviewing obstacles as well as factors to consider for more growth.

Exploring the Future Potential of AI-Enabled Smartphone Processors

debabhi2

Automating Google Workspace (GWS) & more with Apps Script

wesley chun

presentation ICT roal in 21st century education

jfdjdjcjdnsjd

Modernizing Securities Finance: The cloud-native prime brokerage platform transforming capital markets. Madhu Subbu, Managing Director, Head of Securities Finance Engineering Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays Singapore 2024 - Modernizing Securities Finance by Madhu Subbu

apidays

MS Copilot expands with MS Graph connectors

Nanddeep Nachan

Join our latest Connector Corner webinar to discover how UiPath Integration Service revolutionizes API-centric automation in a 'Quote to Cash' process—and how that automation empowers businesses to accelerate revenue generation. A comprehensive demo will explore connecting systems, GenAI, and people, through powerful pre-built connectors designed to speed process cycle times. Speakers: James Dickson, Senior Software Engineer Charlie Greenberg, Host, Product Marketing Manager

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

DianaGray10

Architecting Cloud Native Applications

WSO2

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

Zilliz

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

sammart93

Kürzlich hochgeladen (20)

Corporate and higher education May webinar.pptx

ICT role in 21st century education and its challenges

Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

Strategies for Landing an Oracle DBA Job as a Fresher

Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...

Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

MINDCTI Revenue Release Quarter One 2024

Real Time Object Detection Using Open CV

Exploring the Future Potential of AI-Enabled Smartphone Processors

Automating Google Workspace (GWS) & more with Apps Script

presentation ICT roal in 21st century education

Apidays Singapore 2024 - Modernizing Securities Finance by Madhu Subbu

MS Copilot expands with MS Graph connectors

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Architecting Cloud Native Applications

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Java zone 2015 How to make life with kafka easier.

1. PubSub++ How to make your life with Kafka easier Krzysztof Dębski @DebskiChris JavaZone 2015

2. Who am I @DebskiChris http://hermes.allegro.tech

3. Allegro Group 500+ people in IT 50+ independent teams 16 years on market 2 years after technical revolution

4. Kafka as a backbone

5. Kafka

6. Hermes

7. Kafka data

8. Partitioning Round robin partitioning (default) Key based partitioning

9. Performance issues

10. Rebalancing leaders Broker 1 P1 P0 Broker 2 P2 P1 Broker 3 P0 P2 Topic: test Partition count: 3 Replication factor: 1 Configs: retention.ms=86400000 Topic: test Partition: 0 Leader: 3 Replicas: 3, 1 ISR: 3, 1 Topic: test Partition: 1 Leader: 1 Replicas: 1, 2 ISR: 1, 2 Topic: test Partition: 2 Leader: 2 Replicas: 2, 3 ISR: 2, 3

11. Rebalancing leaders Broker 1 P1 P0 Broker 2 P2 P1 Broker 3 P0 P2 Topic: test Partition count: 3 Replication factor: 1 Configs: retention.ms=86400000 Topic: test Partition: 0 Leader: 3 Replicas: 3, 1 ISR: 3, 1 Topic: test Partition: 1 Leader: 1 Replicas: 1, 2 ISR: 1, 2 Topic: test Partition: 2 Leader: 2 Replicas: 2, 3 ISR: 2, 3 Brokers that should have partition copies

12. Rebalancing leaders Broker 1 P1 P0 Broker 2 P2 P1 Broker 3 P0 P2 Topic: test Partition count: 3 Replication factor: 1 Configs: retention.ms=86400000 Topic: test Partition: 0 Leader: 3 Replicas: 3, 1 ISR: 3, 1 Topic: test Partition: 1 Leader: 1 Replicas: 1, 2 ISR: 1, 2 Topic: test Partition: 2 Leader: 2 Replicas: 2, 3 ISR: 2, 3 In Sync Replicas

13. Rebalancing leaders Broker 1 P1 P0 Broker 2 P2 P1 Broker 3 P0 P2 Topic: test Partition count: 3 Replication factor: 1 Configs: retention.ms=86400000 Topic: test Partition: 0 Leader: 3 Replicas: 3, 1 ISR: 3, 1 Topic: test Partition: 1 Leader: 1 Replicas: 1, 2 ISR: 1, 2 Topic: test Partition: 2 Leader: 2 Replicas: 2, 3 ISR: 2, 3 Leader broker ID

14. Rebalancing leaders Broker 1 P1 P0 Broker 2 P2 P1 Broker 3 P0 P2 Topic: test Partition count: 3 Replication factor: 1 Configs: retention.ms=86400000 Topic: test Partition: 0 Leader: 3 Replicas: 3, 1 ISR: 3, 1 Topic: test Partition: 1 Leader: 1 Replicas: 1, 2 ISR: 1, 2 Topic: test Partition: 2 Leader: 2 Replicas: 2, 3 ISR: 2, 3

15. Rebalancing leaders Broker 1 P1 P0 Broker 2 P2 P1 Broker 3 P0 P2 Topic: test Partition count: 3 Replication factor: 1 Configs: retention.ms=86400000 Topic: test Partition: 0 Leader: 1 Replicas: 3, 1 ISR: 1 Topic: test Partition: 1 Leader: 1 Replicas: 1, 2 ISR: 1, 2 Topic: test Partition: 2 Leader: 2 Replicas: 2, 3 ISR: 2

16. Rebalancing leaders Broker 1 P1 P0 Broker 2 P2 P1 Broker 3 P0 P2 Topic: test Partition count: 3 Replication factor: 1 Configs: retention.ms=86400000 Topic: test Partition: 0 Leader: 1 Replicas: 3, 1 ISR: 1, 3 Topic: test Partition: 1 Leader: 1 Replicas: 1, 2 ISR: 1, 2 Topic: test Partition: 2 Leader: 2 Replicas: 2, 3 ISR: 2, 3

17. Lost events

18. ACK levels 0 - don’t wait for response from the leader 1 - only the leader has to respond -1 - all replicas must be in sync Speed Safety

19. Event identification

20. Lost events

21. Lost events ERROR [Replica Manager on Broker 2]: Error when processing fetch request for partition [test,1] offset 10000 from consumer with correlation id 0. Possible cause: Request for offset 10000 but we only have log segments in the range 8000 to 9000. (kafka.server.ReplicaManager)

22. Lost events Broker 1 Broker 2 Producer ACK = 1 Replication factor = 1 replica.lag.max.messages = 2000 commited offset = 10000 commited offset = 9000 Zookeeper

23. Lost events Broker 1 Broker 2 Producer ACK = 1 Replication factor = 1 replica.lag.max.messages = 2000 commited offset = 10000 commited offset = 9000 Zookeeper

24. Lost events Broker 1 Broker 2 Producer ACK = 1 Replication factor = 1 replica.lag.max.messages = 2000 commited offset = 10000 commited offset = 9000 Zookeeper commited offset = 9000

25. Slow responses

26. Slow responses 75% 99% 99,9% responsetime

27. Slow responses messagesize 75% 99% 99,9% Is response time correlated to message size?

28. Slow responses responsetime 75% 99% 99,9% Same distribution for fixed message size.

29. Slow responses responsetime 75% 99% 99,9% Hermes overhead is just about 1 ms.

30. Kafka kernel 3.2.x

31. Kafka kernel 3.2.x

32. Kafka kernel 3.2.x kernel >= 3.8.x

33. Normal operation

34. Slow responses

35. Message size

36. Optimize message sizemessagesize 99,9% all topics 99,9% biggest topic

37. Optimize message size JSON human readable big memory and network footprint poor support for Hadoop

38. Optimize message size JSON Snappy ERROR Error when sending message to topic t3 with key: 4 bytes, value: 100 bytes with error: The server experienced an unexpected error when processing the request (org.apache.kafka.clients.producer.internals. ErrorLoggingCallback) java: target/snappy-1.1.1/snappy.cc:423: char* snappy::internal:: CompressFragment(const char*, size_t, char*, snappy::uint16*, int): Assertion `0 == memcmp(base, candidate, matched)' failed. errors on publishing large amount of messages

39. Optimize message size JSON Snappy Lz4 failed on distributed data compressionratio single topic multiple topics

40. Optimize message size JSON Snappy Lz4 Avro small network footprint Hadoop friendly easy schema verification

41. Improvements

42. Multi data center

43. Consumer backoff You can’t have exactly one delivery http://bravenewgeek.com/you-cannot-have-exactly-once-delivery/

44. Kafka offsets <=0.8.1 - Zookeeper >=0.8.2 - Zookeeper or Kafka >=0.9(?) - Kafka

45. Kafka Offset Monitor

46. Manage your topics

47. Improved security Authentication and authorization interfaces provided By Default: You can create any topic in your group You can publish everywhere (in progress) Group owner defines subscriptions

48. Improved offset management

49. Improved offset management

50. Improved offset management

51. Improved offset management

52. Turn back the time PUT /groups/{group}/topics/{topic}/subscriptions/{subscription}/retransmission -8h

53. Blog: http://allegro.tech Twitter: @allegrotechblog Twitter: @debskichris