SolrCloud Cluster management via APIs

•

1 gefällt mir•313 views

Anshum Gupta

My talk at Apache Big Data Europe 2016 in Seville, Spain

Software

SolrCloud cluster management APIs
Anshum Gupta
Apache Lucene/Solr PMC member & committer
IBM Watson

About me
• Anshum Gupta, Apache Lucene/Solr committer
and PMC member, IBM Watson Search team.
• Interested in search and related stuff.
• Apache Lucene since 2006 and Solr since 2010.

Apache Solr is the most widely-used search
solution on the planet.
Solr has tens of thousands of
applications in production.
You use everyday.
8,000,000+
Total downloads
Solr is both established
and growing.
250,000+
Monthly downloads
2,500+
Open Solr jobs and the largest
community of developers.

• SolrCloud - Introduced in 4.0
• No APIs to manage distributed clusters
• We’ve come a long way since then
A little bit of history

01
SolrCloud - Physical Architecture
ZooKeeper
Node 1 Node 2
Load
Balancer
Client
Client
Client
Client
Client
Client
Client
Client
Client
Lots
Of
Interaction
Coins by Creative Stall from the Noun Project

• Solr APIs
• Creating a collection
• Managing conﬁguration
Starting up

• Easy!
• Replica placement strategy
• Machine, rack, DC aware placement
• Add more complicated rules
• Use the rules during all replica linked API calls
Scaling Up

• LIST
• OVERSEERSTATUS
• CLUSTERSTATUS
• CoreAdmin STATUS
Get STATUS

• ADD/REMOVE ROLE
• Only Overseer
• ADD/DELETE REPLICAPROP
• Only PreferredLeader
• BALANCESHARDUNIQUE
• Evenly distribute an arbitrary property among nodes for a collection
• REBALANCELEADERS
• FORCELEADER
Election Time!

• Monitor disk-size
• CLUSTERSTATUS & Core STATUS
• SPLITSHARD
• Make sure there’s enough spare disk space
• Add one more replica
• Force leader election
• Delete old INACTIVE shard
Shard Splitting

• Auto-add replica
• Monitor cluster status - CLUSTERSTATUS API
• ADD Replica using replica placement strategy
• Remove replicas when there are too many
High availability

• CLUSTERSTATUS
• Overseer, Leaders, etc.
• Move the Overseer ﬁrst
• Rebalance leaders
• REPLACENODE - Ver 6.2+ only
• Force leader elections if required
Migrating Cluster Infrastructure

• Reinventing the wheel - why?
• SOLR-9735 : Umbrella JIRA for cluster management
• Basic APIs
• Commission a new node
• Decommission an existing node
• Collection and aggregation of metrics at collection/node/shard/
replica level
• Use the metrics and trigger recipe(s)
• Contribution in any form would be great!
Imagine… rule based automation!

Connect @
http://www.twitter.com/anshumgupta
http://www.linkedin.com/in/anshumgupta/
anshum@apache.org

Weitere ähnliche Inhalte

Was ist angesagt?

Ease of use in Apache Solr

Anshum Gupta

Best practices for highly available and large scale SolrCloud

Anshum Gupta

Inside Solr 5 - Bangalore Solr/Lucene Meetup

Shalin Shekhar Mangar

The traditional and typical search use case is the one large search collection distributed among many nodes and shared by all users. However, there is a class of applications which need a large number of small or medium collections which can be used, managed and scaled separately. This talk will cover our effort in helping a client set up a large scale SolrCloud setup with thousands of collections running on hundreds of nodes. I will describe the bottlenecks that we found in SolrCloud when running a large number of collections. I will also take you through the multiple features and optimizations that we contributed to Apache Solr to reduce or remove the choke points in the system. Finally, I will talk about the benchmarking process and the lessons learned from supporting such an installation in production.

Scaling SolrCloud to a Large Number of Collections - Fifth Elephant 2014

Shalin Shekhar Mangar

First oslo solr community meetup lightning talk janhoy

Cominvent AS

Intro to Apache Solr

Shalin Shekhar Mangar

Scaling SolrCloud to a Large Number of Collections: Presented by Shalin Shekh...

Lucidworks

SolrCloud Failover and Testing

Mark Miller

Autoscaling Solr - Shalin Shekhar Mangar, Lucidworks

Lucidworks

SolrCloud is a set of features in Apache Solr that enable elastic scaling of search indexes using sharding and replication. In this presentation, Tim Potter will provide an architectural overview of SolrCloud and highlight its most important features. Specifically, Tim covers topics such as: sharding, replication, ZooKeeper fundamentals, leaders/replicas, and failure/recovery scenarios. Any discussion of a complex distributed system would not be complete without a discussion of the CAP theorem. Mr. Potter will describe why Solr is considered a CP system and how that impacts the design of a search application.

Solr Exchange: Introduction to SolrCloud

thelabdude

Autoscaling Suggestions: Simplifying Operations - Varun Thacker, Lucidworks

Lucidworks

Solrcloud Leader Election

ravikgiitk

Enterprise search can grow big, really big! And growing. Tens, yes hundreds of servers may be involved, locally or in the cloud. Managing this has been complex and time consuming - until now :) SolrCloud to the rescue Using the world's most popular Open Source search engine, Apache Solr™, we will show you how the new upcoming version 4.0 makes scaling search in the cloud really simple and robust. A new feature called SolrCloud adds centralized configuration, distributed indexing & searching, automatic failover, recovery and leader election. Scaling is now as simple as adding a new server to your cluster and it will find its role where it is most needed and start serving searches.

Scaling search with Solr Cloud

Cominvent AS

Solr consistency and recovery internals

Cloudera, Inc.

Долгое время разработка производительных, масштабируемых, надежных и экономически эффективных распределенных систем, была прерогативой узкого круга специалистов. Переезд в «облако», сам по себе, проблему не решил. Обещанная провайдерами дешевая линейная масштабируемость, по-прежнему, недостижимая мечта для всех, сидящих «на игле» реляционных баз данных и монолитных архитектур. С выходом Microsoft Orleans, разработчики, наконец-то, получили максимально простую и удобную платформу для создания масштабируемых и отказоустойчивых распределенных систем, предназначенных для запуска в «облаке» или в приватном дата-центре. В докладе будут рассмотрены основные концепции и прецеденты использования платформы, такие как: Internet Of Things (IoT), распределенная обработка потоков данных, масштабирование РСУБД и любых других ограниченных ресурсов, отказоустойчивая координация длительно выполняющихся бизнес-процессов.

"Walk in a distributed systems park with Orleans" Евгений Бобров

Fwdays

Project Orleans - Actor Model framework

Neil Mackenzie

KEYNOTE: Solr- Past, Present & Future

lucenerevolution

Storm worker redesign

Roshan Naik

Ansible for large scale deployment

Karthik .P.R

Scaling MySQL using Fabric

Karthik .P.R

Was ist angesagt? (20)

Ease of use in Apache Solr

Best practices for highly available and large scale SolrCloud

Inside Solr 5 - Bangalore Solr/Lucene Meetup

Scaling SolrCloud to a Large Number of Collections - Fifth Elephant 2014

First oslo solr community meetup lightning talk janhoy

Intro to Apache Solr

Scaling SolrCloud to a Large Number of Collections: Presented by Shalin Shekh...

SolrCloud Failover and Testing

Autoscaling Solr - Shalin Shekhar Mangar, Lucidworks

Solr Exchange: Introduction to SolrCloud

Autoscaling Suggestions: Simplifying Operations - Varun Thacker, Lucidworks

Solrcloud Leader Election

Scaling search with Solr Cloud

Solr consistency and recovery internals

"Walk in a distributed systems park with Orleans" Евгений Бобров

Project Orleans - Actor Model framework

KEYNOTE: Solr- Past, Present & Future

Storm worker redesign

Ansible for large scale deployment

Scaling MySQL using Fabric

Ähnlich wie SolrCloud Cluster management via APIs

Akshay Sura, a leader in the Sitecore community and Sameer Maggon, a Solr guru, will take the audience through what it takes to design, and build Solr environments tuned for and worthy of a great Sitecore implementation. They will also share Devops considerations and best practices that are critical after Sitecore goes live. In addition to their experience-based comments, they will illustrate a number of these best practices with a live demo of SearchStax, a service that delivers Solr in PaaS and that Sitecore itself uses for its Managed Cloud environment.

SolrCloud-Best Practices for Sitecore. Design, build, and devops considerations

Sameer Maggon

Spinnaker Chadev

Ethan Rogers

Cross Datacenter Replication in Apache Solr 6

Shalin Shekhar Mangar

Meet Solr For The Tirst Again

Varun Thacker

YARN

Alex Moundalexis

Fusion on Kubernetes - Alan Eugenio & Joe Streeky, Lucidworks

Lucidworks

Apache Solr - Enterprise search platform

Tommaso Teofili

Introduction to SolrCloud

Varun Thacker

Organizations continue to adopt Solr because of its ability to scale to meet even the most demanding workflows. Recently, LucidWorks has been leading the effort to identify, measure, and expand the limits of Solr. As part of this effort, we've learned a few things along the way that should prove useful for any organization wanting to scale Solr. Attendees will come away with a better understanding of how sharding and replication impact performance. Also, no benchmark is useful without being repeatable; Tim will also cover how to perform similar tests using the Solr-Scale-Toolkit in Amazon EC2.

Benchmarking Solr Performance at Scale

thelabdude

How do Solr and Azure Search compare?

SearchStax

Apache Solr is a powerful search and analytics engine with features such as full-text search, faceting, joins, sorting and capable of handling large amounts of data across a large number of servers. However, with all that power and scalability comes complexity. Solr 6 supports a Parallel SQL feature which provides a simplified, well-known interface to your data in Solr, performs key operations such as sorts and shuffling inside Solr for massive speedups, provides best-practices based query optimization and by leveraging the scalability of SolrCloud and a clever implementation, allows you to throw massive amounts of computation power behind analytical queries. In this talk, we will explore the why, what and how of Parallel SQL and its building block Streaming Expressions in Solr 6 with a hint of the exciting new developments around this feature.

Parallel SQL and Streaming Expressions in Apache Solr 6

Shalin Shekhar Mangar

Wrapper formulas saltconf 2016

Charles Rodetsky

Lessons Learned Running The Largest OpenStack Clouds

Kenneth Hui

Meetup on Apache Zookeeper

Anshul Patel

This 52min training session summarises what you need to know with regards to Tungsten Clustering architecture and topologies; and then dives into its installation process. TOPICS COVERED - Re-cap of the Architecture And Topology - Plan The Installation - Documentation - Software - Environmental Decisions - Review Pre-Requisites - Review Installation Types And Methods - Walk Through: Simple 3 Node Cluster Installation

Training Slides: 103 - Basics - Simple Tungsten Clustering Installation

Continuent

Introduction to MySQL

Ted Wennmark

MySQL Fabric

Mark Swarbrick

Kubernetes Introduction & Whats new in Kubernetes 1.6

Opcito Technologies

Oracle Fusion Middleware on Exalogic Best Practises

Michel Schildmeijer

High availability of data across geographic regions for search and analytical applications is a challenging task. Mission critical applications need effective failover strategies across data centers. Apache Solr offers Cross Data Center Replication (CDCR) as a feature from 6.0 and has added more features in subsequent releases. The first part of session will center on an active-passive design model with one data-center as the primary and other data-centers as secondary clusters. The second design model centers on designing an active-active bidirectional setup such that both querying and indexing traffic can gracefully be redirected to the failover cluster. The third part of session will center on an actual use case: An analytics application with high availability. We will discuss the improvements observed in terms of maintenance, performance, and throughput. The session concludes with challenges and/or limitations in the current design and what improvements are forthcoming for Cross Data Center Replication in Apache Solr.

Cdcr apachecon-talk

Amrit Sarkar

Ähnlich wie SolrCloud Cluster management via APIs (20)

SolrCloud-Best Practices for Sitecore. Design, build, and devops considerations

Spinnaker Chadev

Cross Datacenter Replication in Apache Solr 6

Meet Solr For The Tirst Again

YARN

Fusion on Kubernetes - Alan Eugenio & Joe Streeky, Lucidworks

Apache Solr - Enterprise search platform

Introduction to SolrCloud

Benchmarking Solr Performance at Scale

How do Solr and Azure Search compare?

Parallel SQL and Streaming Expressions in Apache Solr 6

Wrapper formulas saltconf 2016

Lessons Learned Running The Largest OpenStack Clouds

Meetup on Apache Zookeeper

Training Slides: 103 - Basics - Simple Tungsten Clustering Installation

Introduction to MySQL

MySQL Fabric

Kubernetes Introduction & Whats new in Kubernetes 1.6

Oracle Fusion Middleware on Exalogic Best Practises

Cdcr apachecon-talk

Kürzlich hochgeladen

ADR, or Architecture Decision Record, is a valuable tool in software development for several reasons. It provides a centralized location for documenting and tracking architectural decisions, aiding both current and future team members. ADRs enhance communication among team members by documenting the rationale behind architectural decisions, especially beneficial during onboarding of new team members or when revisiting decisions. They serve as a knowledge base, enabling teams to learn from past decisions and refine their decision-making process. Additionally, ADRs contribute to transparency by helping stakeholders understand the reasons behind specific architectural choices. As with any other tool or process, introducing them into an organization can face several obstacles, and overcoming these challenges is crucial for successful implementation. In this talk I go through some common problems and our way of solving them.

Architecture decision records - How not to get lost in the past

Papp Krisztián

%in Rustenburg+277-882-255-28 abortion pills for sale in Rustenburg

masabamasaba

VTU technical seminar 8Th Sem on Scikit-learn

AmarnathKambale

%in Hazyview+277-882-255-28 abortion pills for sale in Hazyview

masabamasaba

tonesoftg

lanshi9

WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...

WSO2

WSO2CON 2024 - WSO2's Digital Transformation Journey with Choreo: A Platforml...

WSO2

Artyushina_Guest lecture_YorkU CS May 2024.pptx

AnnaArtyushina1

OpenChain - The Ramifications of ISO/IEC 5230 and ISO/IEC 18974 for Legal Pro...

Shane Coughlan

%in tembisa+277-882-255-28 abortion pills for sale in tembisa

masabamasaba

WSO2Con204 - Hard Rock Presentation - Keynote

WSO2

MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...

Jittipong Loespradit

WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...

WSO2

We specialize in Psychic Readings, Psychic Love Spells, Binding Love Spells, Obsession Spells, Voodoo Spells, Lottery Spells, Marriage Spells, Black Magic Spells, Palm Readings & much more. Are you depressed? We perform this come-to-me love spell that works instantly with the aim of Winnipeg back the victim to the person performing the magic. Have you lost your lover? We perform this come-to-me love spell that works instantly with the aim of bringing back the victim to the person performing the magic. Have you lost your lover? Do u need to solve any relationship problem? Contact the powerful spells caster chief kule with love spells that work overnight and love spells that really work. Have you found yourself infatuated with a special someone you think could be the one? Are you looking for a spell to provide them with a nudge in the right direction? Or maybe the spell you cast didn’t achieve the results you were hoping for? Whether you’re new or versed in the ways of spell casting, we’re here to help. Today we’re going to provide you with a detailed guide on the types of love spells to cast. Not only that but there’s something for those who wish to find outside advice from more advanced spell casters. We’re also going to provide you with the top sites available to help you with your dilemma. Let’s begin our journey by educating ourselves on love magic and what a real love caster looks like. Love Magic and Love Casters Love magic made its first appearance back in Ancient Egypt and has been an active practice since. This type of magic is a branch of traditional magic and can be practiced in various ways. Typically the more common use of love magic is through the work of spells, but other methods look like Charms Rituals-LOVE Potions-Dolls and even Amulets If you are interested in becoming a love caster, be prepared for what’s to come. A genuine love caster knows that the art of love casting is no easy feat and shouldn’t be done casually. You should know that not only does it require you to be gifted spiritually, but you must be ready to serve others. Someone who is considered a real love caster has experience in all manner of spells, no matter the difficulty. Training yourself in attraction, commitment, and marriage spells is an excellent place to start. But this by no means will make you a professional. Practice your craft and expand your knowledge; understand that you will possess the ability to help others in time truly. Types of Love Spells What better way to start broadening your experiences with love spells than by learning more about them? These spells work like just about any other spell. Simply apply your intention, use a medium (sigils, mantras, candles, or charm bags), and top it off with establishing the belief that you will receive what you want. So what kind of spells are available and which ones suit your needs the best? Let’s take a look at the many options you have at your disposal. Attraction Spells

%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...

masabamasaba

%in Midrand+277-882-255-28 abortion pills for sale in midrand

masabamasaba

%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...

masabamasaba

We specialize in Psychic Readings, Psychic Love Spells, Binding Love Spells, Obsession Spells, Voodoo Spells, Lottery Spells, Marriage Spells, Black Magic Spells, Palm Readings & much more. Are you depressed? We perform this come-to-me love spell that works instantly with the aim of bringing back the victim to the person performing the magic. Have you lost your lover? We perform this come-to-me love spell that works instantly with the aim of bringing back the victim to the person performing the magic. Have you lost your lover? Do u need to solve any relationship problem? Contact the powerful spells caster chief kule with love spells that work overnight and love spells that really work. Have you found yourself infatuated with a special someone you think could be the one? Are you looking for a spell to provide them with a nudge in the right direction? Or maybe the spell you cast didn’t achieve the results you were hoping for? Whether you’re new or versed in the ways of spell casting, we’re here to help. Today we’re going to provide you with a detailed guide on the types of love spells to cast. Not only that but there’s something for those who wish to find outside advice from more advanced spell casters. We’re also going to provide you with the top sites available to help you with your dilemma. Let’s begin our journey by educating ourselves on love magic and what a real love caster looks like. Love Magic and Love Casters Love magic made its first appearance back in Ancient Egypt and has been an active practice since. This type of magic is a branch of traditional magic and can be practiced in various ways. Typically the more common use of love magic is through the work of spells, but other methods look like Charms Rituals-LOVE Potions-Dolls and even Amulets If you are interested in becoming a love caster, be prepared for what’s to come. A genuine love caster knows that the art of love casting is no easy feat and shouldn’t be done casually. You should know that not only does it require you to be gifted spiritually, but you must be ready to serve others. Someone who is considered a real love caster has experience in all manner of spells, no matter the difficulty. Training yourself in attraction, commitment, and marriage spells is an excellent place to start. But this by no means will make you a professional. Practice your craft and expand your knowledge; understand that you will possess the ability to help others in time truly. Types of Love Spells What better way to start broadening your experiences with love spells than by learning more about them? These spells work like just about any other spell. Simply apply your intention, use a medium (sigils, mantras, candles, or charm bags), and top it off with establishing the belief that you will receive what you want. So what kind of spells are available and which ones suit your needs the best? Let’s take a look at the many options you have at your disposal.

%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...

masabamasaba

Abortion Pill Prices Tembisa [(+27832195400*)] 🏥 Women's Abortion Clinic in Tembisa ● Abortion Pills For Sale in Tembisa ● Tembisa 🏥🚑!! Abortion Clinic Near Me Cost, Price, Women's Clinic Near Me, Abortion Clinic Near, Abortion Doctors Near me, Abortion Services Near Me, Abortion Pills Over The Counter, Abortion Pill Doctors' Offices, Abortion Clinics, Abortion Places Near Me, Cheap Abortion Places Near Me, Medical Abortion & Surgical Abortion, approved cyctotec pills and womb cleaning pills too plus all the instructions needed This Discrete women’s Termination Clinic offers same day services that are safe and pain free, we use approved pills and we clean the womb so that no side effects are present. Our main goal is that of preventing unintended pregnancies and unwanted births every day to enable more women to have children by choice, not chance. We offer Terminations by Pill and The Morning After Pill.” Our Private VIP Abortion Service offers the ultimate in privacy, efficiency and discretion. we do safe and same day termination and we do also womb cleaning as well its done from 1 week up to 28 weeks. We do delivery of our services world wide SAFE ABORTION CLINICS/PILLS ON SALE WE DO DELIVERY OF PILLS ALSO Abortion clinic at very low costs, 100% Guaranteed and it’s safe, pain free and a same day service. It Is A 45 Minutes Procedure, we use tested abortion pills and we do womb cleaning as well. Alternatively the medical abortion pill and womb cleansing !!!

Abortion Pill Prices Tembisa [(+27832195400*)] 🏥 Women's Abortion Clinic in T...

Medical / Health Care (+971588192166) Mifepristone and Misoprostol tablets 200mg

We specialize in Psychic Readings, Psychic Love Spells, Binding Love Spells, Obsession Spells, Voodoo Spells, Lottery Spells, Marriage Spells, Black Magic Spells, Palm Readings & much more. Are you depressed? We perform this come-to-me love spell that works instantly with the aim of bringing back the victim to the person performing the magic. Have you lost your lover? We perform this come-to-me love spell that works instantly with the aim of bringing back the victim to the person performing the magic. Have you lost your lover? Do u need to solve any relationship problem? Contact the powerful spells caster chief kule with love spells that work overnight and love spells that really work. Have you found yourself infatuated with a special someone you think could be the one? Are you looking for a spell to provide them with a nudge in the right direction? Or maybe the spell you cast didn’t achieve the results you were hoping for? Whether you’re new or versed in the ways of spell casting, we’re here to help. Today we’re going to provide you with a detailed guide on the types of love spells to cast. Not only that but there’s something for those who wish to find outside advice from more advanced spell casters. We’re also going to provide you with the top sites available to help you with your dilemma. Let’s begin our journey by educating ourselves on love magic and what a real love caster looks like. Love Magic and Love Casters Love magic made its first appearance back in Ancient Egypt and has been an active practice since. This type of magic is a branch of traditional magic and can be practiced in various ways. Typically the more common use of love magic is through the work of spells, but other methods look like Charms Rituals-LOVE Potions-Dolls and even Amulets If you are interested in becoming a love caster, be prepared for what’s to come. A genuine love caster knows that the art of love casting is no easy feat and shouldn’t be done casually. You should know that not only does it require you to be gifted spiritually, but you must be ready to serve others. Someone who is considered a real love caster has experience in all manner of spells, no matter the difficulty. Training yourself in attraction, commitment, and marriage spells is an excellent place to start. But this by no means will make you a professional. Practice your craft and expand your knowledge; understand that you will possess the ability to help others in time truly. Types of Love Spells What better way to start broadening your experiences with love spells than by learning more about them? These spells work like just about any other spell. Simply apply your intention, use a medium (sigils, mantras, candles, or charm bags), and top it off with establishing the belief that you will receive what you want. So what kind of spells are available and which ones suit your needs the best? Let’s take a look at the many options you have at your disposal.

%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...

masabamasaba

Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...

Bert Jan Schrijver

Kürzlich hochgeladen (20)