Solr 8.4 has a new plugin system that portends of a better future much improved from today: (a) load plugins at runtime, (b) find 3rd party plugins in a registry, (c) fetch, install, and even configure plugins from the CLI, (d) a more slimmed down Solr distribution that is more secure. David will demonstrate what is available now and what the future may look like as the feature blossoms.

1. By David Smiley
On February 2020
At a NEST Meetup

2. › A Principal Software Engineer at Salesforce
› Apache Lucene/Solr committer & PMC member
› Author of Solr Enterprise Search Server books

3. › Quiz: how many plugin types? ____
– … in Lucene & Solr combined?
– A class/interface whose implementation can be specified via
configuration
› Many more actual plugins exist
› Search platforms need to be pluggable

4. › Where do you find out about plugins today?
– Solr Ref Guide
– Colleagues
– Conferences, Meetups
– Blogs
– Search results
› Plugin sources
– built-in: Comes with Solr; no installation step
– contrib: Comes with Solr; need to install JARs(*)
– 3rd party: External; compile/download, install

5. › Place JARs into a suitable directory
– <solr-home>/lib
– <solr-core>/lib
– solrconfig.xml <lib … />
Sometimes necessary:
– server/solr-webapp/webapp/WEB-INF/lib
Jetty/JVM plugins:
– server/lib and server/lib/ext

6. › Read documentation for how to configure
› Sometimes a plugin is always configured the same:
– <queryparser name=“foo”
class=“com.acme.FooQParser”/>
› Often not:
– TokenFilters: to which field type? Where in chain?

7. Consider RPM, Docker Hub, IDEs, many other examples
Why:
› Discoverability via a common registry
› Ease of installation via CLI commands
› And especially the 2nd order effects…
(break up the Solr monolith)

8. › Some built-in plugins become contribs
– Better security
– Fewer dependencies of Solr core (JAR hell deterrent)
– Usable in older Solr releases
– Ex: SolrTextTagger?
› Some contrib plugins become 3rd-party
– Better security
– Less burden on maintaining Solr (test time, docs, dev)
– Smaller Solr distribution / Docker image
– Separate release cycles from Solr
– Ex: DIH, Velocity/Solritas

9. › Support common plugin types
› A public plugin registry
› CLI & Web API commands
– to install a plugin
– to configure a plugin!
– to update a plugin (new version)
› Plugin management at runtime
› Plugin dependency isolation (JAR hell deterrent)
Easy to find
& install plugins!

10. Once upon a time…
› 2013-08, SOLR-5103, Grant Ingersoll
– Mostly discussion
› 2015-02 SOLR-7073, Noble Paul
– runtimeLib & .system collection / blob store
› 2017-05, SOLR-10665, Jan Høydahl
– Designs, POC, PF4J based, conf talk, … abandoned
› 2019-07, SOLR-13661, Noble Paul & Ishan C.
– Wider feedback; committed to v8.4

11. New in 8.4

12. › Plugin type
– A base class/interface that Lucene/Solr loads dynamically
› Plugin
– An implementation of a plugin type
› Package
– A collection of plugins, plus metadata
› Repository
– A web location (URL) hosting plugins

13. › Enable package manager
› Add a repository
› List packages
› Install a package
› Deploy a package
– CLI vs manual

14. › https://github.com/erikhatcher/solritas/

15. Likely more typical than automated…
<requestHandler name="/myhandler”
class="mypackage:full.path.to.MyClass” />
curl "http://localhost:8983/api/collections/collection1/
config/params"
-H 'Content-type:application/json’
-d "{set: {PKG_VERSIONS: {mypackage: '1.0.0'}}}"

16. › Where packages (JARs) are actually placed
› Distributed peer-to-peer synchronized directory
› Potential for other uses
– Large resources – machine learning models
› Compare to “.system” collection…
› Discuss: Good, bad, …

17. › Update a plugin live, in-place
– No core reload!
› Discuss: Good, bad, …

18. › Everything is deemed experimental
› Support more plugin types
– Schema plugins!
– Also hot-loading for more types; some can’t do this
› Convert contribs to a local plugin repository
› Create an official/primary plugin repository
› Better documentation
– See https://lucene.apache.org/solr/guide/8_4/package-
manager.html

19. › Who runs the plugin repository / decides inclusion
› Self-configuring plugins
› Hot-loading plugins
› Assign different package versions to different collections
› How a configSet declares it needs a package
› Eliminating <lib/> in solrconfig.xml
› Security (keys, signing, opting-out)

20. Resources:
› Ref guide
– https://lucene.apache.org/solr/guide/8_4/package-manager-
internals.html
› Solritas packaging example
– https://github.com/erikhatcher/solritas/
› Activate Conference Presentation
– https://youtu.be/YQBhEojf358
“Dynamic Plugin Loading & Safe Mode Solr”