Cassandra Lunch #23: Lucene Based Indexes on Cassandra

•

0 gefällt mir•378 views

During this week's Cassandra Lunch we covered packaged and DIY methods for Lucene-based indexes on Cassandra; as well as some pros and cons for using Lucene Based Indexes on Cassandra. Accompanying Blog: https://blog.anant.us/apache-cassandra-lunch-23-lucene-based-indexes-on-cassandra/ Cassandra Lunch Recording: https://youtu.be/Z0NXWmZAB8s Additional Resources: https://docs.datastax.com/en/dse/6.0/dse-dev/datastax_enterprise/search/searchTOC.html https://github.com/instaclustr/cassandra-lucene-index https://github.com/strapdata/elassandra https://github.com/tjake/Solandra Join Cassandra Lunch Weekly at 12 PM EST Every Wednesday: https://www.meetup.com/Cassandra-DataStax-DC/events/ Awesome Cassandra: https://github.com/Anant/awesome-cassandra Follow Us and Reach Us At: Anant: https://www.anant.us/ Cassandra.Link: https://cassandra.link/ Email: solutions@anant.us LinkedIn: https://www.linkedin.com/company/anant/ Twitter: https://twitter.com/anantcorp Eventbrite: https://www.eventbrite.com/o/anant-1072927283 Facebook: https://www.facebook.com/AnantCorp/

Technologie

Version 1.0
Lucene Based Indexes on Cassandra
An Anant Corporation Story.

Types Covered + Pros and Cons
● Packaged
● DIY
● Pros
● Cons

DSE Search / Solr
● https://docs.datastax.com/en/dse/6.0/dse-dev/datastax_enterprise/search
/searchTOC.html

Cassandra Lucene Index
● instaclustr/cassandra-lucene-index: Lucene based secondary indexes for
Cassandra
a. https://github.com/instaclustr/cassandra-lucene-index
● Mutations to Cassandra -> Mutations on Disk (for that node)

Elessandra / ElasticSearch
● strapdata/elassandra: Elassandra = Elasticsearch + Apache Cassandra
a. https://github.com/strapdata/elassandra
● Cassandra + Elasticsearch
a. Mutations to Cassandra -> Mutations to Elasticsearch

Event -> CQRS -> Cassandra + Index
● Write
○ Event / Command goes into an Event Source repository (Kafka, SQL
Table, etc. )
○ Command Processor processes it into CQL / Elasticsearch or SOLR or
Amazon ... Algolia
● Request
○ Event / Command goes into an Event Source repository (Kafka, SQL
Table, etc. )
○ Command Processor goes to index / ﬁnds the data, goes to Cassandra ,
gets the data, returns.
○ Query the index --

Cassandra -> Batch -> Index
● Writes to Cassandra
● Every now and then - Index to ???
● Cassandra + Spark

Serverless Function -> Cassandra + Index

Apache Niﬁ (Lucene) -> Niﬁ Processor -> Cassandra + Index

Pros
● extremely rich search capabilities
● Geospatial
● synonym
● fuzzy logic search
● Typos
● Stemming
● packaged elastic/solr/lucene -> shorter latency
● separate index -> better separation of concerns and speed

Cons
● pure lucene -> reinventing the wheel of what Solr/ElasticSearch
● external elastic/solr/ ?? -> longer latency between ﬁnding the data / getting
the data
● packaged elastic/solr/lucene -> don't expect it to solve all your problems
● consistency issues (if DIY)
● lucene is memory heavy
● lucene is disk heavy

Strategy: Scalable Fast Data
Architecture: Cassandra, Spark, Kafka
Engineering: Node, Python, JVM,CLR
Operations: Cloud, Container
Rescue: Downtime!! I need help.
www.anant.us | solutions@anant.us | (855) 262-6826
3 Washington Circle, NW | Suite 301 | Washington, DC 20037

Resources
● tjake/Solandra: Solandra = Solr + Cassandra

Weitere ähnliche Inhalte

Was ist angesagt?

In this talk we’ll present how at GetYourGuide we’ve built from scratch a completely new ETL pipeline using Debezium, Kafka, Spark and Airflow, which can automatically handle schema changes. Our starting point was an error prone legacy system that ran daily, and was vulnerable to breaking schema changes, which caused many sleepless on-call nights. As most companies, we also have traditional SQL databases that we need to connect to in order to extract relevant data. This is done usually through either full or partial copies of the data with tools such as sqoop. However another approach that has become quite popular lately is to use Debezium as the Change Data Capture layer which reads databases binlogs, and stream these changes directly to Kafka. As having data once a day is not enough anymore for our bussiness, and we wanted our pipelines to be resilent to upstream schema changes, we’ve decided to rebuild our ETL using Debezium. We’ll walk the audience through the steps we followed to architect and develop such solution using Databricks to reduce operation time. By building this new pipeline we are now able to refresh our data lake multiple times a day, giving our users fresh data, and protecting our nights of sleep.

Modern ETL Pipelines with Change Data Capture

Databricks

Using Hazelcast in the Kappa architecture

Oliver Buckley-Salmon

Introduction to Data Engineer and Data Pipeline at Credit OK

Kriangkrai Chaonithi

Lambda Architecture has been a common way to build data pipelines for a long time, despite difficulties in maintaining two complex systems. An alternative, Kappa Architecture, was proposed in 2014, but many companies are still reluctant to switch to Kappa. And there is a reason for that: even though Kappa generally provides a simpler design and similar or lower latency, there are a lot of practical challenges in areas like exactly-once delivery, late-arriving data, historical backfill and reprocessing. In this talk, I want to show how you can solve those challenges by embracing Apache Kafka as a foundation of your data pipeline and leveraging modern stream-processing frameworks like Apache Kafka Streams and Apache Flink.

It's Time To Stop Using Lambda Architecture | Yaroslav Tkachenko, Shopify

HostedbyConfluent

Designing and Implementing a Real-time Data Lake with Dynamically Changing Sc...

Databricks

ASPgems - kappa architecture

Juantomás García Molina

Lambda architecture

Mario Alexandro Santini

Lambda architecture: from zero to One

Serg Masyutin

We often need to build applications that analyze Kafka data to unlock the most value from event streams, so how can organizations build these real-time analytics applications? In this talk, we examine an indexing approach that enables fast SQL analytics on data from Kafka, without data flattening or denormalization. Rockset is the real-time indexing database that builds an inverted index, a columnar index and a row index on all fields of your Kafka messages, including nested fields and arrays. This Converged Index accelerates various types of analytic queries–search, aggregations and joins–without the need to denormalize or transform data for performance reasons. With indexing delivering significant gains in query performance, we also need to index new data in a timely manner. We discuss several strategies used for efficient ingestion and indexing from Kafka, including rollups, write optimizations on the underlying RocksDB storage engine, and the disaggregation of ingest and query compute.

Sub-Second SQL Search, Aggregations and Joins with Kafka and Rockset | Dhruba...

HostedbyConfluent

Lambda architecture @ Indix

Rajesh Muppalla

Apache Spark Streaming

Bartosz Jankiewicz

Efficiently Building Machine Learning Models for Predictive Maintenance in th...

Databricks

Enforcing format, changing schema, introducing privacy filters have always been a challenge with the classical Kafka-API. In this talk we'll cover how to extend existing applications with webassembly, allowing developers to change the shape of data at runtime, per application without creating additional topics. By leveraging WebAssembly, we can extend the capabilities of the Kafka-API beyond what it was initially imagined. Come and learn about the future of the Kafka-API

Data Policies for the Kafka-API with WebAssembly | Alexander Gallego, Vectorized

HostedbyConfluent

Business leads, executives, analysts, and data scientists rely on up-to-date information to make business decision, adjust to the market, meet needs of their customers or run effective supply chain operations. Come hear how Asurion used Delta, Structured Streaming, AutoLoader and SQL Analytics to improve production data latency from day-minus-one to near real time Asurion’s technical team will share battle tested tips and tricks you only get with certain scale. Asurion data lake executes 4000+ streaming jobs and hosts over 4000 tables in production Data Lake on AWS.

Large Scale Lakehouse Implementation Using Structured Streaming

Databricks

Are you looking for a cloud-based architecture that includes the best of breed streaming and database technologies? In this session you will learn how to setup and configure the Confluent Cloud with MongoDB Atlas. We'll start the journey learning about the basic connectivity between the two cloud services and end with a brief discovery of what you can do with data once it is in MongoDB Atlas. By the end of this session you will know how to securely setup and configure the MongoDB Atlas connectors in the Confluent Cloud in both a source and sink configuration.

Streaming data in the cloud with Confluent and MongoDB Atlas | Robert Waters,...

HostedbyConfluent

Superset druid realtime

arupmalakar

AI is fundamentally transforming how we live and work. Zalando is a data driven company. We deliver an optimal customer experience that drives engagement. We continue to improve this experience by leveraging the latest technologies and machine learning techniques — such as building a cutting edge cloud based infrastructure to support our operations at scale. We provide our data scientists across Zalando with the means to implement artificial intelligence use cases, leveraging data from all parts of our company and the best machine learning techniques from across the industry. Apache Spark delivered through Databricks is at the core of this strategy. In this keynote, I’ll share our AI journey thus far, and share how we are exploring ways to unify data through A.I. with Spark and Databricks.

The Evolution of the Fashion Retail Industry in the Age of AI with Kshitij Ku...

Databricks

Javantura v3 - Real-time BigData ingestion and querying of aggregated data – ...

HUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association

Uber Geo spatial data platform at DataWorks Summit

Zhenxiao Luo

Kafka Summit SF 2017 - Keynote - Managing Data at Scale: The Unreasonable Eff...

confluent

Was ist angesagt? (20)

Modern ETL Pipelines with Change Data Capture

Using Hazelcast in the Kappa architecture

Introduction to Data Engineer and Data Pipeline at Credit OK

It's Time To Stop Using Lambda Architecture | Yaroslav Tkachenko, Shopify

Designing and Implementing a Real-time Data Lake with Dynamically Changing Sc...

ASPgems - kappa architecture

Lambda architecture

Lambda architecture: from zero to One

Sub-Second SQL Search, Aggregations and Joins with Kafka and Rockset | Dhruba...

Lambda architecture @ Indix

Apache Spark Streaming

Efficiently Building Machine Learning Models for Predictive Maintenance in th...

Data Policies for the Kafka-API with WebAssembly | Alexander Gallego, Vectorized

Large Scale Lakehouse Implementation Using Structured Streaming

Streaming data in the cloud with Confluent and MongoDB Atlas | Robert Waters,...

Superset druid realtime

The Evolution of the Fashion Retail Industry in the Age of AI with Kshitij Ku...

Javantura v3 - Real-time BigData ingestion and querying of aggregated data – ...

Uber Geo spatial data platform at DataWorks Summit

Kafka Summit SF 2017 - Keynote - Managing Data at Scale: The Unreasonable Eff...

Ähnlich wie Cassandra Lunch #23: Lucene Based Indexes on Cassandra

In Cassandra Lunch #58, Rahul Singh will be leading a presentation covering a Cassandra topic we're sure you won't want to miss. Accompanying Blog: Coming Soon! Accompanying YouTube: https://youtu.be/l4mAt3MDVZA Sign Up For Our Newsletter: http://eepurl.com/grdMkn Join Cassandra Lunch Weekly at 12 PM EST Every Wednesday: https://www.meetup.com/Cassandra-DataStax-DC/events/ Cassandra.Link: https://cassandra.link/ Follow Us and Reach Us At: Anant: https://www.anant.us/ Awesome Cassandra: https://github.com/Anant/awesome-cassandra Cassandra.Lunch: https://github.com/Anant/Cassandra.Lunch Email: solutions@anant.us LinkedIn: https://www.linkedin.com/company/anant/ Twitter: https://twitter.com/anantcorp Eventbrite: https://www.eventbrite.com/o/anant-1072927283 Facebook: https://www.facebook.com/AnantCorp/ Join The Anant Team: https://www.careers.anant.us

Apache Cassandra Lunch #58: Tools for Cassandra Titans

Anant Corporation

Indexing 3-dimensional trajectories: Apache Spark and Cassandra integration

Cesare Cugnasco

Pollfish is a survey platform which provides access to millions of targeted users. Pollfish allows easy distribution and targeting of surveys through existing mobile apps. (https://www.pollfish.com/). At pollfish we use Cassandra for difference use cases, eg. for application data store to maximize write throughput when appropriate and for our analytics project to find insights in application generated data. As a medium to accomplish our success so far, we use the Datastax's DSE 4.6 environment which integrates Appache Cassadra, Spark and a hadoop compatible file system (CFS). We will discuss how we started, how the journey was and the impressions gained so far along with some tips learned the hard way. This is a result of joint work of an excellent team here at Pollfish.

Cassandra at Pollfish

Pollfish

Cassandra at Pollfish

Stavros Kontopoulos

In Cassandra Lunch #96, Rahul Singh, CEO of Anant, will discuss different ways to get change data into and out of Cassandra using a few different strategies which could work out for your platform. Accompanying Blog: Coming Soon! Accompanying YouTube: https://youtu.be/HjgqAhsCQ64 Sign Up For Our Newsletter: http://eepurl.com/grdMkn Join Cassandra Lunch Weekly at 12 PM EST Every Wednesday: https://www.meetup.com/Cassandra-DataStax-DC/events/ Cassandra.Link: https://cassandra.link/ Follow Us and Reach Us At: Anant: https://www.anant.us/ Awesome Cassandra: https://github.com/Anant/awesome-cassandra Cassandra.Lunch: https://github.com/Anant/Cassandra.Lunch Email: solutions@anant.us LinkedIn: https://www.linkedin.com/company/anant/ Twitter: https://twitter.com/anantcorp Eventbrite: https://www.eventbrite.com/o/anant-1072927283 Facebook: https://www.facebook.com/AnantCorp/ Join The Anant Team: https://www.careers.anant.us

Apache Cassandra Lunch #96: Apache Cassandra Change Data Capture (CDC) Strate...

Anant Corporation

Enterprise Search is a challenging problem for most organizations. Public search technologies such as Google can index content and use link popularity to rank content in addition to the basic keyword matches. Enterprise Search is different. Sometimes it requires specially designed indexes as well as several processing steps. At the U.S. Patent & Trademark Office, part of the Department of Commerce, a team of professionals is building the next generation of search tools using open source technologies. Like any large undertaking, it’s not a simple plug and play project. Main topics to be covered in this talk: + Architectures for Large Scale Enterprise Search + Leveraging Apache Cassandra & Spark + Customizing / Configuring Apache SolR and Indexing + Writing a custom Parser for SolR in Scala

Building Enterprise Search Engines using Open Source Technologies

Rahul Singh

Building Enterprise Search Engines using Open Source Technologies

Anant Corporation

We will dissect the world famous todo app that provides a REST API (which is the foundation of microservices) with data backed by Apache Cassandra. We will leverage the TODO MVC and the TODO backend projects with the back end that we will build with Quarkus and Cassandra. Attendees will get an overview of Cassandra, including the driver for Quarkus. Through live coding (that attendees can try out later) in a cloud-based environment, primarily in Quarkus and Cassandra, attendees will understand how to implement and connect the APIs to the backend and leverage the generic client(s)provided. After attending this session attendees will walk away with a good understanding of implementing microservices using Cassandra and Quarkus. They will also get a working knowledge of how Astra (Cassandra as a service) can be leveraged in other solutions.

A Microservices approach with Cassandra and Quarkus | DevNation Tech Talk

Red Hat Developers

Big Data Technologies

Anant Corporation

Avoiding Pitfalls for Cassandra.pdf

Cédrick Lunven

Presenter: Evan Chan, Principal Software Engineer at Socrata Inc. How do you rapidly derive complex insights on top of really big data sets in Cassandra? This session draws upon Evan's experience building a distributed, interactive, columnar query engine on top of Cassandra and Spark. We will start by surveying the existing query landscape of Cassandra and discuss ways to integrate Cassandra and Spark. We will dive into the design and architecture of a fast, column-oriented query architecture for Spark, and why columnar stores are so advantageous for OLAP workloads. I will present a schema for Parquet-like storage of analytical datasets on Cassandra. Find out why Cassandra and Spark are the perfect match for enabling fast, scalable, complex querying and storage of big analytical data.

Cassandra Summit 2014: Interactive OLAP Queries using Apache Cassandra and Spark

DataStax Academy

Speaker(s): Erich Ess, CTO at SimpleRelevance An introduction to using Spark and integrating Spark with Cassandra to perform analytics and data processing. If you are using Cassandra, then you almost certainly have a large amount of data which you want upon which some form of processing needs to be done. Spark is a new distributed computing platform which makes writing big data analytics incredibly easy and it integrates with Cassandra with minimal effort. In this presentation, I will show how to get Spark setup with Cassandra and how to interact with Cassandra through Spark. Also covered will be architectural details of Spark such as how it handles failure recovery along with the main programming concepts needed to implement simple through complex processes on Spark. Then I will walk through a demonstration of using Spark to perform ETL on a dataset and save the data into Cassandra and then use Spark to do analytics on the data in Cassandra. Time permitting I will explain some things we are using Spark for at SimpleRelevance and cover Spark Streaming.

Kindling: Getting Started with Spark and Cassandra

DataStax Academy

Kafka spark cassandra webinar feb 16 2016

Hiromitsu Komatsu

Kafka spark cassandra webinar feb 16 2016

Hiromitsu Komatsu

Apache Solr! Enterprise Search Solutions at your Fingertips!

Murshed Ahmmad Khan

Spark Introduction

DataStax Academy

Apache Cassandra is rock-solid and widely deployed for OLTP and real-time applications, but it is typically not thought of as an OLAP database for analytical queries. This talk will show architectures and techniques for combining Apache Cassandra and Spark to yield a 10-1000x improvement in OLAP analytical performance. We will then introduce a new open-source project that combines the above performance improvements with the ease of use of Apache Cassandra, and compare it to implementations based on Hadoop and Parquet. First, the existing Cassandra Spark connector allows one to easily load data from Cassandra to Spark. We'll cover how to accelerate queries through different caching options in Spark, and the tradeoffs and limitations around performance, memory, and updating data in real time. We then dive into the use of columnar storage layout and efficient coding techniques that dramatically speed up I/O for OLAP use cases. Cassandra features like triggers and custom secondary indexes allow for easy data ingestion into columnar format. Next, we explore how to integrate this new storage with Spark SQL and its pluggable data storage API. Future developments will enable extreme analytical database performance, including smart caching of column projections, a columnar version of Spark's Catalyst execution planner, and how vectorization makes for fast cache- and GPU-friendly calculations - see Spark's Project Tungsten. FiloDB is a new open-source database using the above techniques to combine very fast Spark SQL analytical queries with the ease of use of Cassandra. We will briefly cover interesting use cases, such as: * Easy exactly-once ingestion from Kafka for streaming and IoT applications * Incremental computed columns and geospatial annotations. We'll discuss how FiloDB improves aggregations needed for choropleth maps over standard PostGIS solutions.

TupleJump: Breakthrough OLAP performance on Cassandra and Spark

DataStax Academy

FiloDB - Breakthrough OLAP Performance with Cassandra and Spark

Evan Chan

In these slides is given an overview of the different parts of Apache Spark. We analyze spark shell both in scala and python. Then we consider Spark SQL with an introduction to Data Frame API. Finally we describe Spark Streaming and we make some code examples. Topics:spark-shell, pyspark, HDFS, how to copy file to HDFS, spark transformations, spark actions, Spark SQL (Shark), spark streaming, streaming transformation stateless vs stateful, sliding windows, examples

11. From Hadoop to Spark 2/2

Fabio Fumarola

Breakthrough OLAP performance with Cassandra and Spark

Evan Chan

Ähnlich wie Cassandra Lunch #23: Lucene Based Indexes on Cassandra (20)

Apache Cassandra Lunch #58: Tools for Cassandra Titans

Indexing 3-dimensional trajectories: Apache Spark and Cassandra integration

Cassandra at Pollfish

Apache Cassandra Lunch #96: Apache Cassandra Change Data Capture (CDC) Strate...

Building Enterprise Search Engines using Open Source Technologies

A Microservices approach with Cassandra and Quarkus | DevNation Tech Talk

Big Data Technologies

Avoiding Pitfalls for Cassandra.pdf

Cassandra Summit 2014: Interactive OLAP Queries using Apache Cassandra and Spark

Kindling: Getting Started with Spark and Cassandra

Kafka spark cassandra webinar feb 16 2016

Apache Solr! Enterprise Search Solutions at your Fingertips!

Spark Introduction

TupleJump: Breakthrough OLAP performance on Cassandra and Spark

FiloDB - Breakthrough OLAP Performance with Cassandra and Spark

11. From Hadoop to Spark 2/2

Breakthrough OLAP performance with Cassandra and Spark

Mehr von Anant Corporation

QLoRA Fine-Tuning on Cassandra Link Data Set (1/2) Cassandra Lunch 137

Anant Corporation

Kono.IntelCraft.Weekly.AI.LLM.Landscape.2024.02.28.pdf

Anant Corporation

Data Engineer's Lunch 96: Intro to Real Time Analytics Using Apache Pinot

Anant Corporation

Series: Using AI / ChatGPT at Work - GPT Automation Are you a small business owner or web developer interested in leveraging the power of GPT (Generative Pretrained Transformer) technology to enhance your business processes? If so, Join us for a series of events focused on using GPT in business. Whether you're a small business owner or a web developer, you'll learn how to leverage GPT to improve your workflow and provide better services to your customers. GPT Automation: What it is and How it Works How Time-Saving GPT Automation Can Improve Your Business Cost-Effective GPT Automation: How it Can Save Your Business Money Using GPT Automation for Customer Service: Benefits and Best Practices The Power of GPT Automation for Content Creation Data Analysis Made Easy with GPT Automation Top GPT-3 Automation Tools for Businesses The Ethical Considerations of GPT Automation Overcoming Bias in GPT Automation: Best Practices The Future of GPT Automation: Trends and Predictions Since we focus on "no code" here, we'll explore the tools that are already out there such as ChatGPT plugins for Chrome, OpenAI GPT API, low-code/no-code platforms like Make/Integromat and Zapier, existing apps like Jasper/Rytr, and ecosystem tools like Everyprompt. We'll also discuss the resources available for those interested in learning more about GPT, including other people’s prompts.

NoCode, Data & AI LLM Inside Bootcamp: Episode 6 - Design Patterns: Retrieval...

Anant Corporation

Episode 3: The LLM / GPT / AI Prompt / Data Engineer Roadmap In this episode, we'll discuss the history, fundamentals, and the different flavors of LLMs available, beyond GPT/ChatGPT. This is a dry run of a session that will be on a LLM Bootcamp ( Fill out the survey on the link if you are interested in an in-person vs. virtual session) Intro / Fundamentals of LLM LLM Foundations History of LLMs Tuning, Training, or "In Context Learning" with LLMs What is "Prompt Engineering" Case for Augmenting LLMs Series: Using AI / ChatGPT at Work - GPT Automation Are you a small business owner or web developer interested in leveraging the power of GPT (Generative Pretrained Transformer) technology to enhance your business processes? If so, Join us for a series of events focused on using GPT in business. Whether you're a small business owner or a web developer, you'll learn how to leverage GPT to improve your workflow and provide better services to your customers. GPT Automation: What it is and How it Works How Time-Saving GPT Automation Can Improve Your Business Cost-Effective GPT Automation: How it Can Save Your Business Money Using GPT Automation for Customer Service: Benefits and Best Practices The Power of GPT Automation for Content Creation Data Analysis Made Easy with GPT Automation Top GPT-3 Automation Tools for Businesses The Ethical Considerations of GPT Automation Overcoming Bias in GPT Automation: Best Practices The Future of GPT Automation: Trends and Predictions Since we focus on "no code" here, we'll explore the tools that are already out there such as ChatGPT plugins for Chrome, OpenAI GPT API, low-code/no-code platforms like Make/Integromat and Zapier, existing apps like Jasper/Rytr, and ecosystem tools like Everyprompt. We'll also discuss the resources available for those interested in learning more about GPT, including other people’s prompts.

Automate your Job and Business with ChatGPT #3 - Fundamentals of LLM/GPT

Anant Corporation

In Apache Cassandra Lunch #131: YugabyteDB Developer Tools, we discussed third party developer tools that are compatible with YugabyteDB. We talked about using Yugabyte Developer Tools for data visualization and schema management. The live recording of Cassandra Lunch, which includes a more in-depth discussion and a demo, is embedded below in case you were not able to attend live. If you would like to attend Apache Cassandra Lunch live, it is hosted every Wednesday at 12 PM EST. Developer tools play a critical role in simplifying and streamlining database development and management. They allow developers and administrators to be more productive, reducing the time and effort required to create and maintain database schemas, write SQL queries, test database performance, and enable collaboration. Developer tools also make it possible to track changes over time, improving the ability to manage the entire development lifecycle.

YugabyteDB Developer Tools

Anant Corporation

In this episode we'll discuss the different flavors of prompt engineering in the LLM/GPT space. According to your skill level you should be able to pick up at any of the following: Leveling up with GPT 1: Use ChatGPT / GPT Powered Apps 2: Become a Prompt Engineer on ChatGPT/GPT 3: Use GPT API with NoCode Automation, App Builders 4: Create Workflows to Automate Tasks with NoCode 5: Use GPT API with Code, make your own APIs 6: Create Workflows to Automate Tasks with Code 7: Use GPT API with your Data / a Framework 8: Use GPT API with your Data / a Framework to Make your own APIs 9: Create Workflows to Automate Tasks with your Data /a Framework 10: Use Another LLM API other than GPT (Cohere, HuggingFace) 11: Use open source LLM models on your computer 12: Finetune / Build your own models Series: Using AI / ChatGPT at Work - GPT Automation Are you a small business owner or web developer interested in leveraging the power of GPT (Generative Pretrained Transformer) technology to enhance your business processes? If so, Join us for a series of events focused on using GPT in business. Whether you're a small business owner or a web developer, you'll learn how to leverage GPT to improve your workflow and provide better services to your customers.

Episode 2: The LLM / GPT / AI Prompt / Data Engineer Roadmap

Anant Corporation

In Data Engineer’s Lunch #89: Machine Learning Orchestration with Airflow, we discussed using Apache Airflow to manage and schedule machine learning tasks. By following the best practices of ML Ops, teams can streamline their ML workflows and build scalable, efficient, and accurate models that deliver real-world business value. Properly implemented ML Ops can help organizations stay ahead of the curve and achieve their goals in the fast-paced world of machine learning. Apache Airflow is an open-source tool for scheduling and automating workflows. Airflow allows you to define workflows in Python, with tasks defined as Python functions that can include Operators for all sorts of external tools. This makes it easy to automate repeated processes and define dependencies between tasks, creating directed-acyclic-graphs of tasks that can be scheduled using cron syntax or frequency tasks. Airflow also features a user-friendly UI for monitoring task progress and viewing logs, giving you greater control over your data pipeline.

Machine Learning Orchestration with Airflow

Anant Corporation

If you didn't attend, you don't want to miss a much shorter synopsis of what was covered and get some thoughts from us as to why they are important. We'll talk about the main topics of the event. 1. ACID transactions on Cassandra by Aaron Ploetz, Datastax 2. Apache Flink with Apache Cassandra at Satyajit Thadeswar, Netflix 3. Durable Execution built on Apache Cassandra by Loren Sands-Ramshaw, Temporal 4. Switching from Mongo to Cassandra with Mongoose & new Stargate JSON API, Valeri Karpov 5. Cloud Native and Realtime AI/ML with Patrick Mcfadin and Davor Boncaci, Datastax

Cassandra Lunch 130: Recap of Cassandra Forward Talks

Anant Corporation

Data Engineer's Lunch 90: Migrating SQL Data with Arcion

Anant Corporation

Data Engineer's Lunch 89: Machine Learning Orchestration with AirflowMachine ...

Anant Corporation

Cassandra Lunch 129: What’s New: Apache Cassandra 4.1+ Features & Future

Anant Corporation

As the demand for real-time data processing continues to grow, so too do the challenges associated with building production-ready applications that can handle large volumes of data and handle it quickly. In this talk, we will explore common problems faced when building real-time applications at scale, with a focus on a specific use case: detecting and responding to cyclist crashes. Using telemetry data collected from a fitness app, we’ll demonstrate how we used a combination of Apache Kafka and Python-based microservices running on Kubernetes to build a pipeline for processing and analyzing this data in real-time. We'll also discuss how we used machine learning techniques to build a model for detecting collisions and how we implemented notifications to alert family members of a crash. Our ultimate goal is to help you navigate the challenges that come with building data-intensive, real-time applications that use ML models. By showcasing a real-world example, we aim to provide practical solutions and insights that you can apply to your own projects. Key takeaways: An understanding of the common challenges faced when building real-time applications at scale Strategies for using Apache Kafka and Python-based microservices to process and analyze data in real-time Tips for implementing machine learning models in a real-time application Best practices for responding to and handling critical events in a real-time application

Data Engineer's Lunch #86: Building Real-Time Applications at Scale: A Case S...

Anant Corporation

Data Engineer's Lunch #85: Designing a Modern Data Stack

Anant Corporation

CL 121

Anant Corporation

Data Engineer's Lunch #83: Strategies for Migration to Apache Iceberg

Anant Corporation

In this lunch, Johnny will show us how easy it is to start monitoring your Cassandra cluster in minutes. He will explain the various aspects and features of Cassandra that need to be monitored, how to do it, and most importantly why! Approaches for backups and Cassandra repairs will be discussed and explored in detail. Learn how AxonOps significantly reduces the complexity and overhead when looking after Cassandra and ensures your Cassandra cluster is reliable and resilient. Experienced developer, DevOps, architect, and AxonOps co-founder, Johnny Miller, has worked with a wide variety of companies – from small start-ups to large enterprises. He has been working with Cassandra for many years and has a deep understanding of the challenges facing modern companies looking to adopt Apache Cassandra.

Apache Cassandra Lunch 120: Apache Cassandra Monitoring Made Easy with AxonOps

Anant Corporation

In Apache Cassandra Lunch #119, Rahul Singh will cover a refresher on GUI desktop/web tools for users that want to get their hands dirty with Cassandra but don't want to deal with CQLSH to do simple queries. Some of the tools are web-based and others are installed on your desktop. Since the beginning days of Cassandra, a lot has changed and there are many options for command-line-haters to use Cassandra.

Apache Cassandra Lunch 119: Desktop GUI Tools for Apache Cassandra

Anant Corporation

Data Engineer's Lunch #82: Automating Apache Cassandra Operations with Apache...

Anant Corporation

Data Engineer's Lunch #60: Series - Developing Enterprise Consciousness

Anant Corporation

Mehr von Anant Corporation (20)

QLoRA Fine-Tuning on Cassandra Link Data Set (1/2) Cassandra Lunch 137

Kono.IntelCraft.Weekly.AI.LLM.Landscape.2024.02.28.pdf

Data Engineer's Lunch 96: Intro to Real Time Analytics Using Apache Pinot

NoCode, Data & AI LLM Inside Bootcamp: Episode 6 - Design Patterns: Retrieval...

Automate your Job and Business with ChatGPT #3 - Fundamentals of LLM/GPT

YugabyteDB Developer Tools

Episode 2: The LLM / GPT / AI Prompt / Data Engineer Roadmap

Machine Learning Orchestration with Airflow

Cassandra Lunch 130: Recap of Cassandra Forward Talks

Data Engineer's Lunch 90: Migrating SQL Data with Arcion

Data Engineer's Lunch 89: Machine Learning Orchestration with AirflowMachine ...

Cassandra Lunch 129: What’s New: Apache Cassandra 4.1+ Features & Future

Data Engineer's Lunch #86: Building Real-Time Applications at Scale: A Case S...

Data Engineer's Lunch #85: Designing a Modern Data Stack

CL 121

Data Engineer's Lunch #83: Strategies for Migration to Apache Iceberg

Apache Cassandra Lunch 120: Apache Cassandra Monitoring Made Easy with AxonOps

Apache Cassandra Lunch 119: Desktop GUI Tools for Apache Cassandra

Data Engineer's Lunch #82: Automating Apache Cassandra Operations with Apache...

Data Engineer's Lunch #60: Series - Developing Enterprise Consciousness

Kürzlich hochgeladen

What's New in Teams Calling, Meetings and Devices April 2024

Stephanie Beckett

PLAI - Acceleration Program for Generative A.I. Startups

Stefano

The Metaverse: Are We There Yet?

Mark Billinghurst

Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf

FIDO Alliance

ScyllaDB has the potential to deliver impressive performance and scalability. The better you understand how it works, the more you can squeeze out of it. But before you squeeze, make sure you know what to monitor! Watch our experienced Postgres developer work through monitoring and performance strategies that help him understand what mistakes he’s made moving to NoSQL. And learn with him as our database performance expert offers friendly guidance on how to use monitoring and performance tuning to get his sample Rust application on the right track. This webinar focuses on using monitoring and performance tuning to discover and correct mistakes that commonly occur when developers move from SQL to NoSQL. For example: - Common issues getting up and running with the monitoring stack - Using the CQL optimizations dashboard - Common issues causing high latency in a node - Common issues causing replica imbalance - What a healthy system looks like in terms of memory - Key metrics to keep an eye on This isn’t “Death-by-Powerpoint.” We’ll walk through problems encountered while migrating a real application from Postgres to ScyllaDB – and try to fix them live as well.

Optimizing NoSQL Performance Through Observability

ScyllaDB

New customer? New industry? New cloud? New team? A lot to handle! How to ensure the success of the project? Start it well! I've created the 3 areas of focus at the beginning of the project that helped me in multiple roles (BA, PO, and Consultant). Learn from real-world experiences and discover how these insights can empower you to deliver unparalleled value to your customers right from the project's start.

Powerful Start- the Key to Project Success, Barbara Laskowska

CzechDreamin

Google I/O Extended 2024 Warsaw

GDSC PJATK

We're living the AI revolution and Salesforce is adapting and bring new value to their customers. Einstein products are evolving rapidly and navigating their limitations, language support, and use cases can be challenging. Let's make review of what Einstein product are available currently, what are the capabilities and what can be used for in CEE region and how Rossie.ai can help to learn Salesforce speak Czech. We will explore the Einstein roadmap and I will make a short live demo (based on your vote) of some Einstein feature.

AI revolution and Salesforce, Jiří Karpíšek

CzechDreamin

Webinar Recording: https://www.panagenda.com/webinars/alles-neu-macht-der-mai-wir-durchleuchten-den-verbesserten-notes-eigenschaftendialog/ Haben Sie sich schon einmal über den zu kleinen Eigenschaftendialog in Notes geärgert? Mussten Sie einen Agenten oder eine Aktion erstellen, um schnell mal ein Feld zu ändern? Haben Sie jedes mal endlos nach dem zu vergleichenden Feld gesucht, nachdem Sie ein neues Dokument ausgewählt haben? Wollten Sie das verdammte Ding einfach nur größer machen? Zum Glück gibt es dafür eine Lösung – und sie ist wahrscheinlich bereits installiert! Mit dem kostenlosen panagenda Document Properties (Pro) erhalten Sie den Eigenschaftendialog, den Sie schon immer haben wollten. Größer, anpassbar, und im Volltext durchsuchbar. Sehen Sie mehrere Dokumente gleichzeitig oder vergleichen Sie mit einem Diff-Viewer. Ändern Sie beliebige Felder und haben Sie endlich eine einfache Möglichkeit, Profildokumente für alle Benutzer zu verwalten. Entdecken Sie mit HCL Ambassador Marc Thomas, wie Document Properties Ihre Arbeit vereinfachen und Sie bei der täglichen Verwendung von Domino-Anwendungen unterstützen kann – im Client oder im Designer. Sie werden es nicht bereuen! Für Sie in diesem Webinar - Was Document Properties ist, welche Editionen es gibt und wo es in Notes und Domino Designer zu finden ist - Wie Sie nach einem beliebigen Feld suchen und es bearbeiten, Dokumente vergleichen oder alle Daten per CSV exportieren können - Suchen, Bearbeiten und auch Löschen von Profildokumenten - Welche Konfigurationseinstellungen verfügbar sind, um Funktionen anzupassen - Wie Ihre Endbenutzer davon profitieren - Sehen Sie alles in einer Live-Demo

Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...

panagenda

Intrigued by why some of the world's largest companies (Netflix, Google, Cisco, Twitter, Uber etc) are using gRPC? In this demo based talk we delve into the world of gRPC in .Net, what it does and why we should use it. We compare the interface with both Rest and graphQL. We will show you how to implement grpc server-side in .net and in the web. Finally, I will show you how the tooling helps you deliver powerful interfaces and interact with them quickly and simply.

Demystifying gRPC in .Net by John Staveley

John Staveley

Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf

FIDO Alliance

Are you wondering why everyone is talking about the Flutterwave scandal? You’re not alone! It’s been in the news a lot. We’re going to tell you all about the Flutterwave Scandal in a way that’s easy to get. Keep reading, because we’re going to share the whole story with you. The company Flutterwave is headquartered in San Francisco, California, United States. In 2016, Iyinoluwa Aboyeji, Olugbenga Agboola, and Adeleke Adekoya established Flutterwave. It offers a payment infrastructure to international merchants and payment service providers throughout Africa. The company operates in various African countries including Nigeria, Kenya, Uganda, Ghana, South Africa, and others. They focus on digital payments, helping businesses accept and process payments on various channels like the web, mobile, ATM, and POS. Recently, they’ve been in the news due to allegations of misconduct within the company, which has affected their reputation and value. Flutterwave is an essential player in the African fintech landscape, aiming to drive growth for banks and businesses through digital payment solutions. The Flutterwave scandal involves allegations of misconduct and inappropriate behaviour towards female employees by the company’s co-founder and CEO, Olugbenga Agboola. Reports have surfaced from both current and former employees about bullying, intimidation, and sexual harassment at work. The allegations of inappropriate behaviour towards female employees at Flutterwave, specifically involving the CEO Olugbenga Agboola, were brought to public attention in April 2022.This was when Clara Wanjiku Odero, an ex-employee and current CEO of Credrails, published a Medium post and a series of tweets on April 4, 2022, accusing Flutterwave and Agboola of bullying. These allegations were part of a broader range of misconduct claims that surfaced around the same time The reasons behind the scandal are rooted in accusations of unethical behavior within the company. The allegations suggest a workplace culture that allowed for misconduct and failed to protect employees from harassment and intimidation. Specifically, the Flutterwave scandal refers to the series of events where the CEO was accused of engaging in improper conduct with female colleagues. This led to a broader investigation into the company’s practices and raised serious concerns about the fintech’s corporate governance. The scandal had significant repercussions, including a drop in the company’s stock price and a loss of trust among customers and partners. The trust that customers and investors had in Flutterwave was greatly damaged. People started to doubt the company’s integrity and whether their money and personal information were safe. Market Impact The scandal shook the entire fintech market. Flutterwave’s competitors saw this as an opportunity and started to attract customers who were leaving Flutterwave.

Breaking Down the Flutterwave Scandal What You Need to Know.pdf

UK Journal

FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...

FIDO Alliance

I'm excited to share my latest predictions on how AI, robotics, and other technological advancements will reshape industries in the coming years. The slides explore the exponential growth of computational power, the future of AI and robotics, and their profound impact on various sectors. Why this matters: The success of new products and investments hinges on precise timing and foresight into emerging categories. This deck equips founders, VCs, and industry leaders with insights to align future products with upcoming tech developments. These insights enhance the ability to forecast industry trends, improve market timing, and predict competitor actions. Highlights: ▪ Exponential Growth in Compute: How $1000 will soon buy the computational power of a human brain ▪ Scaling of AI Models: The journey towards beyond human-scale models and intelligent edge computing ▪ Transformative Technologies: From advanced robotics and brain interfaces to automated healthcare and beyond ▪ Future of Work: How automation will redefine jobs and economic structures by 2040 With so many predictions presented here, some will inevitably be wrong or mistimed, especially with potential external disruptions. For instance, a conflict in Taiwan could severely impact global semiconductor production, affecting compute costs and related advancements. Nonetheless, these slides are intended to guide intuition on future technological trends.

Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl

Peter Udo Diehl

Overview of Hyperledger Foundation

Hyperleger Tokyo Meetup

Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...

FIDO Alliance

Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf

FIDO Alliance

Using IESVE for Room Loads Analysis - UK & Ireland

IES VE

ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...

FIDO Alliance

Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...

FIDO Alliance

Kürzlich hochgeladen (20)

What's New in Teams Calling, Meetings and Devices April 2024

PLAI - Acceleration Program for Generative A.I. Startups

The Metaverse: Are We There Yet?

Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf

Optimizing NoSQL Performance Through Observability

Powerful Start- the Key to Project Success, Barbara Laskowska

Google I/O Extended 2024 Warsaw

AI revolution and Salesforce, Jiří Karpíšek

Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...

Demystifying gRPC in .Net by John Staveley

Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf

Breaking Down the Flutterwave Scandal What You Need to Know.pdf

FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...

Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl

Overview of Hyperledger Foundation

Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...

Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf

Using IESVE for Room Loads Analysis - UK & Ireland

ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...

Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...

Cassandra Lunch #23: Lucene Based Indexes on Cassandra

1. Version 1.0 Lucene Based Indexes on Cassandra An Anant Corporation Story.

2. Types Covered + Pros and Cons ● Packaged ● DIY ● Pros ● Cons

3. Packaged

4. DSE Search / Solr ● https://docs.datastax.com/en/dse/6.0/dse-dev/datastax_enterprise/search /searchTOC.html

5. Cassandra Lucene Index ● instaclustr/cassandra-lucene-index: Lucene based secondary indexes for Cassandra a. https://github.com/instaclustr/cassandra-lucene-index ● Mutations to Cassandra -> Mutations on Disk (for that node)

6. Elessandra / ElasticSearch ● strapdata/elassandra: Elassandra = Elasticsearch + Apache Cassandra a. https://github.com/strapdata/elassandra ● Cassandra + Elasticsearch a. Mutations to Cassandra -> Mutations to Elasticsearch

7. DIY

8. Event -> CQRS -> Cassandra + Index ● Write ○ Event / Command goes into an Event Source repository (Kafka, SQL Table, etc. ) ○ Command Processor processes it into CQL / Elasticsearch or SOLR or Amazon ... Algolia ● Request ○ Event / Command goes into an Event Source repository (Kafka, SQL Table, etc. ) ○ Command Processor goes to index / ﬁnds the data, goes to Cassandra , gets the data, returns. ○ Query the index --

9. Cassandra -> Batch -> Index ● Writes to Cassandra ● Every now and then - Index to ??? ● Cassandra + Spark

10. Cassandra Triggers -> Index

11. Serverless Function -> Cassandra + Index

12. Apache Niﬁ (Lucene) -> Niﬁ Processor -> Cassandra + Index

13. Pros ● extremely rich search capabilities ● Geospatial ● synonym ● fuzzy logic search ● Typos ● Stemming ● packaged elastic/solr/lucene -> shorter latency ● separate index -> better separation of concerns and speed

14. Cons ● pure lucene -> reinventing the wheel of what Solr/ElasticSearch ● external elastic/solr/ ?? -> longer latency between ﬁnding the data / getting the data ● packaged elastic/solr/lucene -> don't expect it to solve all your problems ● consistency issues (if DIY) ● lucene is memory heavy ● lucene is disk heavy

15. Strategy: Scalable Fast Data Architecture: Cassandra, Spark, Kafka Engineering: Node, Python, JVM,CLR Operations: Cloud, Container Rescue: Downtime!! I need help. www.anant.us | solutions@anant.us | (855) 262-6826 3 Washington Circle, NW | Suite 301 | Washington, DC 20037

16. Resources ● tjake/Solandra: Solandra = Solr + Cassandra

Cassandra Lunch #23: Lucene Based Indexes on Cassandra

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Cassandra Lunch #23: Lucene Based Indexes on Cassandra

Ähnlich wie Cassandra Lunch #23: Lucene Based Indexes on Cassandra (20)

Mehr von Anant Corporation

Mehr von Anant Corporation (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Cassandra Lunch #23: Lucene Based Indexes on Cassandra