In this presentation, we will detail two image processing applications which rely on a Cassandra centric architecture to achieve distributed, high accuracy analysis of a variety of image formats, types, and quality, and which require different kinds of metadata processing as well as feature extraction from the image themselves. We will outline the architecture choices made for the two use case studies, and how we found Cassandra to be the ideal choice for the persistence layer implementation technology. In conclusion we will discuss extensions to the two use cases discussed and some of the 'lessons learned' from the two implementation projects. About the Speaker Kerry Koitzsch Project Lead, Kildane Software Technologies, Inc Kerry Koitzsch is a software engineer and architect specializing in big data applications, NoSQL databases, and image processing. He currently works for Correlli Software Systems, a big data analytics company in Sunnyvale CA.

1. Cassandra-Based Image
Processing: Two Case
Studies
KERRY KOITZSCH
‘IMAGE AS BIG DATA SYSTEMS LLC’ (IABDS.COM)

2. Who Am I?
 Image processing enthusiast
 Big data technology enthusiast
 Robotics enthusiast
 NoSQL database enthusiast
 Microscopy enthusiast
 Software engineer
 Author

3. Overview of this Presentation
 This quick overview of some of our ongoing projects describes why Cassandra is a key
part of our ongoing research, development, and client support activities.
 The presentation highlights to areas of research which involve Cassandra technologies
in the “images as big data” arena: an automated microscope slide application
prototype, and a stereo ”smart sensor” mobile robot prototype.
 Both of the use cases describe rely heavily on Cassandra and related “helper libraries”
to provide data storage capabilities for the two software prototypes.
 Throughout the presentation we discuss how the flexibility, high performance, and
ability to “play well with” other components makes Cassandra an essential part of the
applications described here.

4. Some Context About Our Use Cases
 The automated microscopy case study is based upon our current POCs for a
distributed microscope slide analytics application. In it, we use standard optical
microscopes fitted with color digital cameras and an automatic slide-loading cartridge
system to process “decks” of glass or plastic microscope slides, usually of human cell
tissue of various kinds, the staining process depending on what the nature of the tissue
is, and, in the case of hematology slides (human blood) no staining at all. Different
resolutions of magnification, slide quality, and metadata must be processed.
 The stereo vision robotics platform case study is based upon our current POCs for a
stereo vision mobile robotics application. The application uses Hadoop, Spark, Flink,
Solr, and a number of additional machine learning libraries to analyze stereo image
pairs, build two- and three-dimensional models of the environment, and uses Bayesian
sensor fusion to enhance the 3D models with additional environmental information.

5. Our Objectives and How Cassandra Helps us Achieve Them
 In our use cases, we have to integrate mechanical control systems, complex ‘image as big data’
analytics, and a variety of data formats and metadata in a common representation and with
efficient implementation.
 Cassandra and associated ”helper libraries” such as Spring Data Cassandra, Solandra, and Katta
allow us to maintain a flexible technology stack while using off-the-shelf proven technologies
which “play well” together.
 Cassandra has well-defined parameters for optimization, scalability, and availability.
 Cassandra enables us to meet our objective of maintaining flexibility while maintaining high
performance capabilities with a changeable server topology.
 Cassandra based technologies are flexible enough to accommodate a wide range of different
application domains, including microscopy (medical imagery of all kinds) and mobile robotics
applications (stereo vision ‘image as big data’ technology)

6. Our little software secret! :D
 We are not always software engineers….
 Sometimes, we are software ARCHAEOLOGISTS!
 But what is software archaeology? And how does it assist us build modern high-
performance distributed software systems?

7. We are re-inventing software archaeology!

8. Software Archaeology is Based Upon:
 The realization that the “first wave of AI technologies” (up to the mid-1990s)
introduced a host of valid ideas, concepts, strategies, and approaches which were not
completely assimilated into succeeding waves of AI technology.
 The realization that “looking back at our software roots” may be a good thing.
 Identifying “best of breed” strategies of the past and re-thinking them, using up-to-
date software stacks. Some examples of these technologies include pre-Hadoop
distributed processing systems and algorithms, DeKleer’s implementations of “problem
solvers” [1], geometric toolkits such as VANTAGE and Geometer [2],[3].
 This re-thinking and re-implementing process has already been successfully done for
POCs, products, research extensions and particularly for software patents [4]…

9. Migrating ‘First Wave AI’ Concepts
 Quality image processing systems have been lost in the course of time (for
example General Electric’s Geometer [4], the Vantage modeling system [5], and
many others) … while the implementation may need updating many of the
concepts may be --- and have been --- reimplemented using modern
technologies (and in particular Cassandra-centric technologies)
 Migrating our inspirations to modern implementation
 *Inspiration* does not *equal* implementation: we may choose to re-implement in
Clojure rather than Lisp, for example, and we use Cassandra-centric technology
stacks for almost *all* our implementations…

10. Why Our Case Studies are Cassandra-Centric
 Cassandra is a flexible, full-featured noSQL database
 Integrates well with Hadoop, Spark, and Flink-based software components
 A lot of development focus has already been done on different technology stacks, for
example, the SMACK stack
 Integrates well with the battery of Spring Framework components we use
 Cassandra is scalable, available, and highly-tunable
 “Plays well” with Lucene/Solr based technologies (via Solandra and other “glue-ware”)
 It’s easy to integrate Cassandra into use cases and software prototypes, allowing a
“rapid prototyping/development” style
 Its also easy to use Cassandra in a plugin/module based architecture style
 Cassandra ’plays well’ with a variety of machine-learning libraries
 Cassandra is flexible enough for research, solid enough for products

11. POJOs to Cassandra via Spring Data
 Plain old java objects (POJOs) may be mapped to Cassandra using Spring Data
Cassandra
 We would also like to map ontology objects to Cassandra
 We begin with fundamental feature objects of images: regions, contours, lines,
junctions, and points, the kind of object instances generated from, for example,
connected component algorithms, edge extraction operators, and the like
 For our domain examples, the extracted image metadata and associated “fused”
data from other sensors [ for example microscope slide data sets might have other
sensor results associated with them, like spectrometry and chromatographic data ]

12. Images ARE Big Data
 Images have volume, variety, velocity, veracity
 While the image volume may be relatively low, the complexity of the individual
image objects may be extremely complex
 There may be associated signals (such as LIDAR and geolocation data)
 Images may be multispectral and multiresolution in nature
 Images may have significant associated metadata (as with medical imagery)
 Images may play only one part in a larger “big data enterprise”

13. Considerations About the SMACK Stack

14. Case Study 1: An Automated Microscopy
System
 We want to be able to analyze large numbers of microscope slides automatically.
 The image analysis component may be complex and time consuming: we require a
distributed solution
 Images may require cleaning, rescaling, metadata extraction, and image format
conversion
 Analysis results are stored in Cassandra database with associated search
mechanisms via Solandra and custom code.
 The physical apparatus is a standard medical optical microscope with automated
stage and lighting controls and an automated microscope slide cartridge system
holding 25 microscope slides per deck.

15. A View of the Physical Apparatus…

16. … And How We See the Problem...

17. Samples of Input Data from a typical data set

18. Cassandra-Centric Infrastructure
 Cassandra’s flexibility allows us to adapt data models appropriately for new
experiments and ideas.
 We are also able to maintain consistent ontology development by progressively
evolving our table definitions and POJOs.
 We can develop our data models in separate Cassandra instances for purposes of
experimentation.
 Metadata can be easily expressed: this is particularly important for experimental
medical data, as citations, references to other data sets, parameters, and
associated sensor data may be present, as well as timestamps and audit trail /
handler data if information comes from a doctors office, insurance company, or
hospital.

19. Architecture Diagram, Case Study I
Canova is now DataVec

20. Ellipse Feature Extraction Example

21. Technology Stack, Case Study I
 Mechanical controls are implemented in Java and C++: these control the microscope
stage controls, lighting, positioning, cartridge control, and others
 Timestamps are generated by the microscope controls and are associated with analysis
records
 Slide images may be analyzed one or more times using different data analysis pipelines
 Slide images may be retained in the file system and subjected to feature extraction
and analysis
 We avoid saving images as blobs in Cassandra at this time
 We can “ramp up” our complexity gradually as we evolve our software. For example,
the next slide shows contour extraction from a microscope slide: the contour data may
be stored directly into Cassandra.

22. Contour Extraction Example

23. Conclusions, Case Study I
We learned in this case study that Cassandra’s flexible data modeling and
straightforward seamless integration with other ‘image as big data’ components,
including machine learning libraries such as Mahout and deeplearning4j enabled us
to build high performance slide analysis prototypes which were able to store complex
microscopic slide information (derived from feature extraction) and metadata
(associated with the image itself, such as citations, other sensor data, authorship and
auditing/handler data, and the like)

24. Case Study II: A Stereo Vision Robotics
Platform
Using a standard dual-image synchronized stereo camera on a mobile robotics platform, we
are able to obtain timestamped, geolocated/microgrid located image pairs for analysis. Image
pairs may be generated periodically to produce a set of sequential image pairs.
These image pairs are subjected to feature extraction, model building, and other analytic
techniques to produce scene models, which are associated with scene ontologies and scene
model templates to incorporate within a navigation program.
The navigation program and its data, along with internal sensors within the robot itself, allows
the mobile robot to successfully navigate scenes and perform simple tasks.
The navigation program is also able to perform qualitative navigation based on landmarks.

25. Goals of the Stereo Vision Platform
 Use a standard stereo image pair generating dual camera for experimentation
with a mobile robot
 We want to investigate two software problems: qualitative navigation and
integrated sensor fusion with robotic control
 We want to use inspiration from Poggio’s MIT Vision Machine in particular as a
starting point for our software
 However, we wish to use an integrated data pipeline design with “best of breed”
third-party components, including Cassandra, Hadoop/Spark, and Solr.
 We model much of our stereo three-dimensional modeling on Hartley and
Zisserman [5] and Faugeras [6]

26. Hardware of the Robotics Vision System

27. Architecture Diagram: Simple Stereo Vision

28. Images from Our Stereo Camera

29. Image Analysis Example I

30. Image Analysis Example II

31. Feature Extraction
 We can use many of the standard feature extraction techniques in a distributed
computing environment, including corner-finding, edges, junctions, t-junctions,
and the like.
 Connected component, canny edge extraction, deep learning, genetic algorithms
are all appropriate feature analysis technologies.
 Importing and exporting feature extraction data to and from Cassandra is easy
and makes sustainable experimentation possible.
 We can migrate from ontology to POJO to schema based design with ease as our
software can handle each of these representations.

32. Technology Stack, Stereo Vision Platform

33. Data Flow in the Robotics System
 Data flow in the robotics system is primarily mediated by Apache Kafka
 It turns out much of the extraneous ‘image information’ may be ‘thrown away’ at
least temporarily --- for purposes of specific data flows
 Data flows may be mediated by Hadoop, Spark, or Flink based components
 Some machine learning components used in the data pipeline processing are now
technology agnostic : for example, Mahout supports Spark
 All of the image processing components have been found to work extremely well
and seamlessly with Cassandra

34. Conclusions, Case Study II
 Cassandra was a key component in our robotics software system
 We were able to run a Cassandra “image database”, “control database”, and
“environmental database” concurrently on three different servers, and coordinate
the data effectively for sensor analysis, fusion, and control purposes.
 We found we were able to do “rapid prototyping” of robotics software using
Cassandra as our primary data store technology

35. Some Observations about the Technologies
 In spite of the fact that the two applications we discussed here were so different, we evolved both
of the projects in parallel in order to leverage the common elements of the projects with a mind
towards development an ‘image as big data’ toolkit, with which it would be possible to develop a
wide range of domain applications, using the idea of ‘images as big data’ --- and the secondary
technologies of sensor fusion --- as unifying themes.
 Appropriate database technologies are key to the success of the POCs shown here, and Cassandra
proved to be the superior choice for a number of compelling reasons.
 While the idea of distributed image processing is by no means new --- a variety of
implementations on different hardware configurations including Symbolics LISP machines,
Connection Machine, and other parallel/multiprocessor hardware were implemented throughout
the 1990s
 These older distributed concepts have been replaced by Hadoop, Spark, Flink, and multicore/GPU
based applications at the low level, and, to a certain extent, by semantic web/ontology/reasoning
engines at the high level vision processing phase.

36. Conclusions and Future Work
 We intend to pursue our concept of innovation + software archaeology : best of
breed new implementation --- with inspiration from quality software systems of
past and present
 Applying it to new domain areas and POCs
 Particularly sensor fusion, ”smart sensors”, drone and mobile robotics applications
 We intend to pursue the research aspects of “image as big data”, and to
implement POCs, products, and patents accordingly

37. Thank You!
kkoitzsch@kildane.com

38. References and Citations
 [1] Forbus, Kenneth D., and de Kleer, Johan. Building Problem Solvers, Cambridge,
MA: MIT Press, 1993.
 [2] Balakumar, P. et al. VANTAGE: A Frame-Based Geometric Modeling System,
Programmer/Users Manual V1.0. Robotics Institute, Carnagie Mellon University,
1988.
 [3] Barry, Michele, Cyrluk, David, Kapur, Deepak, Mundy, Joseph, and Nguyen,
Van-Duc. A Multi-Level Geometric Reasoning System for Vision. In Geometric
Reasoning, Depak Kapur and Joseph Mundy, eds. Cambridge, MA: MIT Press,
1989.
 [4] Patent, OCR Enabled Management of Accounts Payable and/or Accounts
Receivable Auditing Data, Patent Number 20110213685
 [5] Hartley, Richard, and Zisserman, Andrew. Multiple View Geometry in Computer
Vision. Cambridge University Press, 2000.
 [6] Faugeras, Olivier. Three Dimensional Computer Vision: A Geometric Viewpoint.
Cambridge MA: MIT Press, 1993.