1. Automated Image and
Point Cloud Interpretation
Techniques and
Applications
Translating Human Visual Interpretations into
Algorithms in Spectral, Morphological, and Contextual
Feature Extraction
Mike Bularz
Prof. Robert J. Hasenstab
GEOG – Independent Study

2. CONTEXT – POINT CLOUDS AND BIG DATA TRENDS
The role of large datasets in Information Technology
The last few years in the world of information technology have seen an explosion in the amount
of data being collected about the world around us, with limited realization of the full potenti One
manifestation of demand for being able to process large datasets lies in the world of remote sensing,
image and point cloud interpretation, and computer vision. This is particularly true in many industries or
sectors interested in monitoring and modeling precise aspects of the natural environment that require
very particular processes to make sense of complex sets of sensed information. . The focus of this
document is to examine point cloud processing, particularly of LiDAR (Light Detecting and Ranging) data
and the fusion of these measurements with imagery. The goal here is to examine a few processing
algorithm types in the abstract, and discuss potential applications of these processes for feature
extraction in a geospatially enabled environment.
Point Clouds and LiDAR processing Demand
Government, Intelligence, Natural Resources (extraction or management), as well as marketing
and consumer electronics companies are seeking to exploit large data sets such as point clouds, and the
computer algorithms to process them to make technology more interactive or information more
exploitable. Government agencies increasingly are seeking out finer scale data about the built
envirionment and ways to process this data into actionable information products. For example, LiDAR is
being employed in floodplain mapping, building footprint delineation, bridge monitoring, as well as
more abstract interpretations such as determining rooftop areas suitable for solar panel installation,
modeling of noise in urban environments, and security applications such as crowd evacuations or
line-of-sight calculations. Google’s Autonomous Vehicles employ point cloud processing of live laser
range-finding with LiDAR technology,4 and has begun road testing vehicles due to passage of enabling
legislature in California.5 These vehicles present one example of the private sector pushing for rapid and
automated processing of large point clouds. The robotics community has been working out a standard of
point cloud processing to enable piloting of robots in environments. Another potential application of

3. point cloud processing in the private sector includes drones, which are expected to soon populate our
airspace under new FAA (Federal Aviation Administration) regulation and guidelines.6
HUMAN INTERPRETATION TO PROCESSING AND SEGMENTATION
LiDAR Processing – Popular Approaches
The explosion of LiDAR data, particularly in the geospatial community in the last 5 to 7 years has
prompted a search to streamline feature extraction while maintaining accuracy. A few common
methodologies to feature extraction from LiDAR have culminated from these efforts, and are the
underlying processes in popular tools and software extensions claiming to automate or semi-automate
the feature extraction process.
It is worth noting, LiDAR
data in the geospatial industry and
related professions is currently
categorized, in terms of processing
approach and application based on
its collection method: Airborne and
Terrestrial.7 Airborne LiDAR can be
DSM (Digital Surface Model) nDSM (Normalized Digital
points sensed from overhead in a Surface Model)
fixed-wing aircraft flyover, which yields larger areas of data collection, or from a closer range from
helicopter flyovers, which yields much higher resolutions of points. Terrestrial LiDAR can be deployed
from a fixed position on a tripod system, or from having the sensors affixed to vehicles, boats, and
remote-control sensing robots. Each data collection method is dependent on the necessary precision of
the application, and has a different common processing approach.
Height-based segmentation
The most common approach, and most easily implementable, is a height-based segmentation of
features. This is most commonly applied for data collected from fixed-wing and helicopter aerial
flyovers, but can be applied to terrestrial point clouds as well. Height based segmentation relies on

4. vendor pre-classified point clouds into bare-earth, and the structures above the earth at minimum. The
classification is based on interpolated surface models from the points.The bare earth surface model,
which is the Earth’s surface elevation of the ground without any of the structures or vegetation, is
referred to commonly as a DEM or DTM (Digital Elevation Model / Digital Terrain Model). All points
collected are commonly referred to as DSM (Digital Surface Model) and incorporate the bare earth,
vegetation, structures, and other features captured.
The processing approach itself is simple : subtract the bare earth surface (DEM) from the DSM to
obtain a normalized DSM (nDSM). This produces a “heat map” of structures above the ground, with the
height values representing relative distance from ground. It is useful in classifying trees, buildings, power
lines, and other infrastructure, but does not segment the different types alone. The process is essentially
fully automated.
Shape – Fitting
Fitting shapes to a point cloud involve one of either: Determining how “rectangular” TINs (Triangular
Irregular Networks, or a triangulated surface from points) is, and segmenting out manmade structures
from vegetation, or in line fitting to determine roads and edges of manmade structures. A third, but
related variation of this is a semi-automated process of drawing lines by hand into point clouds, using
snapping or fitting user interfaces.89
Both processes rely on Morphological interpretation of the shapes of interpolated surfaces, which is
explained further in the context of many processing approaches in the following section.
Spectral Fusion / Intensity and RGB Values
More ambitious classifications utilized the laser’s collected “intensity” of the reflected laser beam (if the
vendor has included it in the attributes), RGB values collected by the vendor, or burning in bands from
imagery into the points or vice versa, burning in intensity into an image band. These are spectral
classifications, as they rely on spectral characterisitics of a remotely sensed target, either by image
collection (passive remote sensing) or point intensity (active sensing). Common application for this is
seen in biology and forestry, where it is useful to use spectral information to discern various tree and
shrub species.

5. Point Cloud and Image Processing in a “Computer Vision” Environment
Recently, there have been developments in much more advanced and effective ways to process point
cloud information and imagery. Some of the methods stem from LiDAR processing software, while
others are being transplanted from the general image processing community. The goal of these software
tools and packages is to provide a means by which to translate “human vision”, or how we segment
parts of an image of point cloud that we perceive, into computer algorithms to replicate the
segmentation power of the human mind. There are several general categories of how we perceive our
environment through visual cues.
Image-based visual clues
The first types of visual clues are from actual value statements about objects: color / tone,
morphology (shape, height, size), texture (roughness / smoothness). The second type are essentially
auxillary clues: shadows, context (ex. These brown-colored objects are house roofs because they are
near a road. Ex. Green blob is an island because it is surrounded by water, whereas the green around
the lake is regular land), and pattern (there is a cluster of houses here, and roads, therefore this is a
residential area, and the largest structure is probably a school).
Vector and Second-order visual clues
The second type of visual interpretations are based off of defined shapes. Defined shapes are
objects we have recognized as true, such as vectors delineating what may be houses, or our mental
outline of the various perceived features. Theoretically, vector-based visual clues are typically
second-order visual clues in interpretation, as we are defining an object based on first order clues of
morphology, color, pattern, texture, or context and then analyzing the vector’s or defined-object’ s
properties. Interpretations of vectors can range from analyzing (mentally or in software) the number of
angles, sides, general shape, area, jaggedness and roughness of edges vs. smoothness, orientation,
symmetry, and contextual analyses based on other vectors.
ALGORITHMS, MANIFESTATION AND AVAILABILITY IN PROCESSING
AND SOFTWARE PACKAGES

6. Image Based Algorithms
Trying to translate visual cues from image based-perceptions relies on manipulating data in a format
representative of its color, texture, brightness, etc. This can be done in vectors representing a the
average values of these at an area, but the primary method for interpreting these characteristics is
raster, or gridded image based.
Spectral
Many of the concepts in image-based interpretation stem from classification methods used on
remotely-sensed satellite images. They vary from unsupervised classifications (ex. Specifying a spectral
parameter, that Near-infrared and overall brightness will be lower in water, therefore classify all pixels
as water that meet this requirement) to supervised, where samples of features such as grass, trees,
roads, and water bodies are traced as spectral samples throughout an image, and a statistical grouping
Slope (Perspective) Slope (Aerial)
based off of the modeled samples is performed.
Slope
There are some particularly unique image-based processing techniques that apply to point clouds,
though, and these are all based off of interpolated elevation models’ characteristics. One example of
this is by grouping pixels by slope, by calculating the slope of each pixel in an interpolated point cloud
elevation grid, and selecting the highest slopes. This is a translation of how the human visual
interpretation segments buildings and structures out of an image – by determining where the slope is
the highest we define the rough location of walls, and tall trees.

7. Curvature
A further classification method is based off of texture – often referred to as “curvature” of the elevation
model. By calculating the change in values in the nDSM pixels, to assess how “curvy” or “bumpy” a
surface is we can classify out certain features much easier: Trees and shrubbery have much higher
curvatures that mandmade structures such as houses.
Aspect
Aspect, or a calculation of the average orientation of the pixel, is similar to slope, but can be used to
segment features based on general orientation. For instance, house walls will be typically oriented
towards roads, and combining this contextual information can help place seeds (house walls facing
roads) by which to grow into features (houses).
Vector based Algorithms
By deriving vectors representing either
features, or the rough area of features using
image segmentation algorithms, we can further
attempt to locate certain features by definite
characteristics such as feature size (area), Curvature (Perspective)
shape (rectangular, smooth, bumpy, long, orientation, perimeter length), and context (distance to other
features, density, etc.).
Size, Perimeter
When looking at images, we segment objects by size as well: homes will be larger than trees, shopping
centers larger than homes, and forest stands will be larger than individual patches of foliage and tree
plantings. Deriving vectors from images, and segmenting them by size (shape area) allows to further
discern between these features. Perimeter length of the vectors plays a role too, a although a house and
a large tree may be about the same area in an aerial-derived vector, the tree will be more jagged and
curvy, and have a more complex shape and larger perimeter. Vector parameters can be derived from
two-dimensional (X, Y) or three-dimensional (X,Y,Z) calculations, for example, a 3-Dimensional sizing is

8. volume, which takes into account the height of derived features.
Shape – Characteristics
To further continue on this path of dissecting the basic geometric characteristics of derived vectors, it is
also plausible to calculate the angles, or sum of angles in the vector to classify the shape. The number of
sides of the vector may be a clue to classification as well, based on simple geometric definitions. A tree
vector will be jagged, have many sides, and many acute angles. The average of all angles in a tree will be
smaller than a less jagged shape. Manmade structures will typically be less complex, with long sides on
buildings, and limited acute angles. Using calculations to highlight continuous lines such as building
edges can help to refine the vector during classification, or to ouline features such as roads or power
lines.
Shape, Type
A further elaboration on these concepts lies in shape-fitting algorithms, that try to fit shapes to point
clouds, iteratively. Shape-fitting of rectangles is a technique being employed by researchers attempting
to classify point clouds and images into man-made structures such as buildings.10 Shape fitting can be
done in two-dimensional (X, Y) or three-dimensional (X,Y,Z) space.
Curvature (Perspective)

9. Contextual and Growing Algorithms
Contextual Segmentation
A combination of characteristics derived from vectors and images can be used to translate common
human-interpreted patterns: “Homes are next to roads”, “Commercial buildings are at major
intersections”, “Islands are surrounded by water”, “Urban Features cluster around major highways and
roads”, “This tree species grows in dense patches, while this one grows in sparse distributions”. “These
types of flora grow within a distance of these plotted bird sightings (bringing in more data)”. Contextual
analysis can be applied to search for non-static environmental objects, based off of basic buffer and
spatial analysis: “Military encampments will be within this distance from their origin, in this photo taken
X days ago” but these calculations stray into the field of time-sensitive data and predictive spatial
analysis, rather than mapping of the physical environment.
Growing Algorithms
Using common image-processing algorithms termed “Region Growing” algorithms, which “grow” or
expand into pixels around a designated “seed” starting point based on characteristics of these points
helps to determine boundaries of features as well. This process is less human-vision oriented as it is to
be definitive for computers. Determining locations of centers of homes in a residential area, and region
growing to the edge of the roofline using high-resoultion imagery and LiDAR height elevations can
delineate homes very well. This information can be contextual as well – region growing around a known
fire-starting point into a forest patch can delineate burn paths.
WORKFLOWS AND APPLICATIONS CASES – DEPENDENT ON OUTPUT
NEED
The workflow or sequence of algorithms (and likely software packages) you use will vary based on the
type, and precision of classification you are seeking. The simplest scenario: You are attempting to just
classify ground points in your cloud to create a surface model for flood and stormwater mapping
purposes, all you will need is to calculate height extremities to single out the features, although it is
likely the data vendor has already done this for you. More often than not, this is not the case, and has

10. led you to exploring the world of point and image processing.
Examples:
Workflow for classifying buildings and vegetation in a suburban residential area
Classifying buildings is a relatively challenging, but feasible process which depends on the degree of
accuracy necessary. If your goal is to have an estimate of the number of buildings, simple locations of
buildings are easy to determine. If your project is trying to model buildings, and requires definitions for
footprints, and roof overhangs, among other building components like facades, then you will be
spending a lot more time trying to fit shapes and other complex processes that haven’t been perfected
for this purpose yet.
nDSM Calculation
The focus of this approach is to classifying buildings in a non-dense, suburban area where visibility of the
building outlines is clear in the data. Basic techniques, such as obtaining an nDSM by subtracting the
DEM from the DSM are used to outline all structures. The next steps deal with shape-based
classifications of first, raster surfaces and derivatives representing the interpolated nDSM, and then with
vector-based fine-tuning of the classification.
Curvatures and Slope
First, vegetation is highlighted using the Curvature approach. It is solidified by calculating slopes, and
highlighting areas that both, have a high slope and high curvature. Second, the vegetation curvature
derivatives can be buffered or region grown, to produce a surface to clip out of the nDSM and leave
behind the other structures, which will be mostly buildings but may be other structures like water
towers or power lines.
Vector Calculations: Shape Area and Perimeter Length
Next, the vector-based fine-tuning of our features continues. The remaining building shapes are
converted into vectors, and a range representing the high and low value for shape area, or the square
footage of the house footprint is specified to remove smaller vectors (noise) and larger vectors
representing other features). A perimeter threshold is applied as well to remove a large forest patch to

11. the north.
Potential steps to fine tune the building shapes:
Simplification of Tree and Building Vectors
At this point, rough vectors are generated for our
buildings, and trees. We can either simplify the shapes,
applying a smoothing to the vectors for trees, and a
orthogonal / rectangular simplification for the buildings.
This may not be appropriate for all building shapes, if
they are not rectangular.
Region Growing from Seeds
An alternate approach at this step can be to calculate
Segmentation using nDSM, Curvature
polygon centroid for each vector, and use these points as
seeds for region growing across a color, color-infrared, or
fusion raster (raster with LiDAR intensity or heights burnt
in as a band). This can be done to get pixel shapes for
trees and houses which can be converted to vectors.
Region growing can be useful for non-linear and circular
building shapes.
Segmentations in dense urban areas
Dense Urban areas present a unique challenge because
they are compacted, layered in mixed-use developments,
Vector Segmentation (Forest Patch
and often of unique shapes dreamed up by architects. above removed by Permieter and Area)
Extremes in High / low building heights can have affects
on calculations and interpolations as well. Using region
growing, or spectral fusion of multi-band rasters with

12. interpolated point cloud metrics as bands can be
helpful.
Delineating roads with line fitting ERDAS - Objectve
Road Classification Process
Road delineation is only semi-complex, and can be
done without using point clouds on just imagery. A
point cloud could be used to interpolate heights in
the end, or can be used to create an “intensity”
band from the LiDAR metrics with which to aid
spectral segmentation of the roads.
First, an image (potentially including intensity as
one of the bands) is segmented using a classification
method from remote sensing techniques. An
appropriate segmentation can be a multi-resolution
segmentation which computes statistics at different
scales between neighboring cells, or other
supervised classifications.
Roads are identified by defining spectral thresholds
which determine a road in Red, Green, Blue,
Infrared, and Intensity space. The pixels are
coverted to a raster, and a centerline is calculated.
Classifying a vegetation and natural features with
spectral information
Vegetation can be much more precisely classified by
incorporating LiDAR metrics with spectral
information. Instead of merging derivative elevation Planar Fitting
and intensity rasters into a BGR, IR image, the
image is burned onto the points if it is of

13. Context Based Island Detection (eCognition)
appropriate resolution. Precise definition of various species of trees can be defined and put in as
thresholds by which to classify images.
Classifying lakes and islands with contextual clues
Contextual information is often the most neglected in common remote-sensing and point cloud
segmentation techniques, but it is arguably the closest to human vision and perception. When we look
at an image, we examine individual objects first, but also examine the rest of the image to understand
context. These interpretations are based on our conceived notions of typical distributions in
environments. For instance, a green blotch in a lower resolution aerial of a residential area (ex. 5m)
would not fool us to be a large tree considering factors in how it is situated in the context of other
houses, and house orientation which we assume by the context of where the house is next to a road,
and where the driveway is. Advanced contextual clues underlie some of the principles of
photogrammetry, or the visual interpretation of images, and have potential to be translated into rules in
software for image processing.
CROSS-POLLINATION AND POTENTIAL W/ OTHER INDUSTRIES AND
DATA
Synthesis / expansion of remote sensing
Advances in image processing spurred by collection of high-resolution imagery and point cloud
information are furthering the development of technologies in geospatial applications and remote
sensing. In the past few years, major software vendors in the GIS (Geographic Information Systems)
community including Integraph, ESRI, and Overwatch Industries have released point cloud processing
applications or amended the functionality of their software to support the .las (American Society of

14. Remote Sensing standard file format for LiDAR data). Major industry publications, including Imaging
Notes (Remote Sensing) and ESRI ArcGIS Newsletter have published cover stories on the topic. There is a
rapidly growing demand for LiDAR and point cloud information in the geo-sphere, and it is bringing
demand for techniques and processes to segment point clouds into useful datasets and derivatives.
Cross-pollination with robotics
The world of robotics and computer vision has enjoyed a cult following of hobbyists and enthusiasts,
along with intense research funding from DARPA for autonomous navigation of robots, and autonomous
navigating vehicles which utilize point cloud and environmental sensors to guide themselves through the
environment and obstacles.11 Google is also pushing the initiative with their experimentation with
autonomous navigating vehicles, as mentioned earlier, and drone technology is being adopted, which
could benefit greatly from being able to collect and process high-resolution imagery and range-finding
data from sensors.
Improving human-interface devices and experiences
From the consumer marketing perspective – application of sensing technologies is going through major
revolutions as well. Cell phones, such as the latest Google phone, incorporate face-scanning
technologies as a password system.12 Gaming and Entertainment systems incorporate gesture and
body-based interaction, such as the Microsoft X-Box Kinect, and the Playstation Eye. Decoding human
posture and gait (way of walking) is part of criminal detection systems as well, finding criminals by their
walking pattern and posture.13
ROLE OF POINT CLOUD PROCESSES IN BIG DATA
Uses of Precise Data
Derivatives from point cloud data and large datasets have huge potential applications and public
management, environmental management, resource extraction, intelligence and defense technologies,
consumer marketing, and social sciences. Derivatives such as building footprints, heights, and models
have many applications as mentioned previously and in some of the case processing examples.
Having large datasets available is one asset, having precise deliverables and information about these

15. datasets is another. Being able to process these large datasets into actionable information and
informative research will harness the amount of information currently being collected.
Works Cited
1
Hill, David. "The Rise of Data-Drive Intelligence." Network Computing. N.p., 05 Dec. 2012. Web. 14 Dec. 2012.
2
"The New Media Reader [Hardcover]." Amazon.com: The New Media Reader (9780262232272): Noah
Wardrip-Fruin, Nick Montfort: Books. N.p., n.d. Web. 14 Dec. 2012.
3
"2012 Year in Review: Big Data." Government Technology News. N.p., n.d. Web. 14 Dec. 2012.
4
"Sebastian Thrun: Google's Driverless Car." TED: Ideas worth Spreading. N.p., n.d. Web. 14 Dec. 2012.
5
"Autonomous Vehicles Now Legal in California." Wired.com. Conde Nast Digital, 23 Sept. 0012. Web. 14 Dec.
2012.
6
"Unmanned Aircraft Systems (UAS)." Unmanned Aircraft Systems (UAS). Federal Aviation Administration, n.d.
Web. 14 Dec. 2012.
7
"Types of LiDAR." ArcGIS Help 10.1. N.p., n.d. Web. 14 Dec. 2012.
8
Schwallbe, Ellen, Haans-Gerd Maas, and Frank Seidel. "3D Building Model Generation from Airborne Laser
Scanner Data using 2D GIS data and Orthogonal Point cloud projections." ISPRS. Conference Proceedings.
9
Morgan, Michael, and Ayman Habib. "Interpolation of LiDAR Data and Automated Building Extraction." Ohio State
University - Department of Civil and Material Engineering (n.d.): n. pag. Web.
10
Schwallbe, Ellen, Haans-Gerd Maas, and Frank Seidel. "3D Building Model Generation from Airborne Laser
Scanner Data using 2D GIS data and Orthogonal Point cloud projections." ISPRS. Conference Proceedings.
11
"DARPA Urban Challenge 2008." Welcome. Defense Advanced Research Projects Agency, n.d. Web. 14 Dec. 2012.
12
Wills, Danyll. "New Google Smart Phone Recognizes Your Face." MIT Technology Review. Massachusetts Institute
of Technology, 19 Oct. 2011. Web. 14 Dec. 2012.
13
Greenemeier, Larry, and Scientific American. "Something in the Way You Move: Cameras May Soon Recognize
Criminals by Their Gait." PBS. PBS, 29 Sept. 2011. Web. 14 Dec. 2012.