Dynamic Log Analysis is a security product from HAWK Network Defense that uses Naive-Bayesian algorithms and Boolean rule sets to analyze log data and assign threat scores. This allows it to identify true threats by filtering out non-threatening events. It aims to transform average security analysts into a team of "super-detectives" that can quickly discern and prevent real security risks.
Sophisticated Security - Naïve Bayesian Algorithms by Tim Shelton
1. Achieving Sophisticated Security
Using Naïve-Bayesian Algorithms
By Tim Shelton, VP of Research and Development, HAWK Network Defense
A New Era: New approach
In an era in which dollars count more than ever, a true solution will enable an organization to
more efficiently prevent a security breach and respond to each appropriately. Security
breaches of all types will continue to affect an organization’s bottom line. It is no longer
sufficient to merely respond to breaches. What is needed is a solution that enables an
organization to effectively and efficiently anticipate threats.
Recently, Network World stated it best:
“ ‘Correlation’ has long been the buzzword used around event reduction, and all of the
products we tested contained a correlation engine of some sort. The engines vary in
complexity, but they all allow for basic comparisons: if the engine sees A and also sees B or C,
then it will go do X. Otherwise, file the event away in storage and move onto the next. We'd
love to see someone attack the event reduction challenge with something creative like
Bayesian filtering, but for now correlation-based event reduction appears to be the de facto
standard.”
Quite simply, the current marketplace tools rely on basic Boolean rule sets. Although
somewhat effective, it usefulness is only as good as the person analyzing, assessing and
monitoring the events to identify potential threats. Sophisticated Bayesian filtering will enable
the analyst to identify threats and precursors to threats.
Famed scientist Thomas Kuhn stated that individuals are unlikely to relinquish an unworkable
paradigm, despite many indications that the paradigm is not functioning properly, until a
better paradigm can be presented. By utilizing a Naïve-Bayesian Histogram algorithm, Dynamic
Log Analysis™ is the next paradigm shift.
Dynamic Log Analysis™. By eliminating events that are not of threat, one can identify real
threats. Much like a ‘super-detective’ who is constantly monitoring, learning and adapting to
threats, dynamic log analysis is constantly performing predictive activities to eliminate false
threats. As a result, the average analyst is transformed into a team of veteran ‘super-detectives’
with the ability to immediately decipher a real threat from a minor daily occurrence. This
proactive method mitigates the probability that a network will be fully infiltrated.
Dynamic Log Analysis™ enables the average analyst to utilize a team of resources that can
differentiate events that are not of threat, so that real threats can be identified and prevented.
Dynamic Log Analysis™ refers to an event driven solution that iteratively assesses the
probability that certain types of events will produce a threat. Using a Naïve-Bayesian Histogram
algorithm to assign ‘scores’ as well as utilizing Boolean rule sets, the system learns and places
importance on certain types of correlated events. The system then assigns a ‘score’ to the
About HAWK Network Defense. HAWK Network Defense, Inc., is a privately funded security information event management
company founded in 2006. HAWK’s patent-pending product, Dynamic Log Analysis™, uses Naïve-Bayesian Histogram
Analysis technology that acts as a team of experienced security analysts to proactively mitigate risk and transforms the
tedious and time consuming task of event logging into a dynamic, powerful experience. HAWK Network Defense is
headquartered in the Dallas, Texas and its product, Dynamic Log Analysis™, is exclusively distributed through Clear
Technologies of Coppell, Texas. For more information, visit www.hawknetworkdefense.com or
www.cleartechnologies.net/DynamicLogAnalysis.
-1-
2. Achieving Sophisticated Security
Using Naïve-Bayesian Algorithms
By Tim Shelton, VP of Research and Development, HAWK Network Defense
threat. Dynamic Log Analysis™’s scoring technology determines the priority of an ‘event’ for
alerting and responding and its Multi-Decision Tree Matching Algorithm increases speed of
matching of events to rules developed by the administrator. By combining these two
processes, the time to identify, respond and remediate an event is greatly reduced.
Scoring. Just as a team of ‘super-detectives’ uses their shared experiences to identify and place
emphasis on significant threats, the Bayesian Histogram algorithm and Boolean Ruleset assigns
a score to define the magnitude of a threat. The ‘score’ is then placed in the database and the
administrator is alerted on the most perilous threats. The unique total score is determined by
utilizing the naïve Bayesian learning algorithm, the Boolean rule-set, as well as information
acquired during the normalization and matching process. All of the gathered information is
taken into account before the total score is determined.
In its simplest form, the solution performs the following:
Once the event, which is any user action, log entry, security notification, and performance
statistic, has been selected for processing, its contents are inserted into the database. After
database insertion, the event goes through the unique multifaceted scoring process that first
includes a determination of the naïve Bayesian score by analyzing the standard deviation. The
system is then able to match against those target events that have not been previously
identified. In addition, this Naïve-Bayesian algorithm is specifically designed to match against
known or trained information. Together, the engine establishes an operating baseline, and to
looks for deviations against this standard norm.
Next, Bayesian score is included along with the existing event properties to be processed by
the Boolean rule-sets, which is list of rules associated with a positive or negative score. Once a
Boolean rule-set is matched against a provided event, the associated score is added to the
existing score, which in most cases is zero. Once all the rules have been compared against the
event, a total score is determined, allowing future actions to be taken based upon the pre-
configured score threshold.
At this stage, the unique total score only applies to a single event. By assigning each event a
unique score, an analyst is able receive alerts on isolated, specific events that exceed a specified
score threshold. In addition, isolating and assigning a unique score to each event enables the
analyst to conduct a trend analysis and rapidly adjust to changes in overall activity.
Dynamic Log Analysis™’s Multi-Decision Tree Matching Algorithm. In the same way a team of
‘super-detectives’ relies on their shared knowledge and experiences in order to quickly match
threats to specific, predetermined high-risk behavior, the decision and matching technology
then matches the provided event to its related ‘rule’ faster. This technology is designed in three
layers.
About HAWK Network Defense. HAWK Network Defense, Inc., is a privately funded security information event management
company founded in 2006. HAWK’s patent-pending product, Dynamic Log Analysis™, uses Naïve-Bayesian Histogram
Analysis technology that acts as a team of experienced security analysts to proactively mitigate risk and transforms the
tedious and time consuming task of event logging into a dynamic, powerful experience. HAWK Network Defense is
headquartered in the Dallas, Texas and its product, Dynamic Log Analysis™, is exclusively distributed through Clear
Technologies of Coppell, Texas. For more information, visit www.hawknetworkdefense.com or
www.cleartechnologies.net/DynamicLogAnalysis.
-2-
3. Achieving Sophisticated Security
Using Naïve-Bayesian Algorithms
By Tim Shelton, VP of Research and Development, HAWK Network Defense
When an event is received by the dynamic log analysis engine, it converts the received
information into a normalized event, matches it against its pre-defined rule set and is then
separated into two types; compiled modules, and a textual rule-set. The textual rule-sets are
separated into three basic classifications that provide the means for matching against our rule-
set: triggers, rule-groups, and rules. A trigger is a regular expression that must match a threat
in order for the rules within the module to continue processing. If it does, the event proceeds
to one of the rules groups and within the rule group, a rule is applied. A rule contains all the
given information Dynamic Log Analysis™ requires for improved matching, correlation, and
scoring. Each rule contains the alert name, category, knowledgebase id, host and network
packet information, as well as audit procedure information for compliance monitoring and
scoring. The final rule, upon successful match, allows the administrator to assign the specific
information to the event’s normalized hash table. The final rule allows for multiple matching
rules as well as using the ‘not’ indicator. Once these activities have been completed, the event
is passed into the processing queue for archiving, scoring and additional correlation.
Dynamic Log Analysis™’s Information Event Console. Lastly, in the same manner that a team of
‘super detectives’ combines all of their respective experiences and knowledge into one shared,
cohesive view to visualize the extent of the threat, the Dynamic Log Analysis™'s Information
Event Console presents an overall view of the highest and lowest priority alerts, all arranged by
severity of correlation. Further, it acts as the management and data retrieval interface with the
relational database, provides a historical retrieval of logged information, and, over secure
encrypted sessions, provides role based access controls.
In conclusion, HAWK Network Defense has developed this, patent-pending, technology that
transforms the tedious and time consuming tasks of event logging into a dynamic, powerful
experience that proactively mitigates risk. Not only will the analyst be able to rely on experience
of the tool to prevent threats, but also be able to utilize his own experience by writing, through
regular expression, rules that will place a ‘score’ on specific inter-organizational nuances which
are not a threat.
About HAWK Network Defense. HAWK Network Defense, Inc., is a privately funded security information event management
company founded in 2006. HAWK’s patent-pending product, Dynamic Log Analysis™, uses Naïve-Bayesian Histogram
Analysis technology that acts as a team of experienced security analysts to proactively mitigate risk and transforms the
tedious and time consuming task of event logging into a dynamic, powerful experience. HAWK Network Defense is
headquartered in the Dallas, Texas and its product, Dynamic Log Analysis™, is exclusively distributed through Clear
Technologies of Coppell, Texas. For more information, visit www.hawknetworkdefense.com or
www.cleartechnologies.net/DynamicLogAnalysis.
-3-