Dealing with Dark Data

Agenda

What is dark data?

Josef Elliott

Using HP ControlPoint to find, analyse & classify unstructured
enterprise data

Dominic Johnstone

Using AIO to analyse and manage database and application
content

Mark Gower

What do I do with the data I find?

Dominic Johnstone

Securing and managing data with HP Records Manager

Craig Adams

Hosting your data in the secure HP Cloud

Bharat Mistry

Putting it all together DEMO of ControlPoint and Records Manager

Dominic Johnstone
Arthur Leclerc-Chalvet

Q&A and Prize Draw for HP Tablet Device

Josef Elliott
1

Today’s Information Reality
Overload or opportunity?

EXPLOSION OF
DATA
• Data doubling every
12-18 months
• New, unstructured
data types to manage
• Storage growth is
outpacing IT budget
• Growth of
ungoverned
structured data

DATA
EVERYWHERE

REAL-TIME
ACCESS

INCREASED
REGULATIONS

• Mobile, virtual, cloud
joins physical and onpremise
• New Data Security
considerations
• 67% of users have
three or more
computing platforms

• Always-on
applications and users
• Users have high
up time expectations
• Business continuity
moves to forefront

• Escalating regulatory
and higher volumes
• Find the “needle in
haystack”
• Utilize corporate
information assets

Growing IT Burden, Business Inefficiency and Risk Exposure driving Information Governance

2

Enterprise Information Environment

“Missed Opportunity”

Social Media

“Increased Risk”

“Cost & Complexity”

Video

HRMS

Supply Chain
Management/
Inventory
Mgmt

Email
Texts

Messages

Procurement

CRM

ERP
Images

Word, Excel

Audio

Transactional
Data

Human Information

90%

Logs

Clickstream
Data

Structured Information

Organisations need to handle
100% of this information

10%

3

Why is it growing…..?

Current behaviour leads to over-retention
The myth of cheap storage, coupled with lack of clear policy creates confusion

We are running out of capacity

Let„s add more disks

Applications are slowing down

Upgrade infrastructure

Backup takes longer and longer

Change backup infrastructure

We need to retain information...

Keep tapes

…For a certain period of time

We keep everything forever

We need to be compliant

Implement archive, DMS, RM,...

We need to retrieve information...

Look into different sources

…Historical information

Recover tapes
6

Dark Data

“like our own physical universe, the digital
universe is rapidly expanding and
incredibly diverse, with vast regions that
are unexplored and some that are,
frankly, scary”.
(IDC, Digital Universe, 2013)

7

Defining Dark Data
Characteristics of dark data
What is it?
•
•
•
•
•
•

Human readable
Unstructured
Not indexed
Unmanaged
Inactive
Orphaned

Where is it?
•
•
•

File Servers
SharePoint
Email Servers

9

Opportunities and Risks of Dark Data
What lies hidden in your dark data?
Understanding your dark data can have significant benefits:
• Cost savings through the reduction of storage and management overhead by allowing the defensible
destruction of non-required data
• Prepares and organizes valuable legacy information to provide insight into current and future
business processes
• Allows you to recognize information types and structures and develop policies to properly govern
information in future

Data that remains dark has the potential risk of:
•
•
•
•

Containing sensitive information that is unprotected from data leakage and misuse
Information being used out of context
Decisions are based on outdated versions of information
Duplicate effort is spent in producing the same information again

10

Confusion, Cost, Risk

The Information Paradox

Information
Gartner Inc 2012
11

Document Production
Document Production Requirement

Applicable for Litigation,
Regulatory Submission or
Internal enquiry purposes

Major financial benefit accrues when “document production” becomes a
repeatable, defensible process
12

It’s actually all about Information Management

13

Six Steps to Redemption

1. Understand the logical information environment
2. Identify existing information repositories
3. Index to understand your existing data
4. Deep dive analysis and advanced content analytics to extract meaning
5. Archive, Protect, Dispose
6. Secure the benefits

14

Applying Information Governance to Dark Data
A stepwise approach to making sense of your dark data
1.
2.
3.
4.

Identify your dark data sources
Create a light metadata index
Reduce obvious ROT
Perform deeper analysis, incorporating advanced data analytics to add
meaning and context to the data (data becomes information)
5. Categorize the information
6. Define and apply auditable policy to the information
Analyse

Execute

Sample

Auditable
Policy

Approve

15

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Tag

Reducing Redundant, Obsolete and Trivial Data
Freeing up space and reducing volumes in preparation for in-depth analysis

Redundant:
•
•

•

16

Obsolete:

Identify duplicate
•
documents
Identify the master
(assess against obvious •
master sources)
Get rid of the copies
•

Identify documents that
haven‟t been accessed for a
long time
Compare against your
retention schedules
Get rid of the ones that are
obviously past their use by
date


Trivial:
•
•
•
•

Identify documents that have no
valuable content
System Files, Crash Dumps,
Thumbnails etc.
Assess on the basis of file types
Get rid of the obviously trivial
ones

Use Advanced Content Analytics to Extract
Meaning
Develop an understanding of the business context of your dark data
Clustering
•
•

Visualize common content patterns in your dark data
Identify groupings for policy development and application

Trained Categories
•
•
•

Leverage your investment in records management
Use known documents to train categories
Identify category matches in your dark data

Eduction
•
•

Identify sensitive information in documents
e.g. Credit Card Numbers, Social Security Numbers etc.
Secure documents appropriately

17


Policy Application and Execution
Dealing with the categorized information

Defensible Destruction
•
•

•

Seek approval from identified owners
Provide audit reports on all stages of your decision process (identification,
sampling, tagging etc.)
Maintain audit logs of deletion process

Migration
•
•
•

Migrate to records management, archives or secondary storage system
according to the business value of the information
Migrate metadata and tags if possible
Maintain audit logs of the migration process

In-Place Management
•
•
•
18

Use a policy engine to enforce future application of the policy you have selected
now
Quality of information and policy application depends on the capabilities of the
system under management
Maintain audit trails of any actions applied


Ongoing Information Governance
Keeping everything under control

Identify how business processes use and generate information
•
•

Adjust ongoing information governance policies according to business and regulatory
requirements
Cover both structured and unstructured information

Keep indexes up to date
•
•
•

Leverage for enterprise searching
Keeps data accessible
Allows ongoing categorisation

Automate as much as possible:
•
•
•
•

19

Implement performance and compliance archives
Apply ongoing policies based on indexed categories
Auto-capture records
Manage records in-place


Summary
Information Governance is an ongoing business

Leverage the investment in Legacy Data
Cleanup for ongoing Information
Governance – Don’t let new information
get dark!

20


Stages of Legacy Data Cleanup
Design structure

1. Identify and
Index

21

2. Analyse

3. Organize

4. Reduce


5.
Manage/Migrate

Identify and Index
Getting started
Software Deployment
• Legacy data cleanup volume licensing
• Packaged deployment tool now includes Exchange Connector

Repository registration
• Repositories are automatically registered based on IDOL databases
• File system and SharePoint repositories can be added and configured for indexing through
ControlPoint UI

Indexing
• Different levels of indexing are available
• Preconfigured eduction task to find personally identifiable information

22


Identify and Index

23


Analyse
Understanding what is there

Visualise the statistics compiled by the analytics as a summary report:
• Based on file level metadata and hashes:
– Redundant data: statistics on duplicates
– Trivial data: based on file types with no content value (e.g *.exe, system files, thumbnails etc)
– Obsolete data: based on date created, modified, accessed
• Based on advanced content analysis:
– Category matches
– Personally identifiable information (eduction)

Provide detail graphs and linked document grid for:
•
•
•
•

24

Cluster visualisation
Duplicates
Analytical data: By size, type, age, user, categories and custom fields
Applied Tags


Analyse

25


Analyse

26


Organise
Preparing for policy assignment
Supporting a structured approach consisting of:
•
•
•
•

Filtering – to combine main selection criteria with other facets
Sampling – to randomly select fixed size or percentage samples
Document inspection – to view properties, advanced properties or native content
Tagging – to group the documents into actionable categories

Assign policies to tagged categories
• Use the standard ControlPoint policy phases, including workflow policies that can route the data
through an approval process

27


Organise

28


Organise

29


Reduce
Cut down on the data volume
Provide defensible deletion
• Apply deletion and deduplication based on ControlPoint policy
• Standard ControlPoint policy review step is available
• Use ControlPoint workflow policy actions for extended review processes (requires APA)

30


Manage/Migrate
Taking the step from cleanup to ongoing information governance
Merge legacy data into current information governance policies
• Declare move, secure move, hold, manage in place through ControlPoint policies
• Migration of organised legacy data between repositories(e.g. File System to HP Records Manager
SharePoint, Exchange to ACA etc.)
• Declare legacy data as records (in place or migration)
• Ongoing application of ControlPoint policies to new data identified based on trained categories

31


Legacy Data Cleanup
Summary

Legacy Data
Inactive
Unknown
Orphaned

Dealing with the past

32

Continuous Information Governance


Legacy Data Cleanup
Summary

ControlPoint 4.1
Legacy Data
Inactive
Unknown
Orphaned

Dealing with the past

33

Continuous Information Governance


Case study: auto records classification
Defense Contract Management Agency (DCMA)

Products: HP TRIM, ControlPoint (5000 users)
Use case:
•
•

Working with suppliers to align purchasing of products and services
with policies and requirements.
Seeking to improve management of SharePoint and file shares; also
to improve legal hold programs

Why we won:
•
•
•
•

Made classification and records management transparent to users
File share and SharePoint content under strict policy management
ControlPoint and ALH for identifying content for legal hold
APA to improve program workflows

Case study: SharePoint governance
Product: ControlPoint
Use case:
• Uses ControlPoint to lower operating costs of
SharePoint by:
• Migrating content from SharePoint farms to
Autonomy cloud storage
• Using ControlPoint to identify and move content
from operations to Autonomy cloud

Benefit:
• Achieved significant cost reductions in SharePoint TCO

Case study: Legacy Data Cleanup
Product:
Control Point 4.0
Use case:
• Selected Control Point 4.0 for dark data identification
• Initial project is to identify, analyse and transform 100 terabytes
- full project to include 2 petabytes
• Project sold in conjunction with ES Application
Optimisation program
• Will identify unstructured content with high business
context and records value
• Goal is information footprint reduction AND identification
of unknown high-risk content

HP
Application Information
Optimizer
Mark Gower
31st October 2013


Application Information Optimizer
Addressing the problems of dark structured data

Exploding Data Growth
Lack of Policy
Management

Performance Issues

Governance and eDiscovery

38


Classifying Application Data

• Define a broad classification of the data in the app database

Data by Classification

• Eg. Trading Data & Audit information

• Identify structure within classifications
• Analyse by date quantity of data for each classification
• Propose data management policy

Other
1%
Oracle
System
2%

Trading Data
22%

Audit Data
75%

39


Managing Applications
With Application Information Optimizer
•
•
•

Relocate Data to a second datastore
Relocate Data to external archive / governance platform
Improve Performance & Reduce Risk

Reporting/rendering
tools

Business Application

Retired records

Inactive Data
Primary Datastore

40

HP RM8


Defensible Disposal

Enterprise Structured Data Management

HP IT‟s ongoing database archiving strategy
powered by HP AIO since November 2009
•Query time reduced by 89%
•Storage reduced by 48%
•Full backup window reduced by 37%

By July 2012 - 122 source databases, 836 db-db jobs, 57+billion rows
archived

41


What do I do with the data I find?
•
•
•

Unstructured Data
Repositories

Structured Data
Repositories

•
•
•

Archive
Protect
Dispose

Know what
information you
have

HP Records
Manager

Know where the
information is
located

Autonomy
Consolidated
Archive

Analyse
Classify
Take action

Know your
information is
secure

Additional
Active
Repositories

Know the
information is
managed
appropriately

Autonomy
ControlPoint

HP Application
Information
Optimizer


HP Autonomy
Records Management
Strategy
Craig Adams, EMEA Information Governance Manager
31st October 2013


HP Records Management - Strategy

Microsoft SharePoint

HP Autonomy Records Manager

44


HP Records Manager 8.0
HP Records Manager 8.0
• Next generation, scalable electronic document and records management
solution
• Designed to meet the needs of Government and regulated industry
• Combining the best of HP TRIM, ARM & Meridio
• Standards Compliant platform
• A cornerstone of HP‟s Information Governance framework

Transforming Records Management
for Information Governance

45


Transforming records management
HP Records Manager use cases supporting information governance

Interactive Document
Management

SharePoint
Governance

ControlPoint Auto-Declare
In-Place Management

Unstructured
Repositories

SAP
ArchiveLin
k

AIO Structured
Records

COM/.NET SDK
Services API

HP Records Manager

46


Physical
Records

Custom Solutions

HP’s Own Records
Management Journey


HP RM Programme - ERMS Purpose
ERMS will facilitate and enhance HP‟s compliance activities, because we
• Are required by law to manage records
•

Need to find the right records at the right time

•

Need to produce records swiftly and cost-effectively for tax and other government
audits, regulatory compliance, and legal proceedings

•

Need to delete records we do not need to keep to reduce storage and litigation
production costs

•

Need to meet our global needs across:
•
•

16,000 Records Management / Co-Ordinators

•

173 countries

•

Multiple languages & jurisdictions

•

48

330,000 employees

Stakeholders in Tax, Audit, Legal, IT


HP RM Programme - ERMS Scope

49


HP RM Programme - ERMS Schedule
FY08 Q4 –
FY09 Q1

50

FY09-Q2

Phase I
FY09 Q3 FY10 Q1

Phase II
FY10 Q3 FY13

Phase III
FY11 Q4 FY14


Beyond Phase III
FY14+

HP RM Programme - FY13 Registered Records

50,000,000
43,515,280

45,000,000
40,000,000 38,203,252
33,817,366

35,000,000
30,000,000

24,946,475 24,370,349

23,373,470

25,000,000

26,573,487

20,000,000
15,000,000
10,000,000
5,000,000
0
Nov

Dec

Jan

Feb

Mar

Apr

May

Jun

Details from registered records report.

51


Jul

Aug

Sep

Oct

Hosting your data in the
secure HP Cloud
Bharat Mistry
CTO Office – Strategy Group,
HP Enterprise Security Services
October 31, 2013

© Copyright 2012 Hewlett-Packard Development Company, L.P.
The information contained herein is subject to change without notice.

HP Converged Cloud
Hybrid delivery based on common architecture across traditional & all cloud models
Choice
• Open…standards-based across all delivery models
• Heterogeneous…hypervisors, development, infrastructure
• Extensible…partner ecosystem

Confidence

INFORMATION
APPLICATIONS
INFRASTRUCTURE

Traditional

Public Cloud
Private Cloud

53

• Security…across info, apps, infrastructure, delivery models
• Management …end to end
• Automation…for cloud based architectures & processes

Managed Cloud


Consistency
• Common architecture…across all delivery models
• Portability…for flexibility & optimization
• Consumption experience…one simple model

Security Risks with the Cloud
A major concern for the CISO
Uncertain ability to enforce provider site security policies
Questionable privileged access control at provider site
Inadequate training and IT auditing
Proximity of your data to someone elses
Uncertain ability to recover data
Access across an untrusted network
Uncertain provider regulatory compliance
Uncertain continued existence of provider
Uncertain ability to audit provider

0%

5%

10%

15%

20%

25%

Source: 2012 Global State of Information Security Survey, PricewaterhouseCoopers, CIO magazine, CSO magazine, September 2011

54


30%

35%

Cloud Security Challenges
Cloud Service Provider Perspective

IaaS

PaaS

SaaS

• Increasing security responsibilities at
the information, application & user
layers
• Abstraction and automation of
O/S, Network, and Physical layers

User Management

Application

Application

Application
Platform

Application
Platform

O/S

O/S
image

• PaaS: Heroku, Google App.
Engine
• SaaS: SalesForce.com

55


SaaS
PaaS

• Examples:

• IaaS: HP, Amazon EC2

Application

Network

IaaS
Physical
Consumer responsible

Provider responsible

HP Cloud File Management


Cloud File Management
Secure access, caching, and storage capabilities for your business files

File sync and store
Online workspaces
Data encryption

Base

Device security
SharePoint integration
Storage controllers
Optional

Additional user storage

57


File, sync, and
share
anytime, anywhere
, from any device

Secure access, caching, and storage capabilities for your business files.

File sync and store

Store files in the secure HP cloud and sync automatically to your
device when connected.

Online workspaces

Setup private and public folders in the cloud with password
protection.

Data encryption

Secure data files with 256-bit encryption both in transit and at
rest in the cloud.

Device security

Set up multiple security configurations with flexible access or
more secure modes to provide options.

Storage controllers

Segregated data storage within the HP cloud and on-premise to
meet data governance rules.

Additional user storage

58

Allows users to access SharePoint files from within the Cloud
File Management environment with read/write document access.

Order and configure additional storage in 100 GB blocks, shard
across all users.


Flexible new access across multiple devices

Limited mobile access

ECS–Mobility

Files are shared/stored across systems and
devices
with restricted access.

Files secured in the cloud and synced across
devices
with flexible access.

SharePoint

Email

59

Database

Server

Mobile

Desktop

Notebook


File sync and store
Store files in the secure HP cloud and sync automatically

• Users can store files in a secure cloud
• Synchronize data automatically to any device
for offline viewing
• Includes 5GB of user storage in the HP cloud
• Changed files are automatically updated to
the cloud and accessible by any PC or mobile
device
• Files cached on the device are also available
for offline viewing

60


Online workspaces
Share files securely in the cloud

• Provides private and public folders in the cloud
with password protection
• Users can set access control rights for uploads
and downloads
• Reduces email storage requirements by sending
secure links to files instead of actual attachments.
• Request a file upload from co-workers or third
parties by providing a secure link

61


Data encryption
Secure data both in transit and at rest in the cloud

• In-transit encryption for file transfers using
256 bit SSL for security
• Cloud storage encryption for files uploaded to
Cloud File Management servers
• File metadata has a unique encryption key, stored
separately from the file itself, and encrypted when
uploaded
• Downloaded files are decrypted before their
contents are sent to your browser

• On-device encryption is available through the
Management Essentials package

62


Device security
Multiple security configurations for file access

• Provides flexible options for security administration
• Configure a standard mode with flexible access to
files
• Configure a more secure mode with options like pin
locks, file self-destruct, and disabling offline access
• Custom configurations are also available

63


Configuration
Management

Additional integration opportunity

Users get access to SharePoint files from within
Cloud File Management with read and write
access to documents stored there.

64


Storage controllers
Create and manage local storage centers

• Storage controllers allows administrators to manage
on-premise storage centers alongside cloud-based
storage
• Store select files within the HP converged cloud and/or
on-premise to meet data governance rules
• Integrate storage controllers with any CIFS-based
network share from any storage vendor

• Enable mobile access to file shares and document
management systems through a single interface

65


ControlPoint 4.1
Slide Demonstration


67


68


69


70


71


72


73


74


75


76


HP Records Manager
8.0
Slide Demonstration


78


79


80


81


82


83


84


85


86


87


88


89


90


91


Dealing with Dark Data

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Dealing with Dark Data

Similar to Dealing with Dark Data (20)

Recently uploaded

Recently uploaded (20)

Dealing with Dark Data

Editor's Notes