This document discusses how Hadoop can be used in the Microsoft enterprise to make sense of data fragmentation. It proposes starting simply by using Hadoop (HDInsight) for log storage. Existing ETL processes can be reused by incrementally adding Hadoop. This allows organizations to understand their data in a more complete, contextual way and take a customer-centric view, like seeing all customer interactions across systems. Starting small and getting different departments involved can encourage experimentation with Hadoop and analytics.

1. Hadoop
in the Microsoft Enterprise
@DanRosanova
West Monroe Partners
http://danrosanova.wordpress.com

2. W hat does the Microsoft
Enterprise Look Like

3. Microsoft Enterprise - Basics

4. The reality is slightly more complicated
Client OS Office
Active Directory
System Center
Applications
Server OS
Virtualization
NTFS
ETL OLTP OLAP
SAN / NAS

5. Data lives here

6. Users live here

7. System Administrators Live
HERE

8. Many enterprises are fragmented

9. VERY fragmented

10. How do we
make sense of this
Fragmentation
?

12. Into this

14. Existing and New tools
SSAS
ODBC
For HIVE
+
SSIS

15. Applied to Customer
Transactional
Social Media
Web Logs Sqoop
Hive Flume
HDFS
IVR Logs Email & Chat
Pig
Pig

16. Transactional Data
ID Sector Rev Code Acct Amount
01 Q X QW 10000 1200
02 P X AB 20000 1020
03 O X CD 20000 11221
04 N XX QW 50000 2323
05 M X CD 30000 33
06 L XX AB 10000 323231
07 K X ER 20000 12
08 J XXX CD 20000 3233
09 I X QW 40000 5468
10 H X ER 50000 234
11 G O AB 55000 765
12 F X ER 70000 34538
13 E XX ER 25000 3456476
14 D XXX AB 10000 4564
15 C X QW 10000 456
16 B XX YZ 11000 44
17 A X AB 15000 456

17. Transactional + Temporal Data

18. Complete information awareness
Transactional
Temporal
Spatial
Contextual

19. This sounds familiar
How is this different from other
from other consolidation approaches?

20. Work with departmentalization
Not against it

21. Agile Analytics
Self Service
Accessible
Easy to use

22. How do you eat an elephant

23. Start Simple: Storage
$
1000
SAN
HDInsight
(Hadoop)
50
5
TB
5 25 100 1000

24. Create log storage
• IIS (Web Server) Logs
• Event Logs
• Trace Logs

25. Incorporate log analysis
• Give users access to HDInsight / Hadoop with tools
they know
• Encourage departmental experiementation
• Development and IT departments are good early
adopters
• So is marketing

26. Tap into existing ETL
• You already have a TON of ETL
• Reuse as much as you can
• At first aim for replication
• Incrementally add value

27. Why do all this?

28. How most organizations
view their
Customer

29. Some organizations
see

30. What we should see

31. Understand your data
Understand your
@Dan Rosanova
West Monroe Partners
http://danrosanova.wordpress.com