This document summarizes a presentation given by Robert Lancaster and Jonathan Seidman about how their company, Orbitz, is extending their enterprise data warehouse with Hadoop. They discuss how Hadoop provides scalable storage and processing of large amounts of log and web analytics data. They then provide examples of how this data is used for applications like optimizing hotel search, recommendations, and user segmentation. Finally, they outline their vision of integrating Hadoop and the data warehouse to provide a unified view for business intelligence and analytics tools.

1. Extending the Enterprise Data Warehouse with Hadoop
Robert Lancaster and Jonathan Seidman
Chicago Data Summit
April 26 | 2011

2. Who We Are
• Robert Lancaster
– Solutions Architect, Hotel Supply Team
– rlancaster@orbitz.com
– @rob1lancaster
• Jonathan Seidman
– Lead Engineer, Business Intelligence/Big Data Team
– Co-founder/organizer of Chicago Hadoop User Group
(http://www.meetup.com/Chicago-area-Hadoop-User-
Group-CHUG)
– jseidman@orbitz.com
– @jseidman
page 2

3. Launched: 2001, Chicago, IL
page 3

4. Why are we using Hadoop?
Stop me if you’ve heard this before…
page 4

5. On Orbitz alone we do millions of searches and transactions
daily, which leads to hundreds of gigabytes of log data
every day.
page 5

6. Hadoop provides us with efficient, economical,
scalable, and reliable storage and processing of
these large amounts of data.
$ per TB
page 6

7. And…
Hadoop places no constraints on how data is
processed.
page 7

8. Before Hadoop
page 8

9. With Hadoop
page 9

10. Access to this non-transactional data enables a number of
applications…
page 10

11. Optimizing Hotel Search
page 11

12. Recommendations
page 12

13. Page Performance Tracking
page 13

14. Cache Analysis
100.00%
72% of queries are Queries
singletons and make up
90.00% Searches
nearly a third of total
search volume.
80.00% Reverse Running Total
(Searches)
71.67%
Reverse Running Total
70.00% (Queries)
60.00%
A small number of
queries (3%) make
50.00% up more than a third
of search volume.
40.00%
34.30%
31.87%
30.00%
20.00%
10.00%
2.78%
0.00%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
page 14

15. User Segmentation
page 15

16. All of this is great, but…
Most of these efforts are driven by development
teams.
The challenge now is to unlock the value in this data
by making it more available to the rest of the
organization.
page 16

17. “Given the ubiquity of data in modern organizations, a data
warehouse can keep pace today only by being “magnetic”:
attracting all the data sources that crop up within an
organization regardless of data quality niceties.”*
*MAD Skills: New Analysis Practices for Big Data
page 17

18. In a better world…
page 18

19. Integrating Hadoop with the Enterprise Data Warehouse
Robert Lancaster and Jonathan Seidman
Chicago Data Summit
April 26 | 2011

20. The goal is a unified view of the data, allowing us to use
the power of our existing tools for reporting and analysis.
page 20

21. BI vendors are working on integration with Hadoop…
page 21

22. And one more reporting tool…
page 22

23. Example Processing Pipeline for Web Analytics Data
page 23

24. Aggregating data for import into Data Warehouse
page 24

25. Example Use Case: Beta Data Processing
page 25

26. Example Use Case – Beta Data Processing
page 26

27. Example Use Case – Beta Data Processing Output
page 27

28. Example Use Case: RCDC Processing
page 28

29. Example Use Case – RCDC Processing
page 29

30. Example Use Case: Click Data Processing
page 30

31. Click Data Processing – Current DW Processing
Web
Data
Server
Web Cleansing
Web
Server
Logs ETL DW (Stored DW
Servers
procedure)
3 hours 2 hours ~20%
original
data
size
page 31

32. Click Data Processing – New Hadoop Processing
Web Data
Server
Web Cleansing
Web
Server
Logs HDFS (MapReduce) DW
Servers
page 32

33. Conclusions
• Market is still immature, but Hadoop has already become a
valuable business intelligence tool, and will become an
increasingly important part of a BI infrastructure.
• Hadoop won’t replace your EDW, but any organization with a
large EDW should at least be exploring Hadoop as a
complement to their BI infrastructure.
• Use Hadoop to offload the time and resource intensive
processing of large data sets so you can free up your data
warehouse to serve user needs.
• The challenge now is making Hadoop more accessible to non-
developers. Vendors are addressing this, so expect rapid
advancements in Hadoop accessibility.
page 33

34. Oh, and also…
• Orbitz is looking for a Lead Engineer for the BI/Big Data team.
• Go to http://careers.orbitz.com/ and search for IRC19035.
page 34

35. References
• MAD Skills: New Analysis Practices for Big Data, Jeffrey
Cohen, Brian Dolan, Mark Dunlap, Joseph Hellerstein, and
Caleb Welton, 2009
page 35