Big data science faces challenges related to volume, variety, and velocity of data. Specifically: - Petabytes of structured and unstructured data from 10 million subscribers across 10 touchpoints is too big for traditional data warehouses. This includes event logs, program data, content metadata, and purchase histories. - The data comes in at 140 MB/s (12 TB/day), which is too fast as 95% of the data is dropped currently. - The data has inconsistent structures, fragmented XML, and shifting fields, making it difficult to extract features and implement machine learning algorithms at scale.

1. Big Data Science Challenges
Chandan Rajah – Chief Architect, Big Data
crajah@parallelai.com [ @chandanrajah ]

2. Why Big Data Science ?Big Data
Value & Vision
• Machine learning (clustering, classification, regression, pattern
mining, behaviour analysis, semantic analysis, topic extraction)
• Real time analytics & recommendations
• Central smelting pot
• Cost to data benefits
Volume & Variety
• 10 million subscribers;10 different touch points
• Petabytes of data; structured and unstructured
• Event logs, program data, content metadata, purchase history, etc.
• Too big for traditional data warehouse
Velocity & Veracity
• 140 MB/s approx. 12 TB/day
• Too fast; 95% of the data dropped
• Inconsistent data structure
• No single version of truth

3. Big Data Science ChallengesBig DataBig Data
Data Quality
Feature Extraction
Machine Learning
Visualisation
& Verification
Productizing
• Dirty unstructured data with inconsistent labels
• Start but no end events
• Field shifts between extracts
• XML fragmented data; 100k frags
• Data too big to run in R requires subsampling and effective implementation
• 100s of features; too big for Scala / Scalding tuple
• No clearly identifiable keys
• Algorithm implementation issues (e.g. parallelism, scalability, testability)
• Collaborative filtering, topic modelling, incremental clustering, sentiment
analysis
• Real time versus batch algorithm design
• Visualisation tool support
• Automated testing frameworks
• R -> Scala / Scalding not easy
• Disaster recovery & cross data centre
• On the fly analytics; data streams