1. Inference with
Big Data
A Superpower
Approach
Galit Shmuéli
Indian School of Business
CECR
Mohit Dayal Mingfeng Lin
Lalita Reddi Hank Lucas
Bhim Pochiraju

2. Big data studies (in information systems)
increasingly common
# IS papers with n>10,000 (2004-2010)

3. Large-study IS papers: How Big?
“over 10,000 publicly available feedback text comments… in eBay”
The Nature and Role of Feedback Text Comments in Online Marketplaces
Pavlou & Dimoka, ISR 2006
For our analysis, we have … 784,882 [portal visits]
Household-Specific Regressions Using Clickstream Data
Goldfarb & Lu, Statistical Science 2006
“51,062 rare coin auctions that took place… on eBay”
The Sound of Silence in Online Feedback
Dellarocas & Wood, Management Science 2006
“We collected data on … [175,714] reviews from Amazon”
Examining the Relationship Between Reviews and Sales
Forman et al., ISR 2008
108,333 used vehicles offered in the wholesale automotive market
Electronic vs. Physical Market Mechanisms
Overby & Jap, Management Science 2009
“we use… 3.7 million records, encompassing transactions for the Federal Supply Service
(FSS) of the U.S. Federal government in fiscal year 2000
Using Transaction Prices to Re-Examine Price Dispersion in Electronic Markets
Ghose & Yao, ISR 2011

5. Apply small sample approach to
Big Data studies?

6. It’s all about Power

7. Magnify effects
Separate signal
from noise

8. Artwork: Running the numbers by Chris Jordan (www.chrisjordan.com)
426,000 cell phones retired in the US every day

9. Power = Prob (detect H1 effect)
= f ( sample size, effect size, a, noise )

10. Rare
events
Stronger
validity
Small &
complex effects
The Promise

11. Statistical Technology
Hypotheses
Data Exploration
Models
Model Validation
Inference

12. Chapter 1:
With Mohit Dayal & Lalita Reedi (ISB)
DATA VIZ: “BIG DATA” CHARTS

13. Scaling Up Data Visualization
Missing values

14. Big Data Scatter plot

15. Visualization:
Big Data
Boxplot

16. BIG DATA (SUPERPOWER) APPROACH:
Charts based on aggregation
Interactive viz (zoom & pan, filter, etc.)

17. BIG DATA AND SMALL-SAMPLE
INFERENCE

18. Simple Hypotheses Assumptions?
H1: b1>0
Few control
Few hypotheses variables
𝑓 𝑦 = 𝛽0 + 𝛽1 𝑋1 + 𝛽2 𝑋2 + 𝛽3 𝑋1 𝑋2 + ⋯ + 𝛽 𝑘 𝑋𝑘 + 𝜀
Which model?
What data? Sign +
Statistical
significance
Model fit Robustness
𝑓 𝑦 = 𝛽0 + 𝛽1 𝑋1 + 𝛽 2 𝑋2 + 𝛽 3 𝑋1 𝑋2 + ⋯ + 𝛽 𝑘 𝑋𝑘

19. What doesn’t scale up?
(wrong conclusions)
What are missed opportunities?

20. Chapter 2:
With Hank Lucas (UMD) & Mingfeng Lin (UoA)
TOO BIG TO FAIL:
LARGE SAMPLES AND FALSE DISCOVERIES

21. small p-values*** are not interesting

22. p-value ~ proximity of sample to H0
= f(effect size, sample size, noise)
H0: b=0
Large sample result:
Deflated p-values

23. auctions for digital cameras Aug ’07- Jan ‘08
[thanks to Wolfgang Jank for the data!]
ln Price  b0  b1 *ln(minimumBid )  b 2 * reserve  b3 *ln(sellerFeedback )
 b 4 (Duration)  b5 (controls)  
H1: Higher minimum bids lead to higher final prices (b1>0)
H2: Auctions with reserve price will sell for higher prices (b2>0)
H3: Duration affects price (b4≠0)
H4: The higher the seller feedback, the higher the price (b3>0)
n=341,136

24. n=341,136

26. “In a large sample, we can obtain very large t
statistics with low p-values for our predictors,
when, in fact, their effect on Y is very slight”
Applied Statistics in Business & Economics
Doane & Seward

27. BIG DATA (SUPERPOWER) APPROACH:
Focus on size (ignore p-values)
Subsamples for robustness: “results quantitatively similar”

28. Chapter 3:
MODEL ASSUMPTIONS,
DIAGNOSTICS AND ADJUSTMENT

29. With big data, we’re in the realm of
asymptotic behaviour
𝑛→∞

30. Violated assumptions: less tinkering
Assumption Coefficient Standard Redundant diagnostic
Violation bias errors tests
Under-specification all bias
Endogeneity* all 2SLS is Instrument strength
worse (Sargan)
E(e) =0 𝜷𝟎 bias Lack-of-fit
Non-normality Anderson-Darling
Heteroscedasticity bias Breusch-Pagan
Over-specification
Serial dependence bias Durbin -Watson
Multicollinearity increase Significant correlations
Influential outliers Leverage (multiple testing)
*IV estimators only have desirable asymptotic, not finite sample, properties

31. BIG DATA (SUPERPOWER) APPROACH:
Focus on bias-related assumptions
Avoid statistical tests (p-value challenge)

32. Chapter 4:
With Bhimsankaram Pochiraju & Mohit Dayal (ISB)
COMPLEX EFFECTS &
HETEROGENEITY

33. With Big Data:
Detect small (but important) effects
Detect rare events (in rare minorities)

34. Complex H1: b3>2 Fixed effects Less
hypotheses assumptions
control
variables
𝑓 𝑦 = 𝛽0 + 𝛽1 𝑋1 + 𝛽2 𝑋12 + 𝛽3 𝑋2 𝑋3 + ⋯ + 𝛽 𝑘 𝑋𝑘 + 𝜀
Which model? Heterogeneous
Clustering/Mixtures
Sub-samples
What data? Propensity Scores, 2SLS
Predictive Model fit Robustness magnitude
𝑓 𝑦 = 𝛽0 + 𝛽1 𝑋1 + 𝛽 2 𝑋2 + 𝛽 3 𝑋1 𝑋2 + ⋯ + 𝛽 𝑘 𝑋𝑘

35. Test complex
hypotheses
Moderators
Nonlinear relationships
Multiple categories
Control variables
The rovers have a magnifying camera… that
scientists can use to carefully look at the fine
structure of a rock
Quantify subtle effects
Specific measures (20 eBay categories)
“Moderator variables are difficult to detect” -Aguinis, 1994
Low R2, yet non-zero coefficients

37. Stepwise Selection
OLS with Stepwise (AIC measure)
Logistic with variable selection (RELR)
All independent variables
All control variables
Quadratic terms of continuous variables
2-way interactions
Choose software carefully (R: “out of memory”)

38. Heterogeneity: CART
• Identify non-linearities and
interactions
• Does not identify different models
• Challenge: independent variables
vs. control variables

39. Clustering
1. Cluster all
independent and
control variables
2. Fit separate
regression models
to each cluster
• Popular in risk analytics
• Fast, easy
• Does not guarantee
distinct relationships

40. Finite Mixture
Regression
Search for k separate
regressions
Convergence issues
on entire dataset
For 10 subsamples
(n=30K) converged
for seven

41. Chapter 5:
MODEL VALIDATION

42. Improve model validation, comparison,
and generalization
Internal & external validity
Robustness across subsamples
non-random
random
(overlapping/non)

43. Improve predictive validation
Holdout set
Training set

44. SMALL SAMPLE
MODELING APPROACH

45. Clark Kent ≤ Superman