Matthew S. Weber presented research at the 130th Annual Meeting of Big Data, Big Theory & The Thread of Recent History. The presentation discussed analyzing large-scale datasets to study how complete they are over time. It found datasets on political events and natural disasters became less complete as more webpages and URLs were added over multiple crawls. However, the rate of incompleteness followed exponential functions and could be corrected for using established factors for each dataset. While reliability challenges are not unique, understanding degradation rates can help researchers account for gaps in large internet-sourced datasets.

1. Matthew S. Weber
Rutgers University
@docmattweber
Presented At
130th Annual Meeting of
Big Data, Big Theory &
The Thread of Recent History

2. Credit: Flickr @ilovecology

3. Pekin Daily Times, Pekin, IL, October 8, 2013
Credit: Pekin Daily Times

4. 4

7. What’s in the data?
7
Source | Destination | Date | Frequency | Content Type | Bytes | Content
Link Data:
http://gawker.com/5953665/mitt-romneys-
staff-played-the-media-covering-them-in-a-
friendly-game-of-flag-football
Mitt Romney's Staff Played the Media Covering
Them in a Friendly Game of Flag
http://gawker.com
2012-10-22

8. 8

13. 13
News Media on the Web
(Weber, Ognyanova, Kosterich & Nguyen, 2015)

16. NJ Local News: 2007 - 2012

17. 17
0
1
2
3
4
5
6
7
0
100
200
300
400
500
600
700
800
900
1000
2007 2008 2009 2010 2011 2012
Avg.MBperWebpage
Avg.NumberofWebpages
NJ.com Domain Analysis
Number of Pages Avg MB

18. 18
Dataset Research Potential Dates Captures Unique URLs
Hurricane Katrina Online networks and organizational
resilience (Chewning, Lai and Doerfel,
2012; Perry, Taylor and Doerfel, 2003) in
the wake of disasters; information
dissemination
2003 – 2012 1,694,236 663,740
Superstorm
Sandy
2003 – 2012 41,703,112 20,013,455
US Senate Study the growth of political activity in
online environments (Adamic & Glance,
2005; Bruns, 2007; Chang & Park, 2012);
polarization & media discourse
109th – 112th
Congresses
26,965,770 8,674,397
US House 51,840,777 12,410,014
Occupy Wall
Street
Previous research on NGOs in the online
environment (Bach & Stark, 2004;
Shumate, 2003, 2012; Shumate, Fulk, &
Monge, 2005); use of hyperlink data to
study the formation and role of alliances
between SMOs
2010 – 2012 247,928,272 11,3259,655
US Media
Previous studies of news media
organizations (Greer & Mensing, 2006;
Weber, 2012; Weber & Monge, In
Press); focus on evolutionary patterns
2008 – 2012 1,315,132,555 539,184,823

19. To what degree are large-scale datasets reliable?

20. 20

21. 21

22. 22

23. 0 5 10 15 20 25 30
050000010000001500000200000025000003000000
Potential vs. Actual URLs
CountofPages
23t
CountofURLs
Potential
Actual
Difference

24. 24
0e+002e+064e+066e+06
Changes in Crawl Completeness
CountofPages
t
CountofURLs
OWS
House
Senate
Katrina
existing
potential
b =
set a unit of time for analysis, c
choosing n periods across a total time T

25. In the ideal case, it would be possible to create a factor that corrects
for data degrade:
bt
How does this help?
Each of the illustrated cases fits against an
exponential function ~ b
• Senate: 0.13
• House: 0.13
• Katrina: 0.02
• OWS: 0.10
25
ebt

26. 26
Challenges are not unique to these
data
Courtesy of Marc Smith, NodeXL

27. 27

28. Research support from:
NSF Award #1244727; Additional support from the NetSCI Lab @ Rutgers