Around the currently available large piles of Big Data, there's happening quite a mixed gathering: Business Engineers define which insightswould be precious, Analysts build models, Hadoop programmers tame the flood of data, and Operations people setup machines and networks. It's exactly the interplay of all participants which is central to project success. This setup together with the distributed nature of processing poses new challenges to well-established models of assuring software artifact quality: How can non-programmers define acceptance criteria? How can functionalities be tested which depend on cluster execution, orchestration of, e.g., different hadoop jobs without delaying the development process? Which data selection is suited best for simulating the live environment? How can intermediate results in arbitrary serialization formats be inspected? In this talk, experiences and best practices from approaching these problems in a large-scale log data analysis project will be presented. At 1&1, our team develops hadoop applications which process roughly 1 billion log events (~1 TB) per day. We will give an overview of the hard- and software setup of our quality assurance environment, which includes FitNesse as a wiki-style acceptance testing framework.Starting from a comparison with existing test frameworks like MRUnit, we will explain how we automate the parameterized deployment of our applications, choose test data sampling strategies, perform workflow management and orchestration of jobs / applications, and use Pig for inspection of intermediate results and definition of final acceptance criteria. Our conclusion is that test-driven development in the field of Big Data requires adaption of existing paradigms, but is crucial for maintaining high quality standards for the resulting applications.

1. Bug bites Elephant?
Test-driven Quality Assurance
in Big Data Application Development
Dr. Dominik Benz, inovex GmbH
2013/06/03, Berlin Buzzwords

2. ? TDD!
?
?
?
?
?
?
?
Write/execute tests, specify
acceptance criteria, …
2
Who speaks… … the Elephant language?
Class A
extends
Mapper…
ROI, $$, …
apt-get
install…

3. 3
The road… … to Big Data QA
our Big Data
QA problem
the FitNesse
approach
test data
definition /
selection
job & workflow
control
result
inspection

4. 4
QA problem Web Intelligence @ 1&1
DWH
Hadoop Cluster
~ 1 billion log
events / day,
~ 1 TB (thrift)
logfiles
chains of MR jobs,
running on
20 nodes / 8 cores /
96 GB RAM (CDH)
BI reporting, web
analytics, …

5. 5
QA problem An exemplary workflow
Log Files
(thrift)
Log Files
(thrift)
Log Files
(thrift)
Inter-
mediate
result
(avro)
MR
job 1
… DWH
(RDBMS)
MR
job 2
create
(sample)
input data
create
(sample)
input data
? inspect
(binary)
formats
inspect
(binary)
formats
? control
workflows
control
workflows
?

6. method tests what? issues for our usecase
JUnit isolated functions no integration, Java syntax
MRUnit 1 mapper + 1 reducer „little“ integration, Java
syntax
iTest hadoop jobs/workflows Java / Groovy syntax
Scripts/CLI (manual) scripting/inspect. „script chaos“, syntax
6
QA problem Existing Approaches
FitNesse as suitable addition / solution!

7. 7
The road… … to Big Data QA
Big Data QA
is different!
the FitNesse
approach
test data
definition /
selection
job & workflow
control
result
inspection

8. 8
FitNesse In a nutshell
„fully integrated
standalone wiki and
acceptance testing
framework”
• „executable“ Wiki-
Pages (returning test
results)
• (almost) natural
language test
specification
• connection to SUT
via (Java-)“Fixtures“

9. 9
FitNesse Architecture Overview
script |
check |
num results |
3 |
Browser
FitNesse
Server
public int
numResults
{ ... }
System under Test
Fixtures
„calling java methods from
wiki“, compare return values
Integrates with REST, Jenkins…

10. 10
FitNesse An Exemplary Test

11. 11
FitNesse Exemplary Test Source
!path /home/inovex/lib/*.jar
| script | Hadoop |
| upload | viewLog.csv | to hdfs | /testdata/ |
| hadoop job from jar | viewLog.jar | [...] |
| show | job output |
| check | number of output files | 3 |

12. 12
FitNesse Hadoop Fixture Java Code
public class Hadoop {
public boolean uploadToHdfs(String localFile,
String remoteFile) {...}
public boolean hadoopJobFromJar(String jar,
String input, String output) {...}
public String jobOutput() {...}
public String numberOfOutputFiles() {...}
}

13. 13
The road… … to Big Data QA
Big Data QA
is different!
Fitnesse Wiki
test execution!
test data
definition /
selection
job & workflow
control
result
inspection

14. 14
Test Data CSV

15. ‣ Big Data: Efficient data transfer among heterogeneous sources
‣ Define Interface via IDL, Compiler for many languages
15
Test Data Thrift

16. ‣ Dev/Test Hadoop Cluster: Identical Hardware like Prod, but
fewer nodes
‣ (random/biased) sampling e.g. on daily basis
‣ Feedback loop:
‣ identify „special cases“ from real data
‣ include them in (manual) data definition
‣ Gradually increase test coverage / artefact quality
16
Test Data Real World Data

17. 17
The road… … to Big Data QA
Big Data QA
is different!
FitNesse Wiki
test execution!
Define CSV /
thrift / real-
world test data!
job & workflow
control
result
inspection

18. ‣ Execute arbitrary (shell) commands
‣ Mainly a wrapper around apache.commons.exec.CommandLine
18
Job Control Swiss Army Knife: Shell

19. ‣ Hide complexity from test authors
‣ „define“ appropriate test language via (Java) method names
‣ re-use other fixtures (Shell, …) internally
19
Job Control Hadoop Fixture

20. ‣ FitNesse allows to group tests into suites
‣ Can be used to simulate MR processing chains
‣ SetupSuite / TearDownSuite for creating /
destroying test conditions
‣ Tests can still be executed individually
20
Job Control Workflows & Suites
MR job 1
MR job 2

21. 21
The road… … to Big Data QA
Big Data QA
is different!
FitNesse Wiki
test execution!
Define CSV /
thrift / real-
world data!
Use suites & fixtures
for jobs/workflows!
result
inspection

22. ‣ Validate RDBMS contents (via JDBC)
‣ E.g. for checking the final result
‣ Or use Hive + Hive-Server to query raw data
22
Results Data Warehouse / Hive

23. ‣ Execute arbitrary pig commands from Wiki page
‣ Inspect e.g. binary intermediate results (avro, …)
23
Results Pig

24. public class PigConsole extends PigServer {
public void loadAvroFileUsingAlias(String
filename, String alias) {
this.registerQuery(
alias + "= LOAD" + filename + "USING" +
AVRO_STORAGE_LOADER + ";");
}
}
24
Results Pig Fixture extends PigServer

25. 25
Results Server Infrastructure
Fitnesse Master
TestEnvironments
ProjA ProjB
TestConfigurations
ProjA ProjB
dev qs live dev qs live
Import / edit
tests remotely
QS
ProjA Slave
Dev
ProjA Slave
Live
ProjA SlaveProjA
QS
ProjA Slave
Dev
ProjA Slave
Live
ProjA Slave
Import / edit
config remotely
dev qs live

26. 26
Thank you! dominik.benz@inovex.de
Big Data QA
is different!
FitNesse Wiki
test execution!
Define CSV /
thrift / real-
world data!
Inspect results
via Pig/Hive
Use suites & fixtures
for jobs/workflows!

27. 27
Want more? Inovex trains you!
Android Developer Training (3 days, Karlsruhe/München)
Hadoop Developer Training (3 days, Karlsruhe/Köln)
Certified Scrum Developer Training (5 days, Köln)
Pentaho Data Integration Training (4 days, München/Köln)
Liferay Portal-Admin Training (3 days, Karlsruhe)
Liferay Portal-Developer Training (4 days, Karlsruhe)
information and registration at
www.inovex.de/offene-trainings

28. 28
Inovex @bbuzz
Stefan Kathrin Bernhard
Jörg
Andrew
Christian Christian