Do you have large, complex JSON data sets you need to store and analyze? With the easy-to-use open source OJAI (Open JSON Application Interface) API, you can combine the flexibility of JSON with the scale and performance of NoSQL to quickly write the modern applications your users need. Check out these slides and watch this session of Free Code Fridays here: www.youtube.com/watch?v=MLaOXjmQcuA

1. © 2016 MapR Technologies© 2016 MapR Technologies
Importing Big Data Sets in JSON

2. © 2016 MapR Technologies
Big Datasets and MapR-DB JSON
● Introducing jq
● maprdb ImportJSON
● Write Your Own OJAI Document Builder
MapR-DB JSON & OJAI
● Scalable JSON Document Datastore
● Utilizes the OJAI Open Source API
● Command Line Utility
● Developer Preview Sandbox

3. © 2016 MapR Technologies
Using ‘jq’ -- The ‘sed’ for JSON
● Unix Style Filter (standard in/standard out)
● Can Used to Select JSON Data
● Can be Used to Transform JSON Data
head -n 2000 chicago_crime_data-complete.json > ....
cat chicago_crime_data-partial.json | jq -c '.data[]'
cat chicago_crime_data-partial.json | jq -c '.data[] | {
"id":.[1],"case":.[0]}'
cat citylots.json | jq -c '.features[]'
python -mjson.tool chicago_crime_data-partial.json

4. © 2016 MapR Technologies
maprdb ImportJSON
● Import a Single File or Multiple Files
● Specify a Row Key (_id)
● One JSON Document per Line
● Can Run as YARN Map-Reduce Job
maprdb Examples
maprdb importJSON -src /user/mapr/foo.json -dst /user/mapr/foobar
maprdb shell
maprdb mapr:> find '/user/mapr/foobar' --id 605460
maprdb importJSON -src /user/mapr/import.file -dst /user/mapr/foobar -idField
properties.BLKLOT

5. © 2016 MapR Technologies
Roll Your Own
● Parse JSON from File, Stream, Message Queue, Unix Pipe
● Use DocumentBuilder API to Construct OJAI Document
● Specify the Key or Have ‘_id’ Field in Document
java -jar ojai-import-json-0.1-SNAPSHOT-jar-with-dependencies.jar --help
usage: ojaiimport
-f,--file <arg> Use this file as input instead of standard in
-h,--help This message
-k,--key <arg> JSON element to use as the '_id' (rowkey)
cat single-record.json |
java -jar ojai-import-json-0.1-SNAPSHOT-jar-with-dependencies.jar
/user/mapr/foobar

6. © 2016 MapR Technologies
Links
● https://stedolan.github.io/jq/tutorial/
● https://robots.thoughtbot.com/jq-is-sed-for-json
● http://ojai.github.io/javadocs/latest/
● https://www.mapr.com/hadoop-developer-quicklinks
● https://www.mapr.com/developer-preview/mapr-db-document-
database
● https://s3.amazonaws.com/mapr-fcf/datasets.zip
● https://github.com/mapr-demos/ojai-import-json
● https://data.cityofchicago.org/
● https://data.sfgov.org/

7. © 2015 MapR Technologies 7
Q & A
@mapr maprtech
pcurtis@mapr.com
Engage with us!
MapR
maprtech
mapr-technologies