In this talk I show you how to examine JSON data using the jq command-line tool. Sample files and more discussion can be found on blog at http://aijaz.net/2016/10/25/jq/

1. Slide the First
1 — Aijaz Ansari at NSCoderChicago, 10/25/2016

2. Mythical
Creatures
2 — Aijaz Ansari at NSCoderChicago, 10/25/2016

3. http://weknowyourdreams.com/images/unicorn/unicorn-04.jpg
3 — Aijaz Ansari at NSCoderChicago, 10/25/2016

4. https://shellback0608.files.wordpress.com/2016/07/bird-phoenix-flight-art-drawing.jpg
4 — Aijaz Ansari at NSCoderChicago, 10/25/2016

5. https://i.ytimg.com/vi/kKJQESF0qqk/maxresdefault.jpg
5 — Aijaz Ansari at NSCoderChicago, 10/25/2016

6. API
Docs6 — Aijaz Ansari at NSCoderChicago, 10/25/2016

7. Exploring JSON
With jq
Aijaz Ansari
@_aijaz_
7 — Aijaz Ansari at NSCoderChicago, 10/25/2016

8. What is JSON?
JavaScript Object Notation
[
{ "name": "Alice", "age": 20},
{ "name": "Bob", "age": 30}
]
8 — Aijaz Ansari at NSCoderChicago, 10/25/2016

9. Reading JSON returned by remote servers
@interface AAAJSONResponseSerializer : AFJSONResponseSerializer
@end
@implementation AAAJSONResponseSerializer
- (id)responseObjectForResponse:(NSURLResponse *)response
data:(NSData *)data
error:(NSError *__autoreleasing *)error {
#ifdef DEBUG
NSHTTPURLResponse * httpResponse = (NSHTTPURLResponse *) response;
NSString *body = [[NSString alloc] initWithData:data encoding:NSUTF8StringEncoding];
DDLogInfo(@"RESPONSE BODY: %@", response.URL, body);
DDLogInfo(@"RESPONSE: (%ld) %@ ", httpResponse.statusCode, response.URL);
#endif
// ...
}
@end
9 — Aijaz Ansari at NSCoderChicago, 10/25/2016

10. JSON Looks Like This
[{"name":"Aang","sex":"M","born":-12,"died":153,"bending":["Air","Water","Earth","Fire","Energy"],"identity":{"nationality":"Southern Air Temple",
"ethnicity":"Air Nomad"},"spouse":"Katara","children":[{"sex":"M","name":"Bumi"},{"sex":"F","name":"Kya"},{"sex":"M","name":"Tenzin"}]},{"name":"Katara","sex":
"F","born":85,"died":null,"bending":["Water","Blood"],"identity":{"nationality":"Southern Water Tribe","ethnicity":"Water Tribe"},"spouse":"Aang","children":[{"
sex":"M","name":"Bumi"},{"sex":"F","name":"Kya"},{"sex":"M","name":"Tenzin"}]},{"name":"Sokka","sex":"M","born":84,"died":164,"bending":[],"identity":
{"nationality":"Southern Water Tribe","ethnicity":"Water Tribe"},"spouse":null,"children":[]},{"name":"Toph Beifong","sex":"F","born":88,"died":null,"bending":
["Earth","Metal"],"identity":{"nationality":"Gaoling, Earth Kingdom","ethnicity":"Earth Kingdom"},"spouse":null,"children":[{"sex":"F","name":
"Lin Beifong"},{"sex":"F","name":"Suyin Beifong"}]},{"name":"Iroh","sex":"M","born":null,"died":null,"bending":["Fire","Energy"],"identity":{"nationality":
"Fire Nation Capital, Fire Nation","ethnicity":"Fire Nation"},"spouse":null,"children":[{"sex":"M","name":"Lu Ten"}]},{"name":"Zuko","sex":"M","born":83,
"died":null,"bending":["Fire","Energy"],"identity":{"nationality":"Fire Nation Capital, Fire Nation","ethnicity":"Fire Nation"},"spouse":null,"children":
[{"sex":"F","name":"Izumi"}]},{"name":"Kya","sex":"F","born":null,"died":null,"bending":["Water"],"identity":{"nationality":"Southern Water Tribe","ethnicity":
"Water Tribe, Air Nomad"},"spouse":null,"children":[]},{"name":"Bumi","sex":"M","born":null,"died":null,"bending":["Air"],"identity":{"nationality":
"United Republic","ethnicity":"Water Tribe, Air Nomad"},"spouse":null,"children":[]},{"name":"Tenzin","sex":"M","born":null,"died":null,"bending":["Air"],
"identity":{"nationality":"Republic City, United Republic","ethnicity":"Water Tribe, Air Nomad"},"spouse":null,"children":[{"sex":"F","name":"Jinora"},{"sex":
"F","name":"Ikki"},{"sex":"M","name":"Meelo"},{"sex":"M","name":"Rohan"}]},{"name":"Lin Beifong","sex":"F","born":120,"died":null,"bending":["Earth","Metal"],
"identity":{"nationality":"Republic City, United Republic","ethnicity":"Earth Kingdom"},"spouse":null,"children":[]},{"name":"Suyin Beifong","sex":"F","born":
126,"died":null,"bending":["Earth","Metal"],"identity":{"nationality":"Republic City, United Republic","ethnicity":"Earth Kingdom"},"spouse":null,"children":
[{"sex":"M","name":"Bataar Jr."},{"sex":"F","name":"Opal"},{"sex":"M","name":"Wei"},{"sex":"M","name":"Wing"},{"sex":"M","name":"Huan"}]}]
10 — Aijaz Ansari at NSCoderChicago, 10/25/2016

11. jq
Lightweight & Flexible Command-Line
JSON Processor.
https://stedolan.github.io/jq/
11 — Aijaz Ansari at NSCoderChicago, 10/25/2016

12. Pretty Printing
$ echo '[{"a": 1},{"b": 2}]' | jq .
[
{
"a": 1
},
{
"b": 2
}
]
$
If you remember only one thing about jq, let it be
this, the '.' filter.
12 — Aijaz Ansari at NSCoderChicago, 10/25/2016

13. Filters
→ All jq programs are filters
→ A filter takes in input & produces output
→ 1 is the simplest filter: return 1
→ . is the simplest useful filter: pretty print
13 — Aijaz Ansari at NSCoderChicago, 10/25/2016

14. Object Value Filter
$ echo '{"a": 1, "b": 2, "c":3}' | jq .c
3
$
14 — Aijaz Ansari at NSCoderChicago, 10/25/2016

15. Arrays
$ echo '["a", "b", "c", "d", "e"]' | jq '.[2]'
"c"
$ echo '["a", "b", "c", "d", "e"]' | jq '.[1:4]'
[
"b",
"c",
"d"
]
$
15 — Aijaz Ansari at NSCoderChicago, 10/25/2016

16. Iterators
Read [] as foreach
$ echo '["a", "b", "c", "d", "e"]' | jq '.[]'
"a"
"b"
"c"
"d"
"e"
$
Read .[] as iterating with an identity filter
16 — Aijaz Ansari at NSCoderChicago, 10/25/2016

17. Addition
$ echo '["a", "b", "c", "d", "e"]' | jq '.[]+" hello"'
"a hello"
"b hello"
"c hello"
"d hello"
"e hello"
$
17 — Aijaz Ansari at NSCoderChicago, 10/25/2016

18. Addition
$ echo '["a", "b", "c", "d", "e"]' | jq '"hey "+ .[]+" hello"'
"hey a hello"
"hey b hello"
"hey c hello"
"hey d hello"
"hey e hello"
$
18 — Aijaz Ansari at NSCoderChicago, 10/25/2016

19. Addition
$ echo '[1,2,3,4,5]' | jq '.[]+10'
11
12
13
14
15
$
19 — Aijaz Ansari at NSCoderChicago, 10/25/2016

20. Collecting Results Into an Array
$ echo '[1,2]' | jq '.[]+10'
11
12
$ echo '[1,2]' | jq '[.[]+10]'
[
11,
12
]
20 — Aijaz Ansari at NSCoderChicago, 10/25/2016

21. Filters can be
piped with |
21 — Aijaz Ansari at NSCoderChicago, 10/25/2016

22. Creating Objects
$ echo '[{"a": 1, "b": 2, "c": 3}, {"a": 10, "b": 20, "c": 30}]' | jq '.[]'
{
"a": 1,
"b": 2,
"c": 3
}
{
"a": 10,
"b": 20,
"c": 30
}
$
22 — Aijaz Ansari at NSCoderChicago, 10/25/2016

23. Creating Objects
$ echo '[{"a": 1, "b": 2, "c": 3}, {"a": 10, "b": 20, "c": 30}]' | jq '.[]|{a,c}'
{
"a": 1,
"c": 3
}
{
"a": 10,
"c": 30
}
$
If you remember only 2 things about jq, let this be
the second: extracting just the keys you care about.
23 — Aijaz Ansari at NSCoderChicago, 10/25/2016

24. Creating Objects
$ echo '[{"fn": "Ann", "ln": "Smith"},{"fn": "Ted", "ln": "Jones"}]'
| jq '.[]|{"first": .fn, "last": .ln}'
{
"first": "Ann",
"last": "Smith"
}
{
"first": "Ted",
"last": "Jones"
}
$
24 — Aijaz Ansari at NSCoderChicago, 10/25/2016

25. Creating Objects
$ echo '[{"fn": "Ann", "ln": "Smith"},{"fn": "Ted", "ln": "Jones"}]'
| jq '.[]|{(.fn): .ln}'
{
"Ann": "Smith"
}
{
"Ted": "Jones"
}
$
25 — Aijaz Ansari at NSCoderChicago, 10/25/2016

26. Functions
keys
$ echo '[{"a": 1, "b": 2, "c": 3}, {"a": 10, "b": 20, "c": 30}]' | jq '.[]|keys'
[
"a",
"b",
"c"
]
[
"a",
"b",
"c"
]
$
If you remember only 3 things about jq, let this be
the third: examining keys.
26 — Aijaz Ansari at NSCoderChicago, 10/25/2016

27. Functions
length
$ echo '[{"a": 1, "b": 2, "c": 3}, {"a": 10, "b": 20, "c": 30}]' | jq '.[]|length'
3
3
$
27 — Aijaz Ansari at NSCoderChicago, 10/25/2016

28. Functions
map
$ echo '[1,2,3,4,5]' | jq 'map(.+10)'
[
11,
12,
13,
14,
15
]
map(x) is equivalent to [.[] | x]
28 — Aijaz Ansari at NSCoderChicago, 10/25/2016

29. Addition
$ echo '[1,2,3,4,5]' | jq '.[]+10'
11
12
13
14
15
$
29 — Aijaz Ansari at NSCoderChicago, 10/25/2016

30. Addition
$ echo '[1,2,3,4,5]' | jq '[.[]+10]'
[
11,
12,
13,
14,
15
]
$
30 — Aijaz Ansari at NSCoderChicago, 10/25/2016

31. Functions
map
$ echo '[1,2,3,4,5]' | jq 'map(.+10)'
[
11,
12,
13,
14,
15
]
31 — Aijaz Ansari at NSCoderChicago, 10/25/2016

32. Functions
reduce
$ echo '[1,2,3,4,5]' | jq 'reduce .[] as $i (0 ; . + $i)'
15
$
Somewhat equivalent to:
$i = 0
for var in .[] {
$i = $var + $i
}
32 — Aijaz Ansari at NSCoderChicago, 10/25/2016

33. More Functions
→ has("foo")
→ in({"a": "b"})
→ echo '[1,2,3]' | jq 'map(.+2)' # [3,4,5]
→ select(func) - like grep where func is true
→ ... and a whole lot more
33 — Aijaz Ansari at NSCoderChicago, 10/25/2016

34. Detailed Examples With Real Data
34 — Aijaz Ansari at NSCoderChicago, 10/25/2016

35. Sample JSON Body (sample.json)
[{"name":"Aang","sex":"M","born":-12,"died":153,"bending":["Air","Water","Earth","Fire","Energy"],"identity":{"nationality":"Southern Air Temple",
"ethnicity":"Air Nomad"},"spouse":"Katara","children":[{"sex":"M","name":"Bumi"},{"sex":"F","name":"Kya"},{"sex":"M","name":"Tenzin"}]},{"name":"Katara","sex":
"F","born":85,"died":null,"bending":["Water","Blood"],"identity":{"nationality":"Southern Water Tribe","ethnicity":"Water Tribe"},"spouse":"Aang","children":[{"
sex":"M","name":"Bumi"},{"sex":"F","name":"Kya"},{"sex":"M","name":"Tenzin"}]},{"name":"Sokka","sex":"M","born":84,"died":164,"bending":[],"identity":
{"nationality":"Southern Water Tribe","ethnicity":"Water Tribe"},"spouse":null,"children":[]},{"name":"Toph Beifong","sex":"F","born":88,"died":null,"bending":
["Earth","Metal"],"identity":{"nationality":"Gaoling, Earth Kingdom","ethnicity":"Earth Kingdom"},"spouse":null,"children":[{"sex":"F","name":
"Lin Beifong"},{"sex":"F","name":"Suyin Beifong"}]},{"name":"Iroh","sex":"M","born":null,"died":null,"bending":["Fire","Energy"],"identity":{"nationality":
"Fire Nation Capital, Fire Nation","ethnicity":"Fire Nation"},"spouse":null,"children":[{"sex":"M","name":"Lu Ten"}]},{"name":"Zuko","sex":"M","born":83,
"died":null,"bending":["Fire","Energy"],"identity":{"nationality":"Fire Nation Capital, Fire Nation","ethnicity":"Fire Nation"},"spouse":null,"children":
[{"sex":"F","name":"Izumi"}]},{"name":"Kya","sex":"F","born":null,"died":null,"bending":["Water"],"identity":{"nationality":"Southern Water Tribe","ethnicity":
"Water Tribe, Air Nomad"},"spouse":null,"children":[]},{"name":"Bumi","sex":"M","born":null,"died":null,"bending":["Air"],"identity":{"nationality":
"United Republic","ethnicity":"Water Tribe, Air Nomad"},"spouse":null,"children":[]},{"name":"Tenzin","sex":"M","born":null,"died":null,"bending":["Air"],
"identity":{"nationality":"Republic City, United Republic","ethnicity":"Water Tribe, Air Nomad"},"spouse":null,"children":[{"sex":"F","name":"Jinora"},{"sex":
"F","name":"Ikki"},{"sex":"M","name":"Meelo"},{"sex":"M","name":"Rohan"}]},{"name":"Lin Beifong","sex":"F","born":120,"died":null,"bending":["Earth","Metal"],
"identity":{"nationality":"Republic City, United Republic","ethnicity":"Earth Kingdom"},"spouse":null,"children":[]},{"name":"Suyin Beifong","sex":"F","born":
126,"died":null,"bending":["Earth","Metal"],"identity":{"nationality":"Republic City, United Republic","ethnicity":"Earth Kingdom"},"spouse":null,"children":
[{"sex":"M","name":"Bataar Jr."},{"sex":"F","name":"Opal"},{"sex":"M","name":"Wei"},{"sex":"M","name":"Wing"},{"sex":"M","name":"Huan"}]}]
http://aijaz.net/2016/10/25/jq/
35 — Aijaz Ansari at NSCoderChicago, 10/25/2016

36. http://feelgrafix.com/data_images/out/28/973590-avatar-the-last-airbender.jpg
Example data of characters from the 'Avatar' series.
36 — Aijaz Ansari at NSCoderChicago, 10/25/2016

37. # Pretty print the jq
$ cat sample.json | jq '.'
...
{
"sex": "M",
"name": "Huan"
}
]
}
]
$
# The ']' on the last line tells me this is an array
# The '}' on the 2nd-last line tells me this is an array of objects
# From now on, for space reasons,
# 'cat sample.json ' will be abbreviated to 'cj'
37 — Aijaz Ansari at NSCoderChicago, 10/25/2016

38. # Just show me the first object
$ cj | jq '.[0]'
38 — Aijaz Ansari at NSCoderChicago, 10/25/2016

39. # Show me an object with the name and
# sex of the first object
$ cj | jq '.[0]|{name, sex}'
{
"name": "Aang",
"sex": "M"
}
$
39 — Aijaz Ansari at NSCoderChicago, 10/25/2016

40. # Show me all the keys of each object
$ cj | jq '.[]|keys'
...
[
"bending",
"born",
"children",
"died",
"identity",
"name",
"sex",
"spouse"
]
$
40 — Aijaz Ansari at NSCoderChicago, 10/25/2016

41. # Show me all the number of keys of each object
$ cj | jq '.[]|length' # same as .[]|keys|length
8
8
8
8
8
8
8
8
8
8
8
$
41 — Aijaz Ansari at NSCoderChicago, 10/25/2016

42. # The name of each character
$ cj | jq '.[].name'
"Aang"
"Katara"
"Sokka"
"Toph Beifong"
"Iroh"
"Zuko"
"Kya"
"Bumi"
"Tenzin"
"Lin Beifong"
42 — Aijaz Ansari at NSCoderChicago, 10/25/2016

43. # The number of records
$ cj | jq '[.[]]|length'
11
$
43 — Aijaz Ansari at NSCoderChicago, 10/25/2016

44. # All female characters
$ cj | jq '.[]|select(.sex == "F")'
44 — Aijaz Ansari at NSCoderChicago, 10/25/2016

45. # The names of all female characters
# Take the resulting objects, put thmm in an array
# Then iterate over that array, printing the name
$ cj | jq '[.[]|select(.sex == "F")]|.[].name'
"Katara"
"Toph Beifong"
"Kya"
"Lin Beifong"
"Suyin Beifong"
$
45 — Aijaz Ansari at NSCoderChicago, 10/25/2016

46. # The name of all members of the Fire Nation
# You can drill down objects with multiple .s
# e.g.: .identity.ethnicity
$ cj | jq '[.[]|select(.identity.ethnicity == "Fire Nation")]|.[].name'
"Iroh"
"Zuko"
46 — Aijaz Ansari at NSCoderChicago, 10/25/2016

47. # The name and bending list of all characters who can bend Fire
# Similar to previous example: iterate, select, construct a list, iterate on that list,
# and extract the name and bending fields
$ cj | jq -c '[.[]|select(.bending[]|select(contains("Fire")))]|.[]|{name, bending}'
{"name":"Aang","bending":["Air","Water","Earth","Fire","Energy"]}
{"name":"Iroh","bending":["Fire","Energy"]}
{"name":"Zuko","bending":["Fire","Energy"]}
$
47 — Aijaz Ansari at NSCoderChicago, 10/25/2016

48. # Similar to previous, but make the name be the key of the object
$ cj | jq -c '[.[]|select(.bending[]|select(contains("Fire")))]|.[]|{(.name): .bending}'
{"Aang":["Air","Water","Earth","Fire","Energy"]}
{"Iroh":["Fire","Energy"]}
{"Zuko":["Fire","Energy"]}
48 — Aijaz Ansari at NSCoderChicago, 10/25/2016

49. # Names and firstborns of each character, as hashes in an array
$ cj | jq '[.[]|{name, firstBorn: .children[0].name}]'
49 — Aijaz Ansari at NSCoderChicago, 10/25/2016

50. # How many children does each person have?
$ cj | jq '[.[]|.name, (.children|length)]'
[
{
"Aang": 3
},
{
"Katara": 3
},
...
{
"Suyin Beifong": 5
}
]
$
50 — Aijaz Ansari at NSCoderChicago, 10/25/2016

51. # List of each person, along with the number of their children
# Essentially, merging the array of objects from the previous example into 1 big object
$ cj | jq '[.[]|[.name, (.children|length)]] | reduce .[] as $i ({}; . + { ($i[0]): $i[1]} )'
{
"Aang": 3,
"Katara": 3,
"Sokka": 0,
"Toph Beifong": 2,
"Iroh": 1,
"Zuko": 1,
"Kya": 0,
"Bumi": 0,
"Tenzin": 4,
"Lin Beifong": 0,
"Suyin Beifong": 5
}
$
51 — Aijaz Ansari at NSCoderChicago, 10/25/2016

52. # Take the object matrix from two slides ago, and transpose it
$ cj | jq '[.[]|[.name, (.children|length)]] | transpose'
[
[
"Aang",
"Katara",
"Sokka",
"Toph Beifong",
...
"Suyin Beifong"
],
[
3,
3,
0,
2,
...
5
]
]
$
52 — Aijaz Ansari at NSCoderChicago, 10/25/2016

53. Three Most Useful Filters
'.' # Pretty print
'.[]|{a, b}' # Extract just the a and b keys from each object
'.[]|keys' # Display the keys of each object in the array
Honorable Mention
'[.[]|select(C)]' # Return an array of only those objects where condition C is true
53 — Aijaz Ansari at NSCoderChicago, 10/25/2016

54. Resources
→ jq: https://stedolan.github.io/jq/
→ jq manual: https://stedolan.github.io/jq/manual/
→ Sample JSON file: http://aijaz.net/2016/10/25/jq/
54 — Aijaz Ansari at NSCoderChicago, 10/25/2016

55. Thanks!
55 — Aijaz Ansari at NSCoderChicago, 10/25/2016