Bob Tiernay explores the fascinating world of jq, "the JSON Processor”. Starting with a motivation, he then covers the language, provides helpful tips, showcases a real world example, cautions some things to avoid and finishes with a discussion of the ecosystem.
6. “jq is a lightweight and flexible
command-line JSON processor
▸ Written in C
▸ No runtime dependencies
▸ Functional programming language
▸ Turing complete
▸ Terse and expressive
▸ Slice, filter, map and transform data with
ease
Much more powerful than Json Path!
Brainf*ck proof!
— Stephen Dolan
7. SOME
USECASES
▸ Exploring JSON APIs
▹ Elasticsearch
▹ GitHub
▹ Docker
▸ Exploring JSONL dumps
▸ Lightweight integration
▸ One-off mini-ETLs
▸ Analytics / aggregations
* Also, just a darn handy JSON formatter!
8. https://stedolan.github.io/jq/
▸ Homepage
▸ Installation instructions
▸ Documentation and resources
Links:
https://jqplay.org/
▸ Interactive jq playground
▸ Great for learning
▸ Share snippets #jq channel on
Freenode
jq tag on
Stackoverflow
9. PLATFORMS
▸ Linux
▸ OSX
▸ FreeBSD
▸ Solaris
▸ Windows
brew install jq
Mac
apt-get install jq
Linux
jq
10. jq INVOCATION1
Inline Input
▸ Read from stdin
▸ Write to stdout
stdin ➜ jq <filter> ➜ stdout
File Input
▸ Read from file
▸ Write to stdout
Most common
curl -s ⋯ | jq ⋯ jq ⋯ file.json
echo ⋯ | jq ⋯
cat ⋯ | jq ⋯
I/O
jq <filter> file ➜ stdout
Try httpie instead!
11. > jq --help
jq - commandline JSON processor [version 1.5]
Usage: jq [options] <jq filter> [file...]
jq is a tool for processing JSON inputs [...]
Some of the options include:
-c compact instead of pretty-printed output;
-n use `null` as the single input value;
-e set the exit status code based on the output;
-s read (slurp) all inputs into an array; apply
-r output raw strings, not JSON texts;
-R read raw strings, not JSON texts;
-C colorize JSON;
-M monochrome (don't colorize JSON);
-S sort keys of objects on output;
--tab use tabs for indentation;
--arg a v set variable $a to value <v>;
--argjson a v set variable $a to JSON value <v>;
--slurpfile a f set variable $a to an array of JSON texts read from <f>;
12. jq INVOCATION2
⋮
⋮
➜ jq ➜ ➜ jq -s ➜
Line by Line “Slurp Mode”
The default
--slurp / -s
Reads as a single array!Read line by line
Input: Output: Input: Output:
Modes
28. VARIABLES$
▸ Usually not needed
▸ Can help to cut down on noise or
repetition
▸ Must start with a $
▸ Scoped over the expression that
defines them
$
expression as $x
Variable definition
29. VARIABLES$
▸ Can use multiple declarations via
“Destructuring”
$
. as {a: $x, b: [$y, $z]}
Variable definitions
31. FUNCTIONS: DEFINITIONS
▸ Unit of reuse and encapsulation
▸ Introduced with the def keyword
▸ Can take arguments
▸ . is an implicit arg!
def name[(args)]: body;
Function definition:
def increment: . + 1; def map(f): [.[] | f];
Examples:
Optional!
f(x)
40. MODULES
▸ Break larger scripts into files
▸ Reuse functions across sessions, modules, etc.
▸ Searches "~/.jq", "$ORIGIN/../lib", ... for *.jq files
import MyModule as MY_MODULE;
Relative path string. Could be at ~/.jq/MyModule.jq
. | MY_MODULE::my_function
Imported prefix
...
41. ESCAPE
@text Just calls tostring
@json Serializes input as JSON
@html Applies HTML/XML escaping
@uri Applies percent encoding
@csv Rendered as CSV with double quotes
@tsv Rendered as TSV (tab-separated values)
@sh Escaped suitable for use in a POSIX shell
@base64 Converted to base64 as specified by RFC 4648
@⋯ | jq @name
Apply formatter
a.k.a String Sauce
43. STUFF
OTHER
▸ Dates
▸ Control flow
▸ Assignment
▸ Generators
▸ Parsers
▸ Streaming
▸ I/O
(that I don’t have time for) if, while,
--stream,
input,
|=,
etc.
53. GOTCHA: Shell Quoting
>jq ⋯|⋯|⋯|⋯
Shell interprets these as pipes!
⬇
>jq ‘⋯|⋯|⋯|⋯’
Add single quotes FTW!
PROBLEM:
SOLUTION:
Prefer single
over double
54. GOTCHA: Property vs Call
>jq ‘x.y’
Expects x to be function!
⬇
>jq ‘.x.y’
Add a dot in front to access property
PROBLEM:
SOLUTION:
jq: error: x/0 is not defined
at <top-level>, line 1:
55. GOTCHA: Property w/ ‘-’
>jq ‘.file-id’
Thinks ‘-’ is subtraction!
⬇
>jq ‘.[“file-id”]’
Need to use array-style access
PROBLEM:
SOLUTION:
jq: error: id/0 is not defined
at <top-level>, line 1:
56. GOTCHA: Recurse
>jq ‘..a’
Won’t work!
⬇
>jq ‘.. | a’
Add a pipe instead
PROBLEM:
SOLUTION:
jq: error: syntax error, unexpected
IDENT, expecting $end (Unix shell
quoting issues?) at <top-level>,
line 1:
58. Java: jq-jacksonJava: jq-jackson
https://github.com/eiiches/jackson-jq
val query = JsonQuery.compile(".name|repeat(3)");
val record = mapper.readTree("{"name":"a"}");
List<JsonNode> result = query.apply(scope, record);
▸ 100% Java
▸ Embeddable
▸ Works with Jackson’s JsonNode
▸ Code functions in Java
59. Node: node-jqNode: node-jqhttps://github.com/sanack/node-jq
import { run } from 'node-jq'
const filter = '. | map(select(.foo > 10))'
const jsonPath = '/path/to/json'
const options = {}
run(filter, jsonPath, options).then(console.log)
▸ npm module
▸ jq wrapper (not native js)
▸ Goal was to make jq syntax
available in Atom with atom-jq.