9. Many places
curl has been around since spring 1998
MacOS (since 2001)
Windows 10 (since 2018)
Linux distros (since 1999)
BSDs, other Unixes, MS-DOS, older systems
Side-loaded - from the official site or elsewhere
Built from source
@bagder@bagder
10. Many flavors
Millions of build combinations
Thirty different dependencies in combinations
Frequent releases
Rarely the same in two places!
@bagder@bagder
11. Many protocols and options
Today we only talk about the tool named curl
curl knows many protocols
curl features 226 command line options
We stick to HTTP(S) in this presentation
Using a small subset of flags
@bagder@bagder
13. Under the hood
GET / HTTP/1.1
Host: www.example.com
Accept: */*
User-Agent: HTTP-eats-the-world/2019
HTTP/1.1 200 OK
Date: Thu, 09 Nov 2018 14:49:00 GMT
Server: my-favorite v3
Last-Modified: Tue, 13 Jun 2000 12:10:00 GMT
Content-Length: 12345
Set-Cookie: this-is-simple=yeah-really;
Content-Type: text/html
[content]
@bagder@bagder
14. Minimal behavior by default
Basic set of headers
No fancy functionality
Let users enable more when wanted
Toggle features on/off one-by-one
@bagder@bagder
15. Long and short options
All short options have a long alternative
-s can also be set with –-silent
I will use the short options today
Many options only exist as long options
@bagder@bagder
16. $ curl example.com
<html>
<body>
<div>
<h1>Example Domain</h1>
<p>This domain is established to be used for illustrative examples.
<p><a href="http://www.iana.org/domains/example">More information...</a></p>
</div>
</body>
</html>
@bagder@bagder
Plain curl HTTP GET
17. $ curl -i https://example.com/
HTTP/1.1 200 OK
date: Wed, 09 Oct 2019 11:16:06 GMT
content-type: text/html
content-length: 306
server: server/3000
<body>
<h1>Example Domain</h1>
<p>This domain is established to be used for illustrative examples.
<p><a href="http://www.iana.org/domains/example">More information...</a></p>
</body>
@bagder@bagder
GET and show me the response headers
18. $ curl https://example.com/json | jq
{
"id": "kr09ddfgbfsf",
"name": "Issues, PRs, Dashboard, Projects",
"status": "operational",
"created_at": "2017-01-31T20:01:46.638Z",
"updated_at": "2019-09-26T14:00:45.562Z",
"position": 3,
"description": "Fake JSON for an example",
"group": false,
"only_show_if_degraded": false
},
@bagder@bagder
GET and show JSON nicely
19. $ curl -I https://example.com/
HTTP/1.1 200 OK
date: Wed, 09 Oct 2019 11:16:06 GMT
content-type: text/html
content-length: 306
server: server/3000
@bagder@bagder
HEAD only shows the response headers
HEAD returns no body!
20. $ curl -I https://example.com/redirected
HTTP/1.1 302 OK
date: Wed, 09 Oct 2019 11:16:06 GMT
content-type: text/html
location: /a/separate/place
content-length: 306
server: server/3000
@bagder@bagder
Curl doesn’t follow redirects by default
HEAD returns no body!
21. $ curl -I -L https://example.com/redirected
HTTP/1.1 200 OK
date: Wed, 19 Oct 2019 11:23:06 GMT
content-type: text/html
content-length: 4676
server: server/3000
@bagder@bagder
Please follow redirects
HEAD returns no body!
22. $ curl -v https://example.com/ -o /dev/null
* Trying 93.184.216.34:443...
* Connected to example.com (93.184.216.34) port 443 (#0)
…
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: C=US; ST=California; L=Los Angeles; O=Corporation; CN=www.example.org
…
> GET / HTTP/2
> Host: example.com
> User-Agent: curl/7.66.0
> Accept: */*
>
< HTTP/2 200
< date: Wed, 09 Oct 2019 11:16:06 GMT
< content-type: text/html
< content-length: 306
< server: server/3000
@bagder@bagder
Verbose shows more from under the hood
24. $ curl -d name=Daniel https://example.com/receiver
HTTP/1.1 200 OK
date: Wed, 09 Oct 2019 11:16:06 GMT
content-type: text/html
content-length: 306
server: server/3000
<body>
<h1>Example Domain</h1>
<p>This domain is established to be used for illustrative examples.
<p><a href="http://www.iana.org/domains/example">More information...</a></p>
</body>
@bagder@bagder
POST some basic data to the remote
25. $ curl -d @file https://example.com/receiver -o saved
$ ls -l | curl -d @- https://example.com/receiver -o saved
$ ls -l | curl --data-binary @- https://example.com/receiver -o saved
$ curl --data-binary @file.json -H "Content-Type: application/json" https://example.com
@bagder@bagder
POST a file
26. $ curl -T localfile https://example.com/remote_name
HTTP/1.1 200 OK
date: Wed, 09 Oct 2019 11:16:06 GMT
content-type: text/html
content-length: 306
server: server/3000
<body>
<h1>Example Domain</h1>
<p>This domain is established to be used for illustrative examples.
<p><a href="http://www.iana.org/domains/example">More information...</a></p>
</body>
@bagder@bagder
PUT a file
27. $ curl -T localfile -X SWOOSH https://example.com/remote_name -o save
@bagder@bagder
With -X we change the method string
28. curl -X in vain leads to pain
https://xkcd.com/386/
@bagder@bagder
29. $ curl -d postdata -X POST https://example.com/remote_name -o save
$ curl -d postdata -X POST https://example.com/remote_name -o save -L
@bagder@bagder
Bad -X leads to pain
Probably not what you wanted!
30. @bagder@bagder
Bad -X
-X means use this method in all requests
-L means follow redirects
Redirects instruct the client what method to use in the
next request
-X overrides the method the server wants the client to use
Use -X if you want a different method than curl would use
32. @bagder@bagder
Cookies are name value pairs
Cookies are name=value pairs sent by servers
Sent back by clients when the URL matches
curl doesn’t know cookies unless you tell it to
curl has separate options for reading and
writing cookie files
33. $ curl -c cookiejar.txt https://example.com/
$ curl -b cookiejar.txt https://example.com/
$ cat cookiejar.txt
# Netscape HTTP Cookie File
# https://curl.haxx.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.
example.com TRUE / FALSE 1602699857 duidd182180ebab48
@bagder@bagder
Cookies!
34. $ curl -c cookiejar.txt https://example.com/login_form
$ curl -b cookiejar.txt -c cookiejar.txt https://example.com/login -d user=daniel
-d password=1234
$ curl -b cookiejar.txt -c cookiejar.txt https://example.com/profile
@bagder@bagder
Cookies in a login procedure
35. $ curl https://127.0.0.1/
$ curl -k https://127.0.0.1/
$ curl -k https://127.0.0.1/ -H "Host: example.com"
$ curl https://example.com/ --resolve example.com:443:127.0.0.1
$ curl https://example.com/ --connect-to example.com:443:host.tld:8443
@bagder@bagder
HTTPS obstacles, locally hosted named site
Causes certificate problems
Ducks for the cert problem
Doesn’t work with cookies
Works with cookies
Doesn’t work with virtual servers
Does TLS and certs
correctly
Works with cookies
Works with virtual
servers
42. SSLKEYLOGFILE
Why trust your tools when you can snoop on them
Wireshark is your friend
Tell curl or your browser to store its secrets in
SSLKEYLOGFILE
Tell Wireshark where the secrets are located
Run curl or your browser
@bagder@bagder
45. Many HTTP versions – but they look the same
HTTP/1.0 shipped in 1996
We’ve learned how HTTP headers look like
HTTP/2 came in 2015
HTTP/3 is “coming soon”
curl makes HTTP/2 and HTTP/3 headers look and work like
HTTP/1
@bagder@bagder
52. License
This presentation and its contents are
licensed under the Creative Commons
Attribution 4.0 license:
http://creativecommons.org/licenses/by/4.0/
@bagder@bagder