This document discusses HTTP logging and Varnish log formats. It provides examples of Varnish log output showing request information like IP address, request method, URL, status code, timestamp and more. It also covers Varnish log filtering techniques like grouping transactions, including/excluding tags, and using Varnishlog queries.

1. HTTP LOGGING MET VARNISHLOG
Thijs Feryn

2. 256

3. LOGS

4. LOGS

5. LOHS

8. 172.17.0.1 - - [04/May/2022:11:57:56 +0000] "GET / HTTP/1.1" 200 45

9. 172.17.0.1 - - [04/May/2022:11:57:56 +0000] "GET / HTTP/1.1" 200 45
NCSA COMMON LOH FORMOAT

10. 172.17.0.1 - - [04/May/2022:11:57:56 +0000] "GET / HTTP/1.1" 200 45
HOST IDENT AUTH
USER
DATE REQUEST STATUS BYTES
NCSA COMMON LOH FORMOAT

11. NATIONAL CENTER FOR
SUPERCOMPUTING
APPLICATIONS

12. 172.17.0.1 - - [04/May/2022:11:57:56 +0000] "GET / HTTP/1.1" 200
45 "http://localhost/test.html" "Mozilla/5.0 (Macintosh; Intel
Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/
100.0.4896.127 Safari/537.36"

13. 172.17.0.1 - - [04/May/2022:11:57:56 +0000] "GET / HTTP/1.1" 200
45 "http://localhost/test.html" "Mozilla/5.0 (Macintosh; Intel
Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/
100.0.4896.127 Safari/537.36"
NCSA COMBINED LOH FORMOAT

14. 172.17.0.1 - - [04/May/2022:11:57:56 +0000] "GET / HTTP/1.1" 200
45 "http://localhost/test.html" "Mozilla/5.0 (Macintosh; Intel
Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/
100.0.4896.127 Safari/537.36"
NCSA COMBINED LOH FORMOAT
REFERER

15. 172.17.0.1 - - [04/May/2022:11:57:56 +0000] "GET / HTTP/1.1" 200
45 "http://localhost/test.html" "Mozilla/5.0 (Macintosh; Intel
Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/
100.0.4896.127 Safari/537.36"
NCSA COMBINED LOH FORMOAT
USER-AGENT

17. CACHE

18. GEBRUKER VARNISH SERVER

19. * << Request >> 10
- Begin req 9 rxreq
- Timestamp Start: 1651671623.893211 0.000000 0.000000
- Timestamp Req: 1651671623.893211 0.000000 0.000000
- ReqStart 172.21.0.1 62448 http
- ReqMethod GET
- ReqURL /
- ReqProtocol HTTP/1.1
- ReqHeader Host: localhost
- ReqHeader User-Agent: curl/7.79.1
- ReqHeader Accept: */*
- ReqHeader X-Forwarded-For: 172.21.0.1
- VCL_call RECV
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- ExpBan 6 banned lookup
- VCL_call MISS
- VCL_return fetch
- Link bereq 11 fetch
- Timestamp Fetch: 1651671623.896113 0.002902 0.002902
- RespProtocol HTTP/1.1
- RespStatus 200
- RespReason OK
- RespHeader Content-Type: application/json; charset=utf-8
- RespHeader Content-Length: 562
- RespHeader ETag: W/"232-+/Gw91fMkC3FNffhpz0wpLeOy1o"
- RespHeader Date: Wed, 04 May 2022 13:40:23 GMT
- RespHeader X-Varnish: 10
- RespHeader Age: 0
- RespHeader Via: 1.1 varnish (Varnish/6.0)

22. YO, IK ZIN THIJS

23. IK WEUNEN IN DIKSMUIDE

25. BACHTEN DE KUPE?

26. IK ZIN DE TECH
EVANHELIST
BIE

29. 9,000,000 WEBSITES
21% VAN DE TOP 10K WEBSITES

30. IK ZIN @THIJSFERYN

33. GEBRUKER VARNISH SERVER

34. OJ CACHT EJ GIN LOHS
IP JE WEB SERVER

35. GEBRUKER VARNISH SERVER

36. VARNISH SHARED
MEMORY LOGS (VSL)

37. CIRCULAR MEMORY BUFFER

38. vsl_space
Value is: 80M [bytes] (default)
Minimum is: 1M
Maximum is: 4G
The amount of space to allocate for the VSL fifo buffer in the
VSM memory segment. If you make this too small,
varnish{ncsa|log} etc will not be able to keep up. Making it
too large just costs memory resources.
NB: This parameter will not take any effect until the child
process has been restarted.

39. varnishd -p vsl_space=150M

40. varnishd -l 150M

41. $ varnishadm param.set vsl_space 150M
$ varnishadm stop
$ varnishadm start

42. VARNISHLOG

43. VARNISHLOG

44. VARNISHLOH

45. bash: varnishloh: command not found

46. Usage: /usr/bin/varnishlog <options>
Options:
[-a] Append to file
[-A] Text output
[-b] Only display backend records
[-c] Only display client records
[-C] Caseless regular expressions
[-d] Process old log entries and exit
[-D] Daemonize
[-g <session|request|vxid|raw>] Grouping mode (default: vxid)
[-h] Usage help
[-i <taglist>] Include tags
[-I <[taglist:]regex>] Include by regex
[-k <num>] Limit transactions
[-L <limit>] Incomplete transaction limit
[-n <dir>] varnishd working directory
[-P <file>] PID file
[-q <query>] VSL query
[-r <filename>] Binary file input
[-R <limit[/duration]>] Output rate limit
[-t <seconds|off>] VSM connection timeout
[-T <seconds>] Transaction end timeout
[-v] Verbose record printing
[-V] Version
[-w <filename>] Output filename
[-x <taglist>] Exclude tags
[-X <[taglist:]regex>] Exclude by regex

47. * << Session >> 1752689
- Begin sess 0 HTTP/1
- SessOpen 172.21.0.1 55962 http 172.21.0.3 80 1651675446.691148 29
- Link req 1752690 rxreq
- SessClose REM_CLOSE 0.004
- End

48. varnishd -F -f /etc/varnish/default.vcl
-a http=:80,HTTP -a proxy=:8443,PROXY
-p feature=+http2 -s malloc,100M
VARNISH RUNTIME PARAMETERS

49. ** << Request >> 1752690
-- Begin req 1752689 rxreq
-- Timestamp Start: 1651675446.691433 0.000000 0.000000
-- Timestamp Req: 1651675446.691433 0.000000 0.000000
-- ReqStart 172.21.0.1 55962 http
-- ReqMethod GET
-- ReqURL /
-- ReqProtocol HTTP/1.1
-- ReqHeader Host: localhost
-- ReqHeader User-Agent: curl/7.79.1
-- ReqHeader Accept: */*
-- ReqHeader X-Forwarded-For: 172.21.0.1
-- VCL_call RECV
-- VCL_return hash
-- VCL_call HASH
-- VCL_return lookup
-- ExpBan 1679159 banned lookup
-- VCL_call MISS
-- VCL_return fetch
-- Link bereq 1752691 fetch
-- Timestamp Fetch: 1651675446.693975 0.002542 0.002542

50. -- RespProtocol HTTP/1.1
-- RespStatus 200
-- RespReason OK
-- RespHeader Content-Type: application/json; charset=utf-8
-- RespHeader Content-Length: 567
-- RespHeader ETag: W/"237-ulyqMICHeGkCSm+psitDJ2K3qqg"
-- RespHeader Date: Wed, 04 May 2022 14:44:06 GMT
-- RespHeader X-Varnish: 1752690
-- RespHeader Age: 0
-- RespHeader Via: 1.1 varnish (Varnish/6.0)
-- VCL_call DELIVER
-- VCL_return deliver
-- Timestamp Process: 1651675446.693981 0.002548 0.000006
-- RespHeader Accept-Ranges: bytes
-- RespHeader Connection: keep-alive
-- Timestamp Resp: 1651675446.694030 0.002597 0.000049
-- ReqAcct 73 0 73 273 567 840
-- End

51. *** << BeReq >> 1752691
--- Begin bereq 1752690 fetch
--- VCL_use boot
--- Timestamp Start: 1651675446.691561 0.000000 0.000000
--- BereqMethod GET
--- BereqURL /
--- BereqProtocol HTTP/1.1
--- BereqHeader Host: localhost
--- BereqHeader User-Agent: curl/7.79.1
--- BereqHeader Accept: */*
--- BereqHeader X-Forwarded-For: 172.21.0.1
--- BereqHeader Accept-Encoding: gzip
--- BereqHeader X-Varnish: 1752691
--- VCL_call BACKEND_FETCH
--- VCL_return fetch
--- BackendOpen 32 boot.default 172.21.0.2 8080 172.21.0.3 36102
--- BackendStart 172.21.0.2 8080
--- Timestamp Bereq: 1651675446.691704 0.000143 0.000143
--- Timestamp Beresp: 1651675446.693860 0.002299 0.002156

52. --- BerespProtocol HTTP/1.1
--- BerespStatus 200
--- BerespReason OK
--- BerespHeader Content-Type: application/json; charset=utf-8
--- BerespHeader Content-Length: 567
--- BerespHeader ETag: W/"237-ulyqMICHeGkCSm+psitDJ2K3qqg"
--- BerespHeader Date: Wed, 04 May 2022 14:44:06 GMT
--- BerespHeader Connection: keep-alive
--- BerespHeader Keep-Alive: timeout=5
--- TTL RFC 120 10 0 1651675447 1651675447 1651675446 0 0 cacheable
--- VCL_call BACKEND_RESPONSE
--- VCL_return deliver
--- Storage malloc s0
--- Fetch_Body 3 length stream
--- BackendReuse 32 boot.default
--- Timestamp BerespBody: 1651675446.693945 0.002384 0.000085
--- Length 567
--- BereqAcct 145 0 145 214 567 781
--- End

54. <esi:include src="/header" />

55. $ varnishlog -i Begin
* << BeReq >> 3
- Begin bereq 2 fetch
* << BeReq >> 5
- Begin bereq 4 fetch
* << Request >> 4
- Begin req 2 esi
* << Request >> 2
- Begin req 1 rxreq
* << Session >> 1
- Begin sess 0 HTTP/1

56. $ varnishlog -i Begin
* << BeReq >> 3
- Begin bereq 2 fetch
* << Request >> 2
- Begin req 1 rxreq
* << Session >> 1
- Begin sess 0 HTTP/1
EEN SLAGSJE SIMPELER

58. TRANSACTIES GROEPEREN
✓ VXID (DEFAULT)
✓ SESSION
✓ REQUEST
✓ RAW

59. $ varnishlog -i Begin -g session
* << Session >> 1
- Begin sess 0 HTTP/1
** << Request >> 2
-- Begin req 1 rxreq
*** << BeReq >> 3
--- Begin bereq 2 fetch
*** << Request >> 4
--- Begin req 2 esi
*4* << BeReq >> 5
-4- Begin bereq 4 fetch

60. $ varnishlog -i Begin -g request
* << Request >> 2
- Begin req 1 rxreq
** << BeReq >> 3
-- Begin bereq 2 fetch
** << Request >> 4
-- Begin req 2 esi
*** << BeReq >> 5
--- Begin bereq 4 fetch

61. $ varnishlog -i Begin -g raw
1 Begin c sess 0 HTTP/1
3 Begin b bereq 2 fetch
5 Begin b bereq 4 fetch
4 Begin c req 2 esi
2 Begin c req 1 rxreq

62. varnishlog -g request
varnishlog -g session
varnishlog -g vxid
varnishlog -g raw

63. BITJE MINDER OUTPUT

64. RATE LIMITING

65. varnishlog -R 1/5S

66. TAG FILTERING

67. varnishlog -g request -c -i "Req*" -i Timestamp

68. varnishlog -g request -b -i "BeReq*" -i Timestamp

69. * << Request >> 1659437
- Timestamp Start: 1651676801.915256 0.000000 0.000000
- Timestamp Req: 1651676801.915256 0.000000 0.000000
- ReqUnset Upgrade: h2c
- ReqUnset HTTP2-Settings: AAMAAABkAAQCAAAAAAIAAAAA
- ReqProtocol HTTP/2.0
- ReqStart 172.21.0.1 57084 http
- ReqMethod GET
- ReqURL /
- ReqProtocol HTTP/2.0
- ReqHeader Host: localhost
- ReqHeader User-Agent: curl/7.79.1
- ReqHeader Accept: */*
- ReqHeader Connection: Upgrade, HTTP2-Settings
- ReqHeader X-Forwarded-For: 172.21.0.1
- Timestamp Process: 1651676801.916362 0.001106 0.001106
- Timestamp Resp: 1651676801.916672 0.001416 0.000311
- ReqAcct 166 0 166 178 566 744

70. varnishlog -g request -c -i "Req*,Resp*" -i Timestamp

71. varnishlog -g request -c -i "Req*" -x ReqHeader

72. varnishlog -g request -i ReqUrl
-I ReqHeader:Accept-Language

73. varnishlog -g request -i ReqUrl
-i RespHeader -X "RespHeader:(X|x)-"

74. VSL QUERIES

75. <record selection criteria> <operator> <operand>

76. ReqUrl eq '/'

77. RECORD SELECTIE CRITERIA

78. {level}taglist:record-prefix[field]

79. {2+}Timestamp:Resp[2]

80. %s: %f %f %f
| | | |
| | | +- Time since last timestamp
| | +---- Time since start of work unit
| +------- Absolute time of event
+----------- Event label
TIMESTAMP FORMOAT

81. * << Request >> 11
- Timestamp Start: 1606398588.811189 0.000000 0.000000
- Timestamp Req: 1606398588.811189 0.000000 0.000000
- Timestamp Fetch: 1606398588.818399 0.007210 0.007210
- Timestamp Process: 1606398588.818432 0.007243 0.000032
- Timestamp Resp: 1606398588.818609 0.007421 0.000178

82. varnishlog -c -i ReqUrl -I Timestamp:Resp
-g request -q "{2+}Time*:Resp[2] > 2.0"

83. varnishlog -g request -q "BerespStatus == 500"

84. varnishlog -g request -q "BerespStatus >= 500"
varnishlog -g request -q "BerespStatus > 500"
varnishlog -g request -q "BerespStatus <= 500"
varnishlog -g request -q "BerespStatus < 500"

85. varnishlog -g request -q "ReqUrl ~ '^/contact'"

86. varnishlog -g request -q "not ReqHeader:Accept-Language"

87. varnishlog -g request -q "ReqUrl eq '/'"
varnishlog -g request -q "ReqUrl ne '/contact'"

88. varnishlog -c -i ReqUrl -I RespHeader:Content-Type
-i reqacct -g request
-q "RespHeader:Content-Type ~ '^image/' and
ReqAcct[5] >= 2000000"

89. %d %d %d %d %d %d
| | | | | |
| | | | | +- Total bytes transmitted
| | | | +---- Body bytes transmitted
| | | +------- Header bytes transmitted
| | +---------- Total bytes received
| +------------- Body bytes received
+---------------- Header bytes received
REQACCT FORMOAT

90. varnishlog -c -g request -i ReqUrl
-I VCL_call:PASS -I VCL_call:MISS
-q "VCL_call eq 'MISS' or VCL_call eq 'PASS'"

91. varnishlog -c -g request -i ReqUrl
-q "TTL[6] eq 'uncacheable' and (BerespHeader:Set-
Cookie or BerespHeader:Cache-Control ~ '(private|no-
cache|no-store)')"

92. DE BUFFER DUMP'N

93. varnishlog -i ReqUrl -d

94. varnishlog -g request -i requrl -d
-q "Timestamp:Start[1] >= $(date -d '10/05/2022
11:00:00' +%s.0)"

95. LEZEN/SCHRIEV'N

96. varnishlog -a -w vsl.log
varnishlog -r vsl.log
varnishlog -A -a -w vsl.log

97. VARNISHLOG

98. VARNISHLOG

99. VARNISHNCSA

100. 172.17.0.1 - - [04/May/2022:11:57:56 +0000] "GET / HTTP/1.1" 200
45 "http://localhost/test.html" "Mozilla/5.0 (Macintosh; Intel
Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/
100.0.4896.127 Safari/537.36"

101. varnishncsa -a -w /var/log/varnish/access.log

102. %h %l %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i"

103. %b Size of the response body in bytes
%D Time taken to serve the request in microseconds
%H The request protocol
%h The hostname or IP address of the remote host
%I Total bytes received
%{X}i Contents of request header X
%l Remote log name
%m Request method
%{X}o Contents of response header X
%O In client mode, total bytes sent to client. In backend mode, total bytes received from the backend
%q The query string
%r First line of the HTTP request composed using other formatters
%s HTTP status code of the response
%t Time when the request was received
%{X}t Time when request was received in the strftime time specification format
%T Time taken to serve the request
%U Request URL without query string
%u Remote authenticated user
%{X}x Extended Varnish & VCL variables

104. varnishncsa -c -b -F '%h %l %u %t "%r" %s %b "%
{Referer}i" "%{User-agent}i" "%{Varnish:side}x"'

105. 172.18.0.3 - - [02/Dec/2020:10:18:38 +0000] "GET http://localhost/
HTTP/1.1" 200 6 "-" "curl" "b"
127.0.0.1 - - [02/Dec/2020:10:18:38 +0000] "GET http://localhost/
HTTP/1.1" 200 6 "-" "curl" "c"

106. varnishncsa -F '%U %{Varnish:handling}x %{Varnish:side}x' -b -c
/test - b
/test miss c
/test hit c
/test - b
/test pass c

107. varnishncsa -b -F '%{VSL:Timestamp:BerespBody[2]}x %
{Host}i %U%q' -q 'Timestamp:BerespBody[2] > 1.0'
2.007142 localhost /

108. varnishncsa -a -D -w /var/log/varnish/access.log
-P /var/run/varnishncsa.pid
VARNISHNCSA DAEMONIZEN

109. [Unit]
Description=Varnish Cache HTTP accelerator NCSA logging daemon
After=varnish.service
[Service]
RuntimeDirectory=varnishncsa
Type=forking
User=varnishlog
Group=varnish
ExecStart=/usr/bin/varnishncsa -a -w /var/log/varnish/
varnishncsa.log -D
ExecReload=/bin/kill -HUP $MAINPID
[Install]
WantedBy=multi-user.target
SYSTEMD

110. /var/log/varnish/varnishncsa.log {
daily
rotate 7
compress
delaycompress
missingok
postrotate
systemctl -q is-active varnishncsa.service || exit 0
systemctl reload varnishncsa.service
endscript
}
LOH ROTOASJE

111. /var/log/varnish/varnishncsa.log
/var/log/varnish/varnishncsa.log.1
/var/log/varnish/varnishncsa.log.2.gz
/var/log/varnish/varnishncsa.log.3.gz
/var/log/varnish/varnishncsa.log.4.gz
/var/log/varnish/varnishncsa.log.5.gz
/var/log/varnish/varnishncsa.log.6.gz
/var/log/varnish/varnishncsa.log.7.gz
LOH ROTOASJE

112. VARNISHTOP

113. varnishtop -i requrl
list length 3
31.67 ReqURL /contact
24.33 ReqURL /products
8.50 ReqURL /

114. varnishtop -i ReqUrl -q "VCL_call eq 'MISS'"
varnishtop -i ReqUrl -q "VCL_return eq 'pass'"
varnishtop -i BeReqUrl -q "TTL[6] eq 'uncacheable'"
varnishtop -i BeReqUrl -q "Timestamp:Beresp[3] > 2.0"

115. varnishtop -i RespProtocol
list length 2
310.29 RespProtocol HTTP/1.1
19.56 RespProtocol HTTP/2.0

116. varnishtop -I Begin:sess
list length 2
234.25 Begin sess 0 HTTP/1
49.89 Begin sess 0 PROXY

117. VSL TAHS IP MOATE

118. VARNISH CONFIGURATION LANGUAGE

119. STRING std.log(STRING s)

120. vcl 4.1;
import std;
import vsthrottle;
backend default {
.host = "localhost";
.port = "8080";
}
sub vcl_recv {
if (vsthrottle.is_denied(client.identity, 15, 10s, 30s)) {
std.log("Ratelimit: " + client.identity +
" blocked for another " + vsthrottle.blocked(client.identity, 15, 10s, 30s)
+ " seconds");
return (synth(429, "Too Many Requests"));
}
}

121. vcl 4.1;
import std;
import vsthrottle;
backend default {
.host = "localhost";
.port = "8080";
}
sub vcl_recv {
if (vsthrottle.is_denied(client.identity, 15, 10s, 30s)) {
std.log("Ratelimit: " + client.identity +
" blocked for another " + vsthrottle.blocked(client.identity, 15, 10s, 30s)
+ " seconds");
return (synth(429, "Too Many Requests"));
}
}

122. varnishlog -g request -i requrl -I VCL_Log:Ratelimit
* << Request >> 557828
- ReqURL /
- VCL_Log Ratelimit: 172.24.0.1 blocked for
another 29.151 seconds

123. DEMOOTJE?