Slides from the eZ Global Partner Conference 2012 (LISBON, PORTUGAL)
Advanced & spicy details about the eZ Publish Caching mechanisms : ini cache, compiled templates, viewcache, template-block caches...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
eZ Publish Caching Mechanisms
1. eZ Partner Conference in Lisbon. Feb 9th
eZ Publish Caching
Mechanisms
// SPICY DETAILS
Gilles Guirand
CTO Kaliop / platinum partner
eZ Publish board member
@gandbox
4. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
CONTEXT
Common questions ( high trafic ) :
➔Is it possible to never expire the ENTIRE viewcache ?
… and which actions actually expire the ENTIRE viewcache ?
➔ Is it possible to reduce the I/O (template-blocks) ?
➔ Is it possible to expire a specific set of viewcache ?
➔ Is it possible to expire a specific set ot template-block
files ?
➔ Is there a best practice to PUSH a daily templates update ?
4 / 58
5. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
CONTEXT
Final questions :
➔ How all these cache mechanisms really work ?
… I mean in details... algorithms, I/O, triggers, exceptions,
undocumented features & settings, unknown bugs, unknown
fixes...
Final solutions :
➔ Don't trust the documentation : READ THE KERNEL / TEST THE
KERNEL... ( be brave )
5 / 58
6. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
CONTEXT
Knowledge exploitation / reuse :
➔ Write a FR document (46 pages) : mix between a
documentation & tuto
➔ Publish it on differents supports (FR + EN)
➔ Split the article into 4 parts :
1 2 3 4
INI VIEWCACHE Template- expiry.php
Tpl compilation block ezcache.php
bench
6 / 58
7. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
CONTEXT
Publication schedule :
FEB MARCH APRIL MAY JUNE
1 2 3 4
FR
1 2 3 4
gandbox.fr FR
share.ez.no EN 1 2 3 4
➔ (depending of translation delay)
7 / 58
9. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
INI CACHE
When eZ generates the INI cache ?
➔ On the fly (if the cache file does not exist) – from an API
call (ezini PHP class), a template call (ezini operator)
➔ By default, eZ automatically check if an INI file has been
changed or not, depending of this setting (config.php)
//define( 'EZP_INI_FILEMTIME_CHECK', false ) ;
/* Set EZP_INI_FILEMTIME_CHECK constant to false to improve performance
( 10% ~ 15% on I/O ) by not checking modified time on ini files. You can also set
it to a string, the name of a ini file you still want to check modified time on, best
example would be to set it to 'site.ini' to make the system still check that but
not the rest. */
9 / 58
10. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
INI CACHE
How eZ stores the INI cache ?
➔ var/cache/ini ( and not /{varDir}/ini ), not clusterized
➔{inifilename}-{hash}.php
design-677247a49e428aa083741 1b52777b920.php
design-704216b4d0e3ea68c09742504bb366c8.php
design-7333039e7e19aa41 1f8dd01836555861.php
md5
FileName / RootDir / DirectAccess / overrideDirs /
internalCharset
10 / 58
11. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
INI CACHE
How to expire the INI cache ?
➔php bin/php/ezcache.php --clear-id=global_ini
➔ Remove var/cache/ini
➔php bin/php/ezcache.php --clear-tag=ini
➔ Remove var/cache/ini &
var/cache/active_extensions_{hash}.php
➔php bin/php/ezcache.php --clear-id=ini
➔ Nothing (bug) : try to recursively delete INI files in
{varDir}/ini
11 / 58
12. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
INI CACHE
How to expire a set of INI files ?
➔ Not possible with eZ Scripts
➔ Possible & safe using a command line
➔ TEST BEFORE : find var/cache/ini -name "site-*"
➔ RM : find var/cache/ini -name "site-*" | xargs rm
12 / 58
13. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
INI CACHE
How to not compile the INI value ?
lib/ezutils/classes/ezini.php
function eZINI( $fileName = 'site.ini', $rootDir = '', $useTextCodec = null,
$useCache = null, $useLocalOverrides = null, $directAccess = false,
$addArrayDefinition = false, $load = true )
{
...
}
Dynamically load the INI values inside your compiled
templates :
site.ini [eZINISettings] / DynamicTemplateMode=enabled
OR
Ezini operator → 'dynamic' parameter
13 / 58
15. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
TPL COMPILATION
What for ?
➔ Compile your .tpl files to .php files
➔ Reduce the RAM / CPU (use the stored compiled
version)
How to set ? site.ini [templatesettings]
➔ TemplateCompile=enabled
➔ Store compiled version {varDir}/cache/template/compiled/
➔ NodeTreeCaching=enabled ( disabled by default )
➔ Store pre-compiled version {varDir}/cache/template/tree/
Common mistake
➔ TemplateCache=enabled → template-block
15 / 58
16. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
TPL COMPILATION
DESIGN {varDir}/cache/template/tree {varDir}/cache/template/compiled
page_mainarea.tpl {hash}-page_mainarea.php page_mainarea-{hash1}.php
NodeTreeCaching=enabled
page_mainarea-{hash2}.php
( disabled by default )
page_mainarea-{hash3}.php
Why eZ generates several
compiled files for the same
.tpl?
16 / 58
17. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
TPL COMPILATION
How eZ stores the final compiled files ?
{VarDir}/cache/template/compiled/, not clusterized
Filenames : {templatename}-{hash}.php
page_head-49f4458d0b68aec3cf2de63a6918fd61.php
page_head-ec3cf2de63a6918fd6149f4458d0b68a.php
17 / 58
18. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
TPL COMPILATION
How eZ calculates the hash ?
$sharedTemplates = false
{VarDir}/cache/template/compiled/
Filenames : {templatename}-{hash}.php
pagelayout-49f4458d0b68aec3cf2de63a6918fd61.php
Md5( filepath ) / internalCharset / language /
useFullUrlText / accessText / pageLayoutVariable /
indexFile / extraName
/layout/set/...
18 / 58
19. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
TPL COMPILATION
How eZ calculate the hash ?
$sharedTemplates = true
{VarDir}/cache/template/compiled/
Filenames : {templatename}-{hash}.php
pagelayout-aec3cf2de63a6918fd6149f4458d0b68.php
Md5( filepath ) / language
No more siteaccess inside the key...
Ezini operator → 'dynamic' parameter
19 / 58
20. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
TPL COMPILATION
How to pre-compile a set of template
files (to avoid : on the fly generating) ?
➔ Force the compilation of 2 template files
php bin/php/eztc.php -s mysiteaccess --force
extension/ezwebin/design/ezwebin/templates/pagelayout.tpl
extension/ezwebin/design/ezwebin/templates/page_leftmenu.tpl
➔ Force the compilation of all template files starting with
« page_header_ » :
find extension/ezwebin/ -name "page_header_*.tpl" | xargs php
bin/php/eztc.php -s mysiteaccess –force
➔ Force the compilation of all template files from a TXT file :
cat template_list.txt | xargs php bin/php/eztc.php -s mysiteaccess
--force
20 / 58
21. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
TPL COMPILATION
The **** layout module
➔ By default ( $shareTemplates = false ), the 'layout' value is used by
the final compiled filename
➔ So if you are using this module with 3 other layouts, you'll produce 4
compiled files (instead of 1)
➔ eztc.php script does not manage the layout parameter
Your 3 additional compiled files ( for each layout )
will be procude « on the fly », and cannot be pre-
compiled with eztc.php
➔ Suggestion : improve eztc.php, add the –layout=... parameter
21 / 58
26. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
VIEWCACHE
How to know the exact matching
between a viewcache file & my URLs ?
{nodeid}-{hash}.php → deduce the nodeID
At the end of file :
"view_parameters";a:6:
{s:6:"offset";b:0;s:4:"year";b:0;s:5:"month";b:0;s:3:"day";b:0;s:10:"n
amefilter";b:0;s:4:"test";s:1:"2";}...
Should be URL : /(test)/2
26 / 58
27. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
VIEWCACHE
How eZ expires a set of viewcache ?
Alteration of cache states - Depending of the filehandler ( file.ini )
➔ Ezfsfilehandler : UNLINK file
➔ Ezfs2filehandler : TOUCH file (25/25/1977) ← Star Wars day
➔ Ezdbfilehandler : "ezdbfile" (UPDATE expired=1)
➔ Ezdfsfilehandler : "ezdfsfile" (UPDATE expired=1)
Comparison of cache states - Depending of the filehandler ( file.ini )
➔ Ezfsfilehandler : file does not exist, or is older than the content-view-
cache timestamp
➔ Ezfs2filehandler : file is older than the content-view-cache timestamp
➔ Ezdbfilehandler : "ezdbfile" (expired=1)
➔ Ezdfsfilehandler : "ezdfsfile" (expired=1)
27 / 58
28. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
VIEWCACHE
How eZ expires the ENTIRE viewcache ?
➔ eZ does not recursively update / delete the files (too many files...
unsafe)
eZ set a global timestamp expiry value :
➔
{VarDir}/cache/expiry.php ← 'content-view-cache' ( current timestamp)
Each time the expiry.php is updated, the ENTIRE viewcache
will expire.
➔ Ok, but my ENTIRE viewcache expires all the
time... i did nothing special ! Don't know why !
28 / 58
29. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
VIEWCACHE
Which event / trigger expires a set of
cache ?
By default : on object publication (or back-end feature)
➔ the related nodes & parent node
Willingly, using a script :
➔ bin/php/ezcontentcache.php –clear-node=2
➔ bin/php/ezcontentcache.php –clear-node=2,46,63
➔ bin/php/ezcontentcache.php –clear-node=/company/about
➔ bin/php/ezcontentcache.php –clear-subtree=/company/about
➔ bin/php/ezcontentcache.php –clear-
subtree=/company/about,/news
29 / 58
30. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
VIEWCACHE
Smart viewcache & cacheThreshold : a
set of cache → the ENTIRE view cache
Automatically, if using the smart view cache / viewcache.ini :
➔ The smart view cache populates a list of related « node_id »
viewcache files to expire, depending of your settings. If this node_id list
is larger than the CacheThreshold limit value (250 by default) → eZ
expires the ENTIRE viewcache !
➔ Some settings could increase the node_id list count & excede the
CacheThreshold limit : ALL, siblings, KeywordNodesCacheClearLimit
➔ Advice : Set KeywordNodesCacheClearLimit = 0, never use ALL, avoid
siblings, log out the CacheThreshold (ezpEvent)
30 / 58
31. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
VIEWCACHE
Log the viewcache node_id list to expire,
prevent the CacheThreshold limit...
➔ Add an ezpEvent : site.ini.append.php
[Event]
Listeners[]=content/cache@test::logviewcache
# test = your PHP class name
# logviewcache = the static method
31 / 58
32. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
VIEWCACHE
Log the viewcache node_id list to expire,
prevent the CacheThreshold limit...
➔ Create a PHP class / generate autoloads
<?php
class test
{
static public function logviewcache( $nodeList )
{
$uri = $GLOBALS['_SERVER']['REQUEST_URI'];
eZLog::write('node_id count : '. count( $nodeList ). ' / node_id list : ' . implode(', ', $nodeList ) . ' / URI : '.$uri,
'chachethresold.log');
return $nodeList;
}}
?>
➔ Log output :
[ Feb 06 2012 17:48:34 ] node_id count : 5 / node_id list : 1 105, 2, 1, 78
16,
/ URI : /content/edit/114/2/fre-FR
32 / 58
33. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
VIEWCACHE
My ENTIRE viewcache still expire...
Maybe you invoked one of these features (back-end, API) :
➔ Create a new content class ( BUG ? )
➔ Update / delete an existing content class ( or attributes )
➔ Update / delete and existing role ( & policies ) OR role assignement
➔ Section assignement
➔ Update an ezshop rule ( discount rules, currency... )
33 / 58
37. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
CACHE-BLOCK
... What for ?
➔ Template-block cache = cache files
related to cache-block, also called
« template cache » ( common mistake with
the template compilation )
➔ Template-block cache is an HTML (or
other) cache of a template part ( cache-
block )
← Illustration : Łukasz Serwatka
http://share.ez.no/learn/ez-publish/ez-publish-performance-
optimization-part-3-of-3-practical-cache-and-template-solutions/
(page)/2
37 / 58
38. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
CACHE-BLOCK
Impact of common parameters
The « keys » parameter :
Please AVOID :
{cache-block keys=$uri_string} 1 template-block file
... for each URI !!!
{/cache-block}
Use instead :
{cache-block keys=$my_custom_limited_range_key}
...
{/cache-block}
Only possible states of your content
inside your block
38 / 58
39. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
CACHE-BLOCK
Impact of common parameters
The « URI key » parameter : stupid but signifiant simulation
➔ A classic Web site, with 1000 nodes, news folder ( year, month
archives ), products catalog ( filters ), calendar, forum...
➔ Could generate 10 000 « indexable » several URI ( google bot )
➔ A cache-block ( only 1 URI key ) to cache the global menu, who is
generating 300 SQL queries
➔ No « expiry » or « ignore_content_expiry » parameters : so the cache-
block expires every 2 hours (or on content publication)
➔ So... 300 SQL queries * 12 expiration per day * 10 000 URI = 36 millions of
SQL queries per day...
39 / 58
40. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
CACHE-BLOCK
Impact of common parameters
The « expiry » parameter :
If possible, always set a custom expiry value & ignore content
expiry :
{cache-block ignore_content_expiry expiry=86400}
...
{/cache-block}
40 / 58
41. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
CACHE-BLOCK
Impact of common parameters
The « sutree_expiry » parameter : no more expiry /
ignore_content_expiry needed
{cache-block subtree_expiry='news/'}
...
{/cache-block}
{cache-block subtree_expiry=142}
...
{/cache-block}
41 / 58
42. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
CACHE-BLOCK
Impact of common parameters
The « sutree_expiry » parameter : tip of the « magic » node
➔ To expire a cache-block, depending of a custom rule ( per content
class, per PHP algorithm ... )
➔ Create a {cache-block subtree_expiry=$magic_node_id}
➔ Create a node somewhere ( safe location, without FULL view )
➔ Create a workflow or customedithandler to include your custom rules
on publication, and update the content object related to the magic
node by the code
42 / 58
43. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
CACHE-BLOCK
Cache-block imbrication
{cache-block expiry=3600}
...
{cache-block ignore_content_expiry expiry=86400}
... {* you save … SQL Queries each hours *}
{/cache-block}
...
{cache-block subtree_expiry='news/'}
...
{/cache-block}
...
{/cache-block}
43 / 58
44. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
CACHE-BLOCK
I need a « real time » block ( TTL=1m )
A cache-block may not be the solution, use instead :
➔ ESI ( Edge Side Include ) : better for front-end loading &
google bot indexing. But reverse-proxy required ( varnish,
squid, akamai... )
➔ AJAX : easier to implement
44 / 58
45. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
CACHE-BLOCK
How eZ stores the template-block cache
{VarDir}/cache/template-block/1/2/3/12345678.cache
{VarDir}/cache/template-block/5/5/5/555666888.cache
{VarDir}/cache/template-block/subtree/1/5/9/cache/1/2/3/12345678.cache
subtree_expiry=159 CRC32
Complex... The
pathname doesn't help
45 / 58
46. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
CACHE-BLOCK
How eZ stores the template-block cache
{VarDir}/cache/template-block/1/2/3/12345678.cache
CRC32 ( 32-bit polynomial checksum )
Cache-block position in the .tpl file / template path /
keys / siteaccess
78_1_78_94_extension/ezwebin/design/ezwebin/templates/pagelayout.tpl
78 = {cache-block... } line start
1 = {cache-block... } column start (a simple ' ' before...)
78 = {cache-block... } line end
94 = {cache-block... } column end
46 / 58
47. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
CACHE-BLOCK
How to expire the ALL of the template-
block cache files ?
php bin/php/ezcache.php –clear-id=template-block
➔ Expires all the template-block files, by updating the global
cache-block expiry time ( global-template-block-cache )
in the {varDir}/expiry.php file
47 / 58
48. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
CACHE-BLOCK
How to expire a set of specific template-
block cache files ?
➔ Not really possible... exept by searching a specific string
inside each files, and deleting matching files ( ezfs /
ezFS2 )
➔ find var/myvardir/cache/template-block/ -mmin -10 -name "*.cache"
-exec grep -iHl "ID:BLOCK1" {} ; | xargs rm
{cache-block ignore_content_expiry expiry=0}
<!-- ID:BLOCK1 -->
...
{/cache-block}
48 / 58
55. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
EZCACHE.PHP SCRIPT
MAIN ALGORITHM
Hardcoded in kernel/classes/ezcache.php.
For each ID :
'id' => ID
'tag' => array( TAG LIST )
'enabled' => Cache enabled or not
'is-clustered' => Use the current filehandler or not ?
'path' => Path to purge, if purging (is-clustered=false)
'expiry-key' => Key to set inside the expiry.php file if needed
'function' => Function to apply only on clearing (not purging)
'function-purge' => Function to apply only on purging (not
clearing)
55 / 58
56. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
EZCACHE.PHP SCRIPT
MAIN ALGORITHM
Add your own ID for your extensions ( site.ini.append.php )
# Cache item entry (for eZ Publish 4.3 and up)
[Cache]
CacheItems[]=ezjscore
[Cache_ezjscore]
name=eZJSCore Public Packer cache
id=ezjscore-packer
tags[]=content
tags[]=template
path=public
isClustered=true
56 / 58
57. :~$ eZ Publish Caching Mechanisms // SPICY DETAILS
EZCACHE.PHP SCRIPT
A BUG HELPS TO UNDERSTAND
php bin/php/ezcache.php --clear-id=ini
array( 'name' => ezpI18n::tr( 'kernel/cache', 'INI cache' ),
'id' => 'ini',
'tag' => array( 'ini' ),
'enabled' => true,
'path' => 'ini' ),
Why is it not working ?
57 / 58