SlideShare ist ein Scribd-Unternehmen logo
1 von 22
upfront diet for plone
Roché Compaan
About me
● Owner of Upfront Systems
● Founded Upfront Systems in 1998
● Zope developer for the past 10 years
Experience
● Largest Plone deployments to date: 5FM & MetroFM
● 5FM
– 20 million page views per month
– 70 thousand members
– 6 million objects in ZODB
● Metro FM
– 15 million page views per month
– 20 thousand members
– 2 million objects in ZODB
● Helping Connexions with performance
● http://www.markhorner.net/2010/04/29/connexions-authoring-
performance/
Can Plone scale to 10 million documents?
ZODB Benchmarks
● Most common myth spread: “ZODB is a high
read, (s)low write database”
● No published ZODB benchmarks
● Wrote some benchmarks that test ZODB
read/write speed in 2008
● collective.zodbbench
ZODB Write Speed
0 500 thousand 2,5 million 5 million 7,5 million 8,5 million 10 million
0
1000
2000
3000
4000
5000
6000
7000
8000
Postgres
ZODB
Number of records
Inserts/second
ZODB Lookup Speed
Number of Objects Lookup Time
100,000 0.003 ms
1,000,000 0.006 ms
10,000,000 2 ms
Benchmark Conclusions
● Thanks Jim, ZODB Rocks!
● Large cache size important for writes too, not
only reads
● Need a solution for more than 10 million objects
● If the ZODB can handle 250 inserts/second at
peak volume why does it take a full minute to
add 250 ATDocuments?
A closer look at Plone
● Added 10,000 documents to Plone, while
profiling with Dieter Maurer's ZopeProfiler
● Timed “apply_index” method to find indexes
performing badly with and without
experimental.catalogqueryplan
● Batch import of content with an without
references.
● Used fsdump to inspect pickle contents.
Plone Benchmark Conclusions
● portal_catalog remains a bottleneck to scaling a
Plone site.
● 1 catalog == many many conflicts
● reference_catalog is a major performance
bottleneck
● Indexing 10,000 documents shows that 90% of
the objects in Data.fs relate to indexing.
What's in a pickle?
2,51755
81,564
9,872
20,024
85
655
1,544
30,455
182,708
7798,088122
Pickle contents of a single transaction adding an ATDocument
object size in bytes
IIBTree
Folder
IOBucket
OIBucket
IIBucket
IOBTree
BTrees.Length
ATDocument
OOBTree
IISet
Archetypes.BaseUnit
OOBucket
IITreeSet
Think before you index!
“Waking up objects is expensive” - this is an
irrational fear. And adding another catalog in
response to this fear requires therapy.
The portal_catalog problem
● No navigation, portal tabs, folder contents, review
queue if your content type is not in the portal_catalog.
● Your content type is automatically subscribed to 20+
indexes in the portal_catalog. Bad for new developers
with hopes of building an enterprise CMS.
● Unless you have 5+ years experience with Zope – it's
not so easy to “opt out”
● portal_catalog should only be used for content
searches – no functionality must depend on it.
What is upfront.diet?
● It's a healthier and happier approach to Plone
development: index only when you really really need
to.
● Set of packages that re-implement plone functionality
without depending on the portal_catalog and
reference_catalog.
● Includes replacements for: portal tabs, navigation
portlet, folder contents, archetype references.
● Provides a catalog blacklist/whitelist to customise
indexing per content type.
More about upfront.diet
● upfront.navportlet
● Separate catalog for navportlet
● upfront.simplereferencefield
●
Store references directly on object using annotations
● still depends uid_catalog
● upfront.foldercontents
● Adapter for IFolderish
● Index allowedRolesAndUsers directly on folder – less conflicts
● upfront.catalogblacklist
● Generic Setup profile with a whitelist and blacklist of indexes per
content type
● patches IndexableObjectWrapper, ugh!
upfront.diet todo list
● Packages:
● Portal tabs (done but not a separate package yet)
● Sitemap
● Review queue
● News and Events Smart Folders
● Site Search
● References without uid_catalog
● Comparative benchmarks with and without it
upfront.diet deployments
● Very few at the moment but looks promising
● Fullmarks.org.za (half a million objects, Data.fs
< 1GB)
Contact Me
● Email: roche@upfrontsystems.co.za
● Blog:
http://upfrontsystems.co.za/Members/roche
● facebook.com/rochecompaan
● twitter.com/rochecompaan
Questions?

Weitere ähnliche Inhalte

Ähnlich wie upfront diet plone conf 2010

G.Bs Presentation Of Guru Nanak Univ. National Conf.2009
G.Bs Presentation Of Guru Nanak Univ. National Conf.2009G.Bs Presentation Of Guru Nanak Univ. National Conf.2009
G.Bs Presentation Of Guru Nanak Univ. National Conf.2009Goutam Biswas
 
10 clues showing that you are doing OSGi in the wrong manner - Jerome Moliere
10 clues showing that you are doing OSGi in the wrong manner - Jerome Moliere10 clues showing that you are doing OSGi in the wrong manner - Jerome Moliere
10 clues showing that you are doing OSGi in the wrong manner - Jerome Molieremfrancis
 
Github for Serious Business Professional
Github for Serious Business ProfessionalGithub for Serious Business Professional
Github for Serious Business Professionalzwheller
 
We’re Going Mobile! Great! Wait… What Does That Mean?
We’re Going Mobile! Great! Wait… What Does That Mean?We’re Going Mobile! Great! Wait… What Does That Mean?
We’re Going Mobile! Great! Wait… What Does That Mean?STC-Philadelphia Metro Chapter
 
Using Joomla, Zoo & SOLR to power Asia's Largest Auction House
Using Joomla, Zoo & SOLR to power Asia's Largest Auction HouseUsing Joomla, Zoo & SOLR to power Asia's Largest Auction House
Using Joomla, Zoo & SOLR to power Asia's Largest Auction HouseParth Lawate
 
Yoast SEO for TYPO3 - TYPO3 Developer Days 2017
Yoast SEO for TYPO3 - TYPO3 Developer Days 2017Yoast SEO for TYPO3 - TYPO3 Developer Days 2017
Yoast SEO for TYPO3 - TYPO3 Developer Days 2017Richard Haeser
 
2022 DOI SKILup Days_Google Uses Monorepo, and I Don't - Here's Why.pptx
2022 DOI SKILup Days_Google Uses Monorepo, and I Don't - Here's Why.pptx2022 DOI SKILup Days_Google Uses Monorepo, and I Don't - Here's Why.pptx
2022 DOI SKILup Days_Google Uses Monorepo, and I Don't - Here's Why.pptxTurja Narayan Chaudhuri
 
Fighting legacy with hexagonal architecture and frameworkless php
Fighting legacy with hexagonal architecture and frameworkless phpFighting legacy with hexagonal architecture and frameworkless php
Fighting legacy with hexagonal architecture and frameworkless phpFabio Pellegrini
 
Functional testing with behat
Functional testing with behatFunctional testing with behat
Functional testing with behatTahmina Khatoon
 
Practical automation for beginners
Practical automation for beginnersPractical automation for beginners
Practical automation for beginnersSeoweon Yoo
 
SharePoint Migration Preparation
SharePoint Migration PreparationSharePoint Migration Preparation
SharePoint Migration PreparationSysKit Ltd
 
Git Going w/ Git
Git Going w/ GitGit Going w/ Git
Git Going w/ GitheyMP
 
Building multi billion ( dollars, users, documents ) search engines on open ...
Building multi billion ( dollars, users, documents ) search engines  on open ...Building multi billion ( dollars, users, documents ) search engines  on open ...
Building multi billion ( dollars, users, documents ) search engines on open ...Andrei Lopatenko
 
Neil Perlin - We're Going Mobile! Great! Are We Ready?
Neil Perlin - We're Going Mobile! Great! Are We Ready?Neil Perlin - We're Going Mobile! Great! Are We Ready?
Neil Perlin - We're Going Mobile! Great! Are We Ready?LavaConConference
 
SEO for Large/Enterprise Websites - Data & Tech Side
SEO for Large/Enterprise Websites - Data & Tech SideSEO for Large/Enterprise Websites - Data & Tech Side
SEO for Large/Enterprise Websites - Data & Tech SideDominic Woodman
 
Easy joget v3 for the absolute beginner
Easy joget v3 for the absolute beginnerEasy joget v3 for the absolute beginner
Easy joget v3 for the absolute beginnermadengly
 
OpenERP R&D
OpenERP R&DOpenERP R&D
OpenERP R&DOdoo
 
MongoDB@sfr.fr
MongoDB@sfr.frMongoDB@sfr.fr
MongoDB@sfr.frbeboutou
 

Ähnlich wie upfront diet plone conf 2010 (20)

Bi Publisher 11g: Only good news
Bi Publisher 11g: Only good newsBi Publisher 11g: Only good news
Bi Publisher 11g: Only good news
 
G.Bs Presentation Of Guru Nanak Univ. National Conf.2009
G.Bs Presentation Of Guru Nanak Univ. National Conf.2009G.Bs Presentation Of Guru Nanak Univ. National Conf.2009
G.Bs Presentation Of Guru Nanak Univ. National Conf.2009
 
10 clues showing that you are doing OSGi in the wrong manner - Jerome Moliere
10 clues showing that you are doing OSGi in the wrong manner - Jerome Moliere10 clues showing that you are doing OSGi in the wrong manner - Jerome Moliere
10 clues showing that you are doing OSGi in the wrong manner - Jerome Moliere
 
Github for Serious Business Professional
Github for Serious Business ProfessionalGithub for Serious Business Professional
Github for Serious Business Professional
 
We’re Going Mobile! Great! Wait… What Does That Mean?
We’re Going Mobile! Great! Wait… What Does That Mean?We’re Going Mobile! Great! Wait… What Does That Mean?
We’re Going Mobile! Great! Wait… What Does That Mean?
 
Using Joomla, Zoo & SOLR to power Asia's Largest Auction House
Using Joomla, Zoo & SOLR to power Asia's Largest Auction HouseUsing Joomla, Zoo & SOLR to power Asia's Largest Auction House
Using Joomla, Zoo & SOLR to power Asia's Largest Auction House
 
Yoast SEO for TYPO3 - TYPO3 Developer Days 2017
Yoast SEO for TYPO3 - TYPO3 Developer Days 2017Yoast SEO for TYPO3 - TYPO3 Developer Days 2017
Yoast SEO for TYPO3 - TYPO3 Developer Days 2017
 
2022 DOI SKILup Days_Google Uses Monorepo, and I Don't - Here's Why.pptx
2022 DOI SKILup Days_Google Uses Monorepo, and I Don't - Here's Why.pptx2022 DOI SKILup Days_Google Uses Monorepo, and I Don't - Here's Why.pptx
2022 DOI SKILup Days_Google Uses Monorepo, and I Don't - Here's Why.pptx
 
Tori ORM for PyCon 2014
Tori ORM for PyCon 2014Tori ORM for PyCon 2014
Tori ORM for PyCon 2014
 
Fighting legacy with hexagonal architecture and frameworkless php
Fighting legacy with hexagonal architecture and frameworkless phpFighting legacy with hexagonal architecture and frameworkless php
Fighting legacy with hexagonal architecture and frameworkless php
 
Functional testing with behat
Functional testing with behatFunctional testing with behat
Functional testing with behat
 
Practical automation for beginners
Practical automation for beginnersPractical automation for beginners
Practical automation for beginners
 
SharePoint Migration Preparation
SharePoint Migration PreparationSharePoint Migration Preparation
SharePoint Migration Preparation
 
Git Going w/ Git
Git Going w/ GitGit Going w/ Git
Git Going w/ Git
 
Building multi billion ( dollars, users, documents ) search engines on open ...
Building multi billion ( dollars, users, documents ) search engines  on open ...Building multi billion ( dollars, users, documents ) search engines  on open ...
Building multi billion ( dollars, users, documents ) search engines on open ...
 
Neil Perlin - We're Going Mobile! Great! Are We Ready?
Neil Perlin - We're Going Mobile! Great! Are We Ready?Neil Perlin - We're Going Mobile! Great! Are We Ready?
Neil Perlin - We're Going Mobile! Great! Are We Ready?
 
SEO for Large/Enterprise Websites - Data & Tech Side
SEO for Large/Enterprise Websites - Data & Tech SideSEO for Large/Enterprise Websites - Data & Tech Side
SEO for Large/Enterprise Websites - Data & Tech Side
 
Easy joget v3 for the absolute beginner
Easy joget v3 for the absolute beginnerEasy joget v3 for the absolute beginner
Easy joget v3 for the absolute beginner
 
OpenERP R&D
OpenERP R&DOpenERP R&D
OpenERP R&D
 
MongoDB@sfr.fr
MongoDB@sfr.frMongoDB@sfr.fr
MongoDB@sfr.fr
 

Kürzlich hochgeladen

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEarley Information Science
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processorsdebabhi2
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024The Digital Insurer
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024The Digital Insurer
 
Developing An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of BrazilDeveloping An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of BrazilV3cube
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking MenDelhi Call girls
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge
 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptxHampshireHUG
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsMaria Levchenko
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...apidays
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc
 
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsTop 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsRoshan Dwivedi
 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Enterprise Knowledge
 
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024The Digital Insurer
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)Gabriella Davis
 
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfThe Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfEnterprise Knowledge
 
Unblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen FramesUnblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen FramesSinan KOZAK
 
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking MenDelhi Call girls
 
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking MenDelhi Call girls
 

Kürzlich hochgeladen (20)

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024
 
Developing An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of BrazilDeveloping An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of Brazil
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
 
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsTop 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...
 
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfThe Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
 
Unblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen FramesUnblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen Frames
 
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men
 
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
 

upfront diet plone conf 2010

  • 1. upfront diet for plone Roché Compaan
  • 2.
  • 3. About me ● Owner of Upfront Systems ● Founded Upfront Systems in 1998 ● Zope developer for the past 10 years
  • 4. Experience ● Largest Plone deployments to date: 5FM & MetroFM ● 5FM – 20 million page views per month – 70 thousand members – 6 million objects in ZODB ● Metro FM – 15 million page views per month – 20 thousand members – 2 million objects in ZODB ● Helping Connexions with performance ● http://www.markhorner.net/2010/04/29/connexions-authoring- performance/
  • 5.
  • 6.
  • 7. Can Plone scale to 10 million documents?
  • 8. ZODB Benchmarks ● Most common myth spread: “ZODB is a high read, (s)low write database” ● No published ZODB benchmarks ● Wrote some benchmarks that test ZODB read/write speed in 2008 ● collective.zodbbench
  • 9. ZODB Write Speed 0 500 thousand 2,5 million 5 million 7,5 million 8,5 million 10 million 0 1000 2000 3000 4000 5000 6000 7000 8000 Postgres ZODB Number of records Inserts/second
  • 10. ZODB Lookup Speed Number of Objects Lookup Time 100,000 0.003 ms 1,000,000 0.006 ms 10,000,000 2 ms
  • 11. Benchmark Conclusions ● Thanks Jim, ZODB Rocks! ● Large cache size important for writes too, not only reads ● Need a solution for more than 10 million objects ● If the ZODB can handle 250 inserts/second at peak volume why does it take a full minute to add 250 ATDocuments?
  • 12. A closer look at Plone ● Added 10,000 documents to Plone, while profiling with Dieter Maurer's ZopeProfiler ● Timed “apply_index” method to find indexes performing badly with and without experimental.catalogqueryplan ● Batch import of content with an without references. ● Used fsdump to inspect pickle contents.
  • 13. Plone Benchmark Conclusions ● portal_catalog remains a bottleneck to scaling a Plone site. ● 1 catalog == many many conflicts ● reference_catalog is a major performance bottleneck ● Indexing 10,000 documents shows that 90% of the objects in Data.fs relate to indexing.
  • 14. What's in a pickle? 2,51755 81,564 9,872 20,024 85 655 1,544 30,455 182,708 7798,088122 Pickle contents of a single transaction adding an ATDocument object size in bytes IIBTree Folder IOBucket OIBucket IIBucket IOBTree BTrees.Length ATDocument OOBTree IISet Archetypes.BaseUnit OOBucket IITreeSet
  • 15. Think before you index! “Waking up objects is expensive” - this is an irrational fear. And adding another catalog in response to this fear requires therapy.
  • 16. The portal_catalog problem ● No navigation, portal tabs, folder contents, review queue if your content type is not in the portal_catalog. ● Your content type is automatically subscribed to 20+ indexes in the portal_catalog. Bad for new developers with hopes of building an enterprise CMS. ● Unless you have 5+ years experience with Zope – it's not so easy to “opt out” ● portal_catalog should only be used for content searches – no functionality must depend on it.
  • 17. What is upfront.diet? ● It's a healthier and happier approach to Plone development: index only when you really really need to. ● Set of packages that re-implement plone functionality without depending on the portal_catalog and reference_catalog. ● Includes replacements for: portal tabs, navigation portlet, folder contents, archetype references. ● Provides a catalog blacklist/whitelist to customise indexing per content type.
  • 18. More about upfront.diet ● upfront.navportlet ● Separate catalog for navportlet ● upfront.simplereferencefield ● Store references directly on object using annotations ● still depends uid_catalog ● upfront.foldercontents ● Adapter for IFolderish ● Index allowedRolesAndUsers directly on folder – less conflicts ● upfront.catalogblacklist ● Generic Setup profile with a whitelist and blacklist of indexes per content type ● patches IndexableObjectWrapper, ugh!
  • 19. upfront.diet todo list ● Packages: ● Portal tabs (done but not a separate package yet) ● Sitemap ● Review queue ● News and Events Smart Folders ● Site Search ● References without uid_catalog ● Comparative benchmarks with and without it
  • 20. upfront.diet deployments ● Very few at the moment but looks promising ● Fullmarks.org.za (half a million objects, Data.fs < 1GB)
  • 21. Contact Me ● Email: roche@upfrontsystems.co.za ● Blog: http://upfrontsystems.co.za/Members/roche ● facebook.com/rochecompaan ● twitter.com/rochecompaan