See conference video - http://www.lucidimagination.com/devzone/events/conferences/revolution/2011

1. A Study of I/O and Virtualization Performance with a Search Engine based on an XML database and Lucene Ed Buech é , EMC edward.bueche@emc.com, May 25, 2011

11. Lessons Learned… Structured Query use-cases Unstructured Query use-cases Fit to use-case

12. Indexes, DB, and IR Structured Query use-cases Unstructured Query use-cases Fit to use-case Scoring, Relevance, Entities Hierarchical data representations (XML) Full Text searches Constantly changing schemas Relational DB technology

13. Indexes, DB, and IR Structured Query use-cases Unstructured Query use-cases Fit to use-case Meta data query Transactions Advanced data management (partitions) JOINs Full Text index technology

14. Indexes, DB, and IR Structured Query use-cases Unstructured Query use-cases Fit to use-case Relational DB technology Full Text index technology

17. Libraries / Collections & Indexes = xDB segment = xDB Library / xPlore collection = xDB Index = xDB xml file ( dftxml , tracking xml, status, metrics, audit)

28. Sample %Ready for a production VM with xPlore deployment for an entire week “ official” area that Indicates pain In this case Avg resp time doubled and max resp time grew by 5x

29. Actual Ready samples during several hour period

33. Sample output from the Bonnie tool ¹ Bonnie is an open source disk I/O driver tool for Linux that can be useful for pretesting Linux disk environments prior to an xPlore/Lucene install. bonnie -s 1024 -y -u -o_direct -v 10 -p 10 This will increase the size of the file to 2 Gb. Examine the output. Focus on the random I/O area: ---Sequential Output (sync)----- ---Sequential Input-- --Rnd Seek- -CharUnlk- -DIOBlock- -DRewrite- -CharUnlk- -DIOBlock- --04k (10)- Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU Mach2 10*2024 73928 97 104142 5.3 26246 2.9 8872 22.5 43794 1.9 735.7 15.2 -s 1024 means that 2 GB files will be created -o_direct means that direct I/O (by-passing buffer cache) will be done -v 10 means that 10 different 2GB files will be created. -p 10 means that 10 different threads will query those files This output means that the random read test saw 735 random I/O’s per sec at 15% CPU busy

34. Linux indicators compared to bonnie output See https://community.emc.com/docs/DOC-9179 for additional example Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn sde 206.10 2402.40 0.80 24024 8 09:29:17 DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util 09:29:27 dev8-65 209.24 4877.97 1.62 23.32 1.62 7.75 3.80 79.59 09:29:17 PM CPU %user %nice %system %iowait %steal %idle 09:29:27 PM all 41.37 0.00 5.56 29.86 0.00 23.21 09:29:27 PM 0 62.44 0.00 10.56 25.38 0.00 1.62 09:29:27 PM 1 30.90 0.00 4.26 35.56 0.00 29.28 09:29:27 PM 2 36.35 0.00 3.96 30.76 0.00 28.93 09:29:27 PM 3 35.77 0.00 3.46 27.64 0.00 33.13 I/O stat output: SAR –d output: SAR –u output: Notice that at 200+ I/Os per sec the underlying volume is 80% busy. Although there could be multiple causes, one could be that some other VM is consuming the remaining I/O capacity (735 – 209 = 500+). High I/O wait

37. Some xPlore Structures for Search ¹ Dictionary of terms Posting list (doc-id’s for term) Stored fields (facets and node-ids) Security indexes (b-tree based) xDB XML store (contains text for summary) 1 st doc N-th doc Facet decompression map ¹ Frequency and position structures ignored for simplicity

38. IO model for search in xPlore Dictionary Posting list (doc-id’s for term) Stored fields Xdb node-id plus facet / security info Security lookup (b-tree based) xDB XML store (contains text for summary) Facet decompression map Search Term: ‘ term1 term2 ’ Result set

39. Separation of “covering values” in stored fields and summary Facet Calc FinalFacet calc values over thousands of results Res-1 - sum Res-2 - sum Res-3 - sum : : Res-350-sum Xdb docs with text for summary Small number for result window Small structure Potentially thousands of results Stored fields (Random access) Potentially thousands of hits Security lookup

40. xPlore Memory Pool areas at-a-glance xPlore Instance (fixed size) memory xDB Buffer Cache Lucene Caches & working memory xPlore caches Other vm working mem Operating System File Buffer cache ( dynamically sized ) Native code content extraction & linguistic processing memory

41. Lucene data resides primarily in OS buffer cache Potential for many things to sweep lucene from that cache