Full Stack Load Testing
Load Testing On Alfresco 5.1
Derek Hulley
Alfresco Founding Engineer
Introduction
•  Resource Limits
•  Load Test Accuracy
•  Load Test Software
Introduction
Resource Limitations
Limiting Factors
•  Some Basic Server Limitations
–  Memory consumption
–  CPU usage
–  IOPS / Bandwidth / IO
•  Other Ser...
Moving Parts
Resources

Caches	
  
Start with a
guess and try it
out in anger
 Alfresco	
  and	
  
Share	
  
20%	
  
Transforma4on
s	
  
10%	
  
SOLR	
  
30%...
Focus

e.g. SOLR
Resources

Alfresco	
  and	
  
Share	
  
20%	
  
Transforma4on
s	
  
10%	
  
SOLR	
  
30%	
  
Database	
 ...
Resource Focus Example: SOLR
•  CPU: underused

•  Memory: underused

•  IO: underused
•  Network: Keep and eye on this!

...
Reduce spend
and iterate …


Resources

Alfresco	
  and	
  
Share	
  
23%	
  
Transforma4on
s	
  
12%	
  
SOLR	
  
18%	
  ...
Load Test Accuracy
Load Test Risk Factors: Scale
•  Increases Cost
–  Your Time
–  Getting the data in does not normally produce results
•  B...
Load Test Risk Factors: Documents
•  Document Types
–  Image thumbnail generation
–  Text to PDF for preview
–  Document l...
Load Test Risk Factors: API Usage Profile
•  Load Profile
–  Burst load
–  Trickle load
•  Data Structure
–  Paging
–  Sorti...
Load Test Risk Factors: UI Complexity
•  Share APIs
–  /private
•  Upload File
–  Stream binary
–  Then all of these at on...
Load Test Risk Factors: UI Complexity
LoadTestAccuracy
Load Test Risk Factors: UI Complexity
•  Page-Object Pattern: Browsers do what they do
Login Event Processor
 Login Page
 ...
Load Test Software
Alfresco Benchmark Server and Tests
References
•  Martin Bergljung’s walkthrough
•  Alfresco Wiki: Benchmark_Testing_with_Alfresco
•  GitHub: AlfrescoBenchmar...
Alfresco
Benchmark
Suite
TestSoftware
Benchmark	
  Server	
  
Tomcat	
  7	
  
Rest	
  API	
  
MongoDB	
  
Config	
  Data	
 ...
Load Tests: /tests/ent-signup
•  Create users
•  User base reused
–  Prerequisite for other tests
TestSoftware
Load Tests: /tests/dataload
•  Reuse user base
•  Create
–  Sites
–  Site Members
•  Site data
–  Folders
–  Files uploade...
Load Tests: /tests/dataload
•  AlfrescoBenchmark/alfresco-text-gen

•  Text data generated in server
•  Option to push dat...
Load Tests: GitHub benchmark-cmis
•  Burst load
•  Easy to extend
TestSoftware
Load Tests: FTP sample files vs spoofing
•  FTP server
–  A few thousand samples
–  Use for Share tests
•  Transformations
•...
Share Load Testing: /tests/share
•  Concurrent Users:
–  A new user logs in every N seconds
–  The user logs out after spe...
Summary
Summary
•  Build up load testing software and skills
•  Design tests around existing tools
–  Understand risks and where t...
Questions
Thank you!
Condividi su
#AlfrescoDayRoma
Nächste SlideShare
Wird geladen in …5
×

Alfresco Day Roma 2015: Full Stack Load Testing

733 Aufrufe

Veröffentlicht am

Alfresco Day Roma 2015: Full Stack Load Testing on Alfresco, Derek Hulley, Founding Engineer, Alfresco

Veröffentlicht in: Technologie
  • Als Erste(r) kommentieren

  • Gehören Sie zu den Ersten, denen das gefällt!

Alfresco Day Roma 2015: Full Stack Load Testing

  1. 1. Full Stack Load Testing Load Testing On Alfresco 5.1
  2. 2. Derek Hulley Alfresco Founding Engineer
  3. 3. Introduction •  Resource Limits •  Load Test Accuracy •  Load Test Software Introduction
  4. 4. Resource Limitations
  5. 5. Limiting Factors •  Some Basic Server Limitations –  Memory consumption –  CPU usage –  IOPS / Bandwidth / IO •  Other Server Limitations –  Monitoring (getting at logs, charts, monitors, etc) –  Backups –  CRON jobs •  Features –  Supported file types –  Auditing –  Jodconverter –  Clustering Resources
  6. 6. Moving Parts Resources Caches  
  7. 7. Start with a guess and try it out in anger Alfresco  and   Share   20%   Transforma4on s   10%   SOLR   30%   Database   30%   Content   5%   Load  Balancer   5%   Resource  Guess  0   Alfresco  and  Share   Transforma2ons   SOLR   Database   Content   Load  Balancer   Resources Monitor all Run for days Resolve Issues Repeat Adjust resources
  8. 8. Focus
 e.g. SOLR Resources Alfresco  and   Share   20%   Transforma4on s   10%   SOLR   30%   Database   30%   Content   5%   Load  Balancer   5%   Resource  Guess  0   Alfresco  and  Share   Transforma2ons   SOLR   Database   Content   Load  Balancer  
  9. 9. Resource Focus Example: SOLR •  CPU: underused •  Memory: underused •  IO: underused •  Network: Keep and eye on this! Resources
  10. 10. Reduce spend and iterate …
 Resources Alfresco  and   Share   23%   Transforma4on s   12%   SOLR   18%   Database   35%   Content   6%   Load  Balancer   6%   Resource  Guess  2   Alfresco  and  Share   Transforma2ons   SOLR   Database   Content   Load  Balancer  
  11. 11. Load Test Accuracy
  12. 12. Load Test Risk Factors: Scale •  Increases Cost –  Your Time –  Getting the data in does not normally produce results •  Bulk Loading –  Different hardware for this phase? –  Simulate bulk loading as accurately as necessary •  Number of Documents –  Easier to account for with experience –  Data storage costs –  Backup/Restore procedures •  Users and Groups –  LDAP sync –  Complex Group Hierarchy –  Site Group Permissions –  Mass-authentication LoadTestAccuracy
  13. 13. Load Test Risk Factors: Documents •  Document Types –  Image thumbnail generation –  Text to PDF for preview –  Document library listings (many transformations a.s.a.p) •  Document Size –  Network and IO costs and limits –  CPU used for streaming –  PDF to text memory consumption –  External converters •  More CPU or more infrastructure LoadTestAccuracy
  14. 14. Load Test Risk Factors: API Usage Profile •  Load Profile –  Burst load –  Trickle load •  Data Structure –  Paging –  Sorting –  Permissions •  The Internet –  CMIS across the Atlantic –  Southern hemisphere client apps LoadTestAccuracy
  15. 15. Load Test Risk Factors: UI Complexity •  Share APIs –  /private •  Upload File –  Stream binary –  Then all of these at once: •  Register user activity •  Generate thumbnail •  Generate PDF preview and display •  Get and display document metadata •  Get and display document version history •  Replay Doom (You Only Live Once) LoadTestAccuracy
  16. 16. Load Test Risk Factors: UI Complexity LoadTestAccuracy
  17. 17. Load Test Risk Factors: UI Complexity •  Page-Object Pattern: Browsers do what they do Login Event Processor Login Page Dashboard Page Browser processEvent loginAs Send keystrokes Login button click DashboardPage Timed render Wait For: Activities Wait For: My Sites Wait For: My Sites Wait For: My Sites Wait For: My Docs Wait For: etc Increasing Complexity Server x100 LoadTestAccuracy
  18. 18. Load Test Software Alfresco Benchmark Server and Tests
  19. 19. References •  Martin Bergljung’s walkthrough •  Alfresco Wiki: Benchmark_Testing_with_Alfresco •  GitHub: AlfrescoBenchmark TestSoftware
  20. 20. Alfresco Benchmark Suite TestSoftware Benchmark  Server   Tomcat  7   Rest  API   MongoDB   Config  Data   Services   MongoDB   Test  Data   UI   Benchmark  Driver  (xN)   Benchmark  Driver  (xN)   Benchmark  Driver   Tomcat  7   Extras   (Selenium)   Servers  /  APIs   Servers  /  APIs   Load  Balancer   Servers  /  APIs   Test   Services   Rest  API  
  21. 21. Load Tests: /tests/ent-signup •  Create users •  User base reused –  Prerequisite for other tests TestSoftware
  22. 22. Load Tests: /tests/dataload •  Reuse user base •  Create –  Sites –  Site Members •  Site data –  Folders –  Files uploaded using CMIS or TestSoftware
  23. 23. Load Tests: /tests/dataload •  AlfrescoBenchmark/alfresco-text-gen •  Text data generated in server •  Option to push data
 into ContentStore TestSoftware
  24. 24. Load Tests: GitHub benchmark-cmis •  Burst load •  Easy to extend TestSoftware
  25. 25. Load Tests: FTP sample files vs spoofing •  FTP server –  A few thousand samples –  Use for Share tests •  Transformations •  Text extractions •  Renditions •  Spoofing –  Stress SOLR –  document library size •  Search: searchterms.txt TestSoftware
  26. 26. Share Load Testing: /tests/share •  Concurrent Users: –  A new user logs in every N seconds –  The user logs out after spending Y seconds –  Number of concurrent users C = Y/N •  Selenium Driver Overload –  Use Windows –  5s to start a browser on EC2 m3.2xlarge –  Up to 40 browsers per load driver on EC2 m3.2xlarge –  Drivers required D = max( 5s/N, C/40) –  m3.2xlarge Windows à spot price of $0.3/hour –  500 concurrent user à20 drivers à $6/hour TestSoftware
  27. 27. Summary
  28. 28. Summary •  Build up load testing software and skills •  Design tests around existing tools –  Understand risks and where to extend or introduce new tests •  Iterate Summary
  29. 29. Questions
  30. 30. Thank you! Condividi su #AlfrescoDayRoma

×