P4 Search is a tool built internally and open-sourced in the Perforce Workshop. It creates and uses an external search index to allow users to search the content of a Perforce Server. This talk will explain the inner workings of P4 Search, its setup and applications, and explore ideas on how to extend this great and essential tool.

1. #
Sven Erik Knop
Technical Marketing Manager
Ralf Gronkowski
Principal Product Consultant

2. #

3. #
• Why P4Search?
• What is P4Search?
• Implementation Details and Demonstration

4. #

5. #
p4 files / p4 fstat / ...
???
File names,
Changes ...
File content?
C#
.h
JAVA
PPTX
PDF

6. #
• Built-in command, since Perforce 2010.1
• Search files stored in P4D based on content
– Case sensitive and insensitive searches
– Can use regular expressions
– Can search through all revisions
– Provide context search
• Returns depot paths

7. #
• A few drawbacks:
– Text search only, limited to 4K lines
– No search for Metadata such as attributes
• Performance concerns:
– Limited to 10,000 revisions by default
– Memory and CPU consumption
– But: lockless with peeking since 2013.3

8. #
p4 files/p4 fstat
index
store
search
Search engine indexes content
Stores it in its own database
Users search the index first
Index returns a depot path
Index and Perforce Server
can live on separate hosts

9. #
• Lucene
– Scalable, high performance indexing
– Search Algorithms
• Solr
– Stand-alone enterprise search server
– HTML Administration interface
– Extensible
• Tika
– Content analysis tool

10. #
• P4Search
– Index queue (processing indexing requests)
– Search controller (security)
– RESTful API (integration into other tools)
– UI (simple searches)
• Runs in Jetty

11. #
//depot/Talkhouse/rel1.0/com/walkerbros/common/widget/EBolt.java#10

12. #
• Changes/Changelists
• Branches
• Jobs
• Users
• Workspaces
• Depots

13. #
• Content
• Metadata (whatever that might be)

14. #

15. #
• Accessible through p4 files / p4 fstat ...

16. #

17. #
• For ordinary folks
– p4 edit file
– p4 attribute –n tags –v cool file
– p4 submit -d “just defined a cool tag on file rev”
• For admins
– p4 attribute –f –n tags –v cool file#rev
• Find them with
• p4 fstat -Oa -F "attr-tags=cool" //depot/...

18. #
• File content
• P4 Metadata
• P4 attributes
• And the common Metadata if desired

19. #

20. #
+ other fields

21. #
No ACL’s, no permission

22. #
• Is query endpoint for users
• Has simplified API
• Provides P4 authentication (password|ticket)
• Filters query results honoring the existing
P4 protections

23. #
P4Search
Search controller
Solr
Search index

24. #
• External index and protection table?
• Solution:
– Use a programmable search engine
– Use Perforce protections to filter results
 Users need read access to files to be able to search

25. #
• Jetty
– Solr
• Lucene
• Jetty
– P4Search
• Search queue/Indexer
• Search controller
• RESTful API
• UI

26. #
• swarm.workshop.perforce.com/projects/perforce-software-
p4search/files/main

27. #
• Download from the Workshop
• Follow the provided instructions to install
• Run two services
– p4search-solr
– p4search-jetty

28. #
• On first run index your entire depot
– You probably don’t want to do this
• On submit index new file revs
– change-commit trigger on depot location
• At any time any given change
– curl POST --data commit,change#
http://p4search:8080/api/queue/{token}

29. #
• Indexing
– With trigger P4D, so ultimately any given client and user
• Searching
– P4Search UI
– Piper
– Commons
– Custom through P4Search API

30. #
• Deep dive after learning Lucene/Solr
• Starting point
p4search/solr/example/solr/collection1/conf
– schema.xml
– solrconfig.xml

31. #

32. #
Sven Erik Knop
sknop@perforce.com
Ralf Gronkowski
rgronkowski@perforce.com
@_gronk