Most of the data that is pulled out of an SCM like Perforce Helix is common across multiple workspaces. Leveraging this fact means only fetching the data once from the repository. By creating cheap copies or clones of this data on demand, it is possible to dramatically reduce the load on the network, disks and Perforce servers, while making near-instant workspaces available to users.

1. Leveraging Structured Data
To Reduce Disk, IO & Network
Bandwidth
Vishal Moondhra
VP Applications
Methodics Inc.

2. 2
The Problem – We Are Drowning in Data!
 According to IBM, 90% of the world’s
data has been created in the last 2
years! (www.ibm.com/smarterplanet)
 Semiconductors, Computational Fluid
Dynamic simulations, Biotech
simulations, game development…
• Smartphone CPU development requires
100GB user workspaces

3. 3
Users Only Modify Small Subsets of Data
 Up to 95% of all data
in a workspace is not
modified
• All the data is needed for
reference – simulations,
builds, rendering…
• Majority of users only
edit/modify their small
portion
USER1 WS
USER2 WS
ModificationsUnmodified

4. 4
CI Makes it worse
User Commit(s)
Same Data
is ‘sync’ed to
multiple
workspaces

5. 5
Design Data is Lazy!
The majority of your design data is
duplicated, occupying expensive
disks and choking your network!

6. 6
‘Data De-duplication’ is not the solution
 De-duplication is done ‘after the fact’ – data is already
created
 Filer CPU and Disk bandwidth hog
• Might run for hours
• 20-30% filer CPU utilization, bringing filer performance down
 It is a very broad technique
• Looks at all available data – cannot be partitioned by project etc.
 Does not change the load on the Helix server

7. 7
Structured Data
 Each workspace has a ‘structure’ – provided by Perforce
Helix
• All workspaces of a project have similar (if not the same) structures
• Workspaces are managed by a P4 client
• The client determines which files are present in the workspace
• P4 Protect determines which files are visible to a user
• Files are versioned and have ‘changelists’
 This structure can be leveraged to create smarter, smaller
and faster workspaces

8. 8
A Smarter way to build workspaces
 Start with a ‘master’ workspace – fully populated with all the
data
 Build ‘clones’ of this master workspace on demand
 Each clone gets its own P4 client
 Each clone can be independently sync’ed to any changelist
• Sync the clone to the required (or latest) changelist when created
• Clone only stores the ‘diff’ between master and the sync’ed changelist

9. 9
Permissions
 The Master workspace has
all content
 P4 Protect is baked into the
clone creation process
• Clones will only get the files
visible to the user creating the
workspace
• Transactional nature of this
process ensures that it cannot be
interrupted / hijacked
Master
Workspace
Clone
Files not
available in
clone
due to P4
protections

10. 10
Bring Data into Clones when needed
 Clones use the Master workspace as reference
• All common data is referred from the master, at the file block level
 When a file is written to, the system delivers the blocks to
the clone – just-in-time
• Happens ‘under the covers’
• Transparent to users
 The modified files are now contained in the clone
• Allows each user to modify his/her workspace independently, as expected

11. 11
Clones are like any normal workspace
 P4 users will see no difference between a clone and a
traditional ‘sync’ from P4
 All P4 commands will work as before
• Add/edit/delete/submit
• Client commands
 Commits go back to P4 server from the clone
 ‘sync’ commands will bring in new data

12. 12
Standard P4 Workspace creation
Project
Reference
Client
P4
Client
Speed:
SLOW
Byte count:
HIGH
TEMPLATE
Create a
new workspace
Sync …
Fetch files
From depot

13. 13
Smarter Workspaces
Project
Reference
Client Masters
Create a
new workspace Pick a
master
Clone
‘sync’
latest
changes
Speed:
FAST
Byte count:
LOW
Speed:
INSTANT
Byte count:
FREE
Speed:
FAST
Byte count:
LOW

14. 14
Sample Disk Savings (1GB workspaces)
Workspaces Disk Space Used
(Traditionally)
Disk Space Used
(Initial)
Disk Space Used
(Steady State)++
Savings
(Total)
Master+ -- 1.00GB 1.5GB --
1 WS 1.00GB 84KB 100.084MB 0%
10 Workspaces 10GB 840KB 260.092MB 78.33%
20 Workspaces 20GB 1.68MB 350.168MB 83.33%
100 Workspaces 100GB 8.4MB 11.15GB 88.61%
++ Estimated. The rule used here is that users will sync up to 10% of their workspace at steady state.

15. 15
Workspace Creation Time

16. Never Pull The Same
Data From Your Helix
Server Twice!

17. Thank you!
vishal@methodics.com