1. Prophet
a path out of the cloud
http://syncwith.us
jesse@bestpractical.com

2. You may know me from...
RT (Request Tracker)
Jifty
SVK
Hiveminder
Perl 6
T-shirts

3. I’ve been hacking on an
open source database
called “Prophet”

4. It has an API like
Amazon SimpleDB or
Google App Engine’s...

5. It’s designed for
“team-scale” apps

6. It’s built for P2P
replication and
disconnected use

7. But first, a brief
digression...

8. ...about cloud
computing
☁
☔

9. Living in the cloud
=
sharecropping（佃农)

10. (That’s bad)
☹

11. The bad old days:

12. Pic of sharecroppers

13. You farmed land you
didn’t own...

14. ...with tools you
couldn’t really afford

15. You paid for it with
part of your harvest...

16. It sounded like a
pretty sweet deal...

17. ...until things got bad

18. (Things always got bad)

20. (Internet)

21. (Cloud)
(Internet)

22. In a bad year, you got
further in debt to
the land owner

24. So, what does this have
to do with software?

25. The (more recent)
bad old days:

26. pic of mainframes

27. You ran code you didn’t
own on hardware you
didn’t own

28. Things started to get
better in the 1980s

30. Pic of PCs

31. Users started to be
able to make choices
about computing...

32. They weren’t all rosy

33. Pic of BSOD

34. Sometimes new
versions of software
broke things...

35. ...leaving you locked in
to old versions

36. pic of win 31?

37. Things got ‘better’

38. rms
che

39. Now, things are getting
worse again...

41. What happens when
your favorite service
goes down?

42. pic of twitter being
down

43. ...or stops accepting
new signups?

45. ...or starts making
arbitrary choices about
what’s ‘safe’ content?

47. ...or breaks

49. You don’t own the
services you use

50. You probably don’t
even have a contract

51. When a service
provider cuts you off,
you lose

52. Not so secret shame:
I still need the cloud

53. My calendar lives at
google.com

55. I make a Web 2.0 todo
list service called
Hiveminder.com

56. pic of hiveminder

57. ☣ Using hosted apps
is going to hurt you! ☣

58. What about Google
Gears,Adobe Air, etc?

59. Great. now you can use
your word processer
while you’re offline!

60. Pic of wordperfect

61. Real offline apps
should not need servers

62. Real offline apps
should sync like you do

63. Back to that
database thing...

64. JesseVincent

65. Chia-liang Kao

66. We work together

67. CL lives in Taipei
Jesse lives in Boston

68. Sometimes we need
to work face to face

69. TPE - BOS:
TPE - HNL:
BOS - HNL:
9410 mi
5,095 mi
5,069 mi

70. Step 1: Go to Hawaii for “work”
Step 2: ???
Step 3: Prophet!
Our Plan

71. The Plan Backfired
We were there for 8 days
We wrote 8000 lines of Perl
We figured out step 2

72. Step 2:
Build a Disconnected
Syncable Database

73. Fallacies of Distributed Computing
1.The network is reliable.
2. Latency is zero.
3. Bandwidth is infinite.
4.The network is secure.
5.Topology doesn't change.
6.There is one administrator.
7.Transport cost is zero.
8.The network is homogeneous.

74. A grounded, peer to peer
replicated, disconnected,
versioned, property
database with self-healing
conflict resolution
Prophet

75. What do all those
buzzwords mean?

76. grounded
Runs here

77. grounded
Not here

78. grounded
Runs at the edge
Doesn’t need to run in the cloud
Syncs with services you already use
(Adaptors talk to “Foreign Replicas”)

79. Update any replica
Pull from any replica
Push to any replica
Publish a replica
Changes will propagate
peer-to-peer replicated

80. Real-time replication is hard to scale
It only “works” with constant connectivity
I don’t have constant connectivity
Neither do you
Prophet sync can happen whenever
disconnected

81. Every update is recorded as a change set
Change sets don’t lose any data
(so you can use them to go backwards)
All history is introspectable
Replication just replays changesets
versioned

82. Atomic operations
CREATE, READ, UPDATE, DELETE, SEARCH
Record types can have optional validation
and canonicalization
Records of the same type do not need to
have the same properties
Add and remove properties at will
property database

83. Remembers all conflict resolutions
Syncs all resolutions with your peers
Detects identical conflicts
Uses your peers’ resolutions to “vote” for
the winner of a conflict
self-healing
conflict resolution

84. Working with Prophet

85. RESTy API
GET /records.json
GET /records/Cars.json
GET /records/Cars/716499-5F9-4AC4-827.json
GET /records/Cars/716499-5F9-4AC4-827/wheels.json
POST /records/Cars.json
POST /records/Cars/716499-5F9-4AC4-827.json
POST /records/Cars/716499-5F9-4AC4-827/wheels.json

86. RESTy API
Yes, we should be using PUT and DELETE
(or just provide an OpenRESTY adaptor)
Yes, you can have a commit bit and
help us fix it :)

87. Native API
(Yes, the core is Perl.)
my $cli = Prophet::CLI->new();
my $cxn = $cli->app_handle->handle;
my $record = Prophet::Record->new( handle => $cxn, type => 'Person' );
my $uuid = $record->create( props => { name => 'Jesse', age => 31 } );
$record->set_prop( name => 'age', value => 32 );
my $people = Prophet::Collection->new( handle => $cxn, type => 'Person' );
$people->matching( sub { shift->prop('species') ne 'cat' } );

88. What could you build
with Prophet?

89. A bug tracker: “simple defects”
•id. Status, Summary
•(Arbitrary other properties too)
•History
•Comments
•Attachments
sd

90. Initialize
./bin/sd init

91. ./bin/sd ticket create --
summary="Can't sync sd with Google Code"
status=new
Created ticket 5 (93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4)
Create

92. ./bin/sd ticket search --regex publish
29 } new the online help doesn't describe publish
34 } new publish a static html view of records
35 } new publish should create a static rss file
List and Search

93. ./bin/sd ticket update
--uuid 93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4
-- status=resolved
Updates

94. Bugs on my laptop
aren’t interesting.

95. Jesse
sd publish --to fsck.com:public_html/sd/
CL
sd clone --from http://my.com/~jesse/sd
Sync!

96. Jesse
sd server
CL
sd pull --local
Hackathon mode!

97. My project has a bug tracker

98. Actually, mine use two:
• RT
• hiveminder.com
My project has a bug tracker

99. Foreign Replicas
Prophet makes Foreign Replicas easy
SD gets them "for free"

100. (Using only the public REST API)
It took an afternoon
Mirror an RT instance into SD
Share it with your peers using prophet
Sync changes back from your peers to RT
Supports Comments and Attachments
Wrote an RT Replica for SD

101. (Using only the public REST API)
...and one for Hiveminder

102. I can sync my bugs with
RT or Hiveminder

103. Actually, it’s better

104. I can sync between RT
and Hiveminder

105. I can sync between two
different RTs, too

106. • Trac
• Launchpad
• Google Code
• SourceForge
• Bugzilla
• Jira
• GForge
• debbugs
• GNATS
• todo.txt
• Lighthouse
• Redmine
• FogBugz
• What else?
We need more replica
definitions:

107. What else can you use
Prophet for?

108. •CRM
•Bug tracking
•Sales orders
•Phone book
•Blog
•Trading Card
Database
•Ideas?
All the databases you
want while offline.

109. How about a P2P BBS?
Prophet doesn’t need a server.
You can sync over sneakernet.
“Private” Social Networks

110. A look inside Prophet

111. Anatomy of a
Prophet Replica

112. The bits and pieces
Database UUID
Replica UUID
Record Store
Changeset Store
Resolution Database
Configuration metadata

113. The Record Store
Stores individual records by type
Not guaranteed to have all old versions

114. The Changeset Store
Stores every change to a set of records
Guaranteed to have all old changesets
Replaying all changesets will create an exact
clone of the replica

115. Replica Backends

116. Filesystem
Readable
Flat files
Compact
Fast
(Not yet fully atomic)

117. HTTP
Designed to let you “publish” databases
Flat-files, Currently read-only.
Same format as the filesystem replica type.

118. Backends are pluggable!
The filesystem is cheap and easy
The filesystem is portable
Help us write new backends:
CouchDB, Postgres, SQLite, MySQL, S3,
AppEngine, $YOUR_FAVORITE_DB

119. Prophet is designed to sync with “other”
databases and systems
They don’t need to support all of Prophet’s
features - Prophet knows how to interpret
mumbo-jumbo from the Cloud
Foreign Replicas will usually be app specific
All current examples are for SD
Foreign Replicas

120. Synchronization

121. Publish
Serialize and export all of a replica's
resolutions and changesets

122. Pull
Integrate unseen resolutions and then
unseen changesets from a replica

123. Push
Integrate new resolutions and changesets
into a replica

124. Conflicts

125. Figures out the best resolution
“Nullifies” the conflict so the changeset can
be cleanly integrated
Integrates the conflicting changeset
Records the resolution as a new changeset
Records the resolution decision in the
resolution database
Resolving Conflicts

126. Prophet has clever ways to figure out the best
resolution.
If there are previous resolutions for the same
conflict and a majority agree, use that
If the merger has specified a “prefer this side”
choice, use that
Prompt the user to make a decision, giving
them info about previous decisions for this
conflict
“The Best Resolution”

127. Scaling

128. Scaling to giant clusters is boring
(Can I play the “They’re not Green” card here?)
Scales to many weakly connected peers
You are not Google
Does anyone here work for Google?
Current target is databases of O(50k) records
How does it scale?

129. We have a political agenda.
Cloud computing is not Open.
APIs for “export” are not good enough.
You should always have full control.
You probably don’t need to store 10 billion
records in one database.
Why not, then?

130. Do you have 10 billion
bugs, customer contacts
or sales orders?

131. Can I come
work for you?

132. We would love a scalable,
high performance
Prophet backend

133. Getting Involved

134. Project Status
Simple, well-defined Perl API
RESTy web API (with microserver)
Fast, lightweight backend
Small, active dev community
Great test coverage
...less than great documentation coverage

135. Better ergonomics
Improved search and indexing
(Including full-text indexing)
Client libraries for other languages
Proper security model
More apps
Our Plans

136. Prophet
8225 lines of code and doc
2120 lines of tests
sd
2751 lines of code and doc
1121 lines of tests
Codebase

137. Prophet is very young
Prophet designed in April
Prophet core implemented in April
SD designed in April
SD built in June and July

138. We need your help!
Kick-ass functional and text indexing
Backend data store improvements
Slick GUIs for syncing
More Foreign Replicas for SD
Documentation improvements
A clever logo
New applications

139. Prophet
http://syncwith.us/prophet/download
SD
http://syncwith.us/sd/download
Getting Prophet

140. http://syncwith.us
prophet-subscribe@lists.bestpractical.com
#prophet on freenode IRC
谢 谢!