13. RT index
Search Architecture
RT stream
Analyzer/
Partitioner
RT index
(Earlybird)
Blender
Archive
index
RT index
Mapreduce
Analyzer
raw
tweets
Tweet archive
HDFS
Search
requests
writes
searches
analyzed
tweets
analyzed
tweets
raw
tweets
14. RT index
Search Architecture
Tweets
Analyzer/
Partitioner
RT index
(Earlybird)
Blender
Archive
index
RT index
queue
HDFS
Search
requests
Updates Deletes/
Engagement (e.g. retweets/favs)
writes
searches
Mapreduce
Analyzer
15. RT index
Search Architecture
RT index
(Earlybird)
Social
graph Social
Blender
Archive
index
RT index
User
search
Search
requests
writes
searches
• Blender is our Thrift
service aggregator
• Queries multiple
Earlybirds, merges results
Social
graph
graph
17. Search Architecture
RT index
(Earlybird)
Archive
index
• For historic reasons, these used
to be entirely different codebases,
but had similar features/
technologies
• Over time cross-dependencies
were introduced to share code
User
search
Lucene
18. Search Architecture
RT index
(Earlybird)
Archive
index
User
search
Lucene
Extensions
Lucene
• New Lucene extension package
• This package is truly generic and
has no dependency on an actual
product/index
• It contains Twitter’s extensions for
real-time search, a thin segment
management layer and other
features
22. Lucene Extension Library
• Abstraction layer for Lucene index segments
• Real-time writer for in-memory index segments
• Schema-based Lucene document factory
• Real-time faceting
23. Lucene Extension Library
• API layer for Lucene segments
• *IndexSegmentWriter
• *IndexSegmentAtomicReader
• Two implementations
• In-memory: RealtimeIndexSegmentWriter (and reader)
• On-disk: LuceneIndexSegmentWriter (and reader)
24. Lucene Extension Library
• IndexSegments can be built ...
• in realtime
• on Mesos or Hadoop (Mapreduce)
• locally on serving machines
• Cluster-management code that deals with IndexSegments
• Share segments across serving machines using HDFS
• Can rebuild segments (e.g. to upgrade Lucene version, change data
schema, etc.)
26. RealtimeIndexSegmentWriter
• Modified Lucene index implementation optimized for realtime search
• IndexWriter buffer is searchable (no need to flush to allow searching)
• In-memory
• Lock-free concurrency model for best performance
27. Concurrency - Definitions
• Pessimistic locking
• A thread holds an exclusive lock on a resource, while an action is
performed [mutual exclusion]
• Usually used when conflicts are expected to be likely
• Optimistic locking
• Operations are tried to be performed atomically without holding a lock;
conflicts can be detected; retry logic is often used in case of conflicts
• Usually used when conflicts are expected to be the exception
28. Concurrency - Definitions
• Non-blocking algorithm
Ensures, that threads competing for shared resources do not have their
execution indefinitely postponed by mutual exclusion.
• Lock-free algorithm
A non-blocking algorithm is lock-free if there is guaranteed system-wide
progress.
• Wait-free algorithm
A non-blocking algorithm is wait-free, if there is guaranteed per-thread
progress.
* Source: Wikipedia
29. Concurrency
• Having a single writer thread simplifies our problem: no locks have to be used
to protect data structures from corruption (only one thread modifies data)
• But: we have to make sure that all readers always see a consistent state of
all data structures -> this is much harder than it sounds!
• In Java, it is not guaranteed that one thread will see changes that another
thread makes in program execution order, unless the same memory barrier is
crossed by both threads -> safe publication
• Safe publication can be achieved in different, subtle ways. Read the great
book “Java concurrency in practice” by Brian Goetz for more information!
30. Java Memory Model
• Program order rule
Each action in a thread happens-before every action in that thread that comes
later in the program order.
• Volatile variable rule
A write to a volatile field happens-before every subsequent read of that same
field.
• Transitivity
If A happens-before B, and B happens-before C, then A happens-before C.
* Source: Brian Goetz: Java Concurrency in Practice
35. Concurrency
RAM 0
int x;
Thread A writes b=1 to RAM,
because b is volatile
5 x = 5;
1
Cache
Thread 1 Thread 2
time
volatile int b;
b = 1;
36. Concurrency
RAM 0
int x;
5 x = 5;
1
Cache
Thread 1 Thread 2
time
volatile int b;
b = 1;
Read volatile b
int dummy = b;
while(x != 5);
37. Concurrency
RAM 0
int x;
5 x = 5;
1
Cache
Thread 1 Thread 2
time
volatile int b;
b = 1;
int dummy = b;
while(x != 5);
happens-before
• Program order rule: Each action in a thread happens-before every action in
that thread that comes later in the program order.
38. Concurrency
RAM 0
int x;
5 x = 5;
1
Cache
Thread 1 Thread 2
time
volatile int b;
b = 1;
int dummy = b;
while(x != 5);
happens-before
• Volatile variable rule: A write to a volatile field happens-before every
subsequent read of that same field.
39. Concurrency
RAM 0
int x;
5 x = 5;
1
Cache
Thread 1 Thread 2
time
volatile int b;
b = 1;
int dummy = b;
while(x != 5);
happens-before
• Transitivity: If A happens-before B, and B happens-before C, then A
happens-before C.
40. Concurrency
RAM 0
int x;
5 x = 5;
1
Cache
Thread 1 Thread 2
time
volatile int b;
b = 1;
int dummy = b;
while(x != 5);
This condition will be
false, i.e. x==5
• Note: x itself doesn’t have to be volatile. There can be many variables like x,
but we need only a single volatile field.
41. Concurrency
RAM 0
int x;
5 x = 5;
1
Cache
Thread 1 Thread 2
time
volatile int b;
b = 1;
int dummy = b;
while(x != 5);
Memory barrier
• Note: x itself doesn’t have to be volatile. There can be many variables like x,
but we need only a single volatile field.
44. Concurrency
RAM 0
int x;
5 x = 5;
1
Cache
Thread 1 Thread 2
time
volatile int b;
b = 1;
int dummy = b;
while(x != 5);
Memory barrier
• Note: x itself doesn’t have to be volatile. There can be many variables like x,
but we need only a single volatile field.
45. Concurrency
IndexWriter IndexReader
time
write 100 docs
maxDoc = 100
in IR.open(): read maxDoc
search upto maxDoc
write more docs
maxDoc is volatile
46. Concurrency
IndexWriter IndexReader
time
write 100 docs
maxDoc = 100
in IR.open(): read maxDoc
search upto maxDoc
write more docs
maxDoc is volatile
happens-before
• Only maxDoc is volatile. All other fields that IW writes to and IR reads from
don’t need to be!
47. Wait-free
• Not a single exclusive lock
• Writer thread can always make progress
• Optimistic locking (retry-logic) in a few places for searcher thread
• Retry logic very simple and guaranteed to always make progress
48. In-memory Real-time Index
• Highly optimized for GC - all data is stored in blocked native arrays
• v1: Optimized for tweets with a term position limit of 255
• v2: Support for 32 bit positions without performance degradation
• v2: Basic support for out-of-order posting list inserts
49. In-memory Real-time Index
• Highly optimized for GC - all data is stored in blocked native arrays
• v1: Optimized for tweets with a term position limit of 255
• v2: Support for 32 bit positions without performance degradation
• v2: Basic support for out-of-order posting list inserts
50. In-memory Real-time Index
• RT term dictionary
• Term lookups using a lock-free hashtable in O(1)
• v2: Additional probabilistic, lock-free skip list maintains ordering on terms
• Perfect skip list not an option: out-of-order inserts would require
rebalancing, which is impractical with our lock-free index
• In a probabilistic skip list the tower height of a new (out-of-order) item can
be determined without knowing its insert position by simply rolling a dice
54. In-memory Real-time Index
• Probabilistic skip list Tower height determined by rolling a dice
BEFORE knowing the insert location; tower height
never has to change for an element, simplifying
memory allocation and concurrency.
55. Schema-based Document factory
• Apps provide one ThriftSchema per index and create a ThriftDocument for
each document
• SchemaDocumentFactory translates ThriftDocument -> Lucene Document
using the Schema
• Default field values
• Extended field settings
• Type-system on top of DocValues
• Validation
56. Schema-based Document factory
Schema
Lucene
Document
SchemaDocument
Factory
Thrift
Document
• Validation
• Fill in default values
• Apply correct Lucene
field settings
57. Schema-based Document factory
Schema
Lucene
Document
SchemaDocument
Factory
Thrift
Document
• Validation
• Fill in default values
• Apply correct Lucene
field settings
Decouples core package from
specific product/index. Similar
to Solr/ElasticSearch.
61. Outlook
• Support for parallel (sliced) segments to support partial segment rebuilds
and other cool posting list update patterns
• Add remaining missing Lucene features to RT index
• Index term statistics for ranking
• Term vectors
• Stored fields
65. Searching for top entities within Tweets
• Task: Find the best photos in a subset of tweets
• We could use a Lucene index, where each photo is a document
• Problem: How to update existing documents when the same photos are
tweeted again?
• In-place posting list updates are hard
• Lucene’s updateDocument() is a delete/add operation - expensive and not
order-preserving
66. Searching for top entities within Tweets
• Task: Find the best photos in a subset of tweets
• Could we use our existing time-ordered tweet index?
• Facets!
67. Searching for top entities within Tweets
Query Doc ids
Inverted
index
Term id Term label
Forward
Doc id index Document
Metadata
Facet
index
Doc id Term ids
69. 5 15 9000 9002 100000 100090
Matching
doc id
Facet
index
Term ids
Top-k heap
Id Count
48239 8
31241 2
Query
Searching for top entities within Tweets
70. 5 15 9000 9002 100000 100090
Matching
doc id
Facet
index
Term ids
Top-k heap
Id Count
48239 15
31241 12
85932 8
6748 3
Query
Searching for top entities within Tweets
71. Searching for top entities within Tweets
5 15 9000 9002 100000 100090
Matching
doc id
Facet
index
Term ids
Top-k heap
Id Count
48239 15
31241 12
85932 8
6748 3
Query
Weighted counts (from
engagement features) used
for relevance scoring
72. Searching for top entities within Tweets
5 15 9000 9002 100000 100090
Matching
doc id
Facet
index
Term ids
Top-k heap
Id Count
48239 15
31241 12
85932 8
6748 3
Query
All query operators can be
used. E.g. find best photos in
San Francisco tweeted by
people I follow
73. Searching for top entities within Tweets
Inverted
Term id index Term label
74. Searching for top entities within Tweets
Id Count Label Count
pic.twitter.com/jknui4w 45
pic.twitter.com/dslkfj83 23
pic.twitter.com/acm3ps 15
pic.twitter.com/948jdsd 11
pic.twitter.com/dsjkf15h 8
pic.twitter.com/irnsoa32 5
48239 45
31241 23
85932 15
6748 11
74294 8
3728 5
Inverted
index
75. Summary
• Indexing tweet entities (e.g. photos) as facets allows to search and rank top-entities
using a tweets index
• All query operators supported
• Documents don’t need to be reindexed
• Approach reusable for different use cases, e.g.: best vines, hashtags,
@mentions, etc.