Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Operating Systems - Concurrency
1. Operating Systems
CMPSCI 377
Concurrency Patterns
Emery Berger
University of Massachusetts Amherst
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science
2. Finishing Up From Last Time
Avoiding deadlock: is this ok?
lock (a); lock (b);
lock (b); lock (a);
unlock (b); unlock (a);
unlock (a); unlock (b);
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 2
3. Finishing Up From Last Time
Not ok – may deadlock.
lock (a); lock (b);
lock (b); lock (a);
unlock (b); unlock (a);
unlock (a); unlock (b);
Solution: impose canonical order (acyclic)
lock (a); lock (a);
lock (b); lock (b);
unlock (b); unlock (b);
unlock (a); unlock (a);
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 3
4. Motivating Example: Web Server
web
server
Client (browser)
Requests HTML, images
not found
Server
Caches requests
http://server/Easter-bunny/
Sends to client
200x100/75.jpg
client
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science
5. Possible Implementation
while (true) {
wait for connection;
read from socket & parse URL;
look up URL contents in cache;
if (!in cache) {
fetch from disk / execute CGI;
put in cache;
}
send data to client;
}
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 5
6. Possible Implementation
while (true) {
wait for connection; // net
read from socket & parse URL; // cpu
look up URL contents in cache; // cpu
if (!in cache) {
fetch from disk / execute CGI;//disk
put in cache; // cpu
}
send data to client; // net
}
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 6
7. Problem: Concurrency
web
Sequential fine until:
server
More clients
Bigger server
Multicores, multiprocessors
Goals:
Hide latency of net & disk I/O
Don’t keep clients waiting
Improve throughput
Serve up more pages
clients
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science
8. Building Concurrent Apps
Patterns / Architectures
Thread pools
Producer-consumer
“Bag of tasks”
Worker threads (work stealing)
Goals:
Minimize latency
Maximize parallelism
Keep progs. simple to program & maintain
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 8
9. Thread Pools
Thread invocation & destruction relatively
expensive
Instead: use pool of threads
When new task arrives, get thread from pool
to work on it; block if pool empty
Faster with many tasks
Limits max threads
( ThreadPoolExecutor class in Java)
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 9
10. Producer-Consumer
Can get pipeline parallelism:
One thread (producer) does work
E.g., I/O
and hands it off to other thread (consumer)
producer consumer
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 10
11. Producer-Consumer
Can get pipeline parallelism:
One thread (producer) does work
E.g., I/O
and hands it off to other thread (consumer)
producer consumer
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 11
12. Producer-Consumer
Can get pipeline parallelism:
One thread (producer) does work
E.g., I/O
and hands it off to other thread (consumer)
producer consumer
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 12
13. Producer-Consumer
Can get pipeline parallelism:
One thread (producer) does work
E.g., I/O
and hands it off to other thread (consumer)
producer consumer
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 13
14. Producer-Consumer
Can get pipeline parallelism:
One thread (producer) does work
E.g., I/O
and hands it off to other thread (consumer)
producer consumer
LinkedBlockingQueue
Blocks on put() if full, poll() if empty
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 14
15. Producer-Consumer Web Server
Use 2 threads: producer & consumer
queue.put(x) and x = queue.poll();
while (true) { while (true) {
wait for connection; do something…
read from socket & parse URL; queue.put (x);
look up URL contents in cache; }
if (!in cache) {
fetch from disk / execute CGI;
put in cache;
}
while (true) {
send data to client;
x = queue.poll();
}
do something…
}
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 15
16. Producer-Consumer Web Server
Pair of threads – one reads, one writes
while (true) { while (true) {
wait for connection; URL = queue.poll();
read from socket & parse URL; look up URL contents in cache;
queue.put (URL); if (!in cache) {
} fetch from disk / execute CGI;
put in cache;
}
send data to client;
}
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 16
17. Producer-Consumer Web Server
More parallelism –
optimizes common case (cache hit)
while (true) { while (true) {
wait for connection; URL = queue1.poll();
read from socket & parse URL; look up URL contents in cache;
queue1.put (URL); if (!in cache) {
1
} queue2.put (URL); return;
}
send data to client;
}
2
while (true) {
URL = queue2.poll();
fetch from disk / execute CGI;
put in cache;
send data to client;
}
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 17
18. When to Use Producer-Consumer
Works well for pairs of threads
Best if producer & consumer are symmetric
Proceed roughly at same rate
Order of operations matters
Not as good for
Many threads
Order doesn’t matter
Different rates of progress
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 18
19. Producer-Consumer Web Server
Have to be careful to balance load across
threads
while (true) { while (true) {
wait for connection; URL = queue1.poll();
read from socket & parse URL; look up URL contents in cache;
queue1.put (URL); if (!in cache) {
1
} queue2.put (URL);
}
send data to client;
}
2
while (true) {
URL = queue2.poll();
fetch from disk / execute CGI;
put in cache;
send data to client;
}
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 19
20. Bag of Tasks
Collection of mostly independent tasks
worker worker worker worker
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 20
21. Bag of Tasks
Collection of mostly independent tasks
worker worker worker worker
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 21
22. Bag of Tasks
Collection of mostly independent tasks
worker worker worker worker
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 22
23. Bag of Tasks
Collection of mostly independent tasks
worker worker worker worker
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 23
24. Bag of Tasks
Collection of mostly independent tasks
worker worker worker worker
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 24
25. Bag of Tasks
Collection of mostly independent tasks
worker worker worker worker
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 25
26. Bag of Tasks
Collection of mostly independent tasks
addWork
worker worker worker worker
Bag could also be LinkedBlockingQueue
(put, poll)
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 26
27. Bag of Tasks Web Server
Re-structure this into bag of tasks:
addWork & worker threads
t = bag.poll() or bag.put(t)
while (true) {
wait for connection;
read from socket & parse URL;
look up URL contents in cache;
if (!in cache) {
fetch from disk / execute CGI;
put in cache;
}
send data to client;
}
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 27
28. Bag of Tasks Web Server
Re-structure this into bag of tasks:
addWork & worker
t = bag.poll() or bag.put(t)
addWork: Worker:
while (true) { while (true) {
wait for connection; URL = bag.poll();
bag.put (URL); look up URL contents in cache;
} if (!in cache) {
fetch from disk / execute CGI;
put in cache;
}
send data to client;
}
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 28
29. Bag of Tasks Web Server
Re-structure this into bag of tasks:
t = bag.poll() or bag.put(t)
addWork: while (true){
wait for connection;
bag.put (URL);
} worker
worker: while (true) {
addWork URL = bag.poll();
look up URL contents in cache;
if (!in cache) {
fetch from disk / execute CGI;
put in cache;
worker worker }
send data to client;
}
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 29
30. Bag of Tasks vs. Prod/Consumer
Exploits more parallelism
Even with coarse-grained threads
Don’t have to break up tasks too finely
Easy to change or add new functionality
But: one major performance problem…
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 30
31. What’s the Problem?
addWork
worker worker worker worker
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 31
32. What’s the Problem?
Contention – single lock on structure
Bottleneck to scalability
addWork
worker worker worker worker
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 32
33. Work Queues
Each thread has own work queue (deque)
No single point of contention
executor executor executor executor
Threads now generic “executors”
Tasks (balls): blue = parse, yellow = connect…
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 33
34. Work Queues
Each thread has own work queue (deque)
No single point of contention
executor executor executor executor
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 34
35. Work Queues
Each thread has own work queue (deque)
No single point of contention
executor executor executor executor
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 35
36. Work Queues
Each thread has own work queue (deque)
No single point of contention
executor executor executor executor
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 36
37. Work Queues
Each thread has own work queue
No single point of contention
executor executor executor executor
Now what?
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 37
38. Work Stealing
When thread runs out of work,
steal work from random other thread
worker worker worker worker
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 38
39. Work Stealing
When thread runs out of work,
steal work from top of random deque
worker worker worker worker
Optimal load balancing algorithm
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 39
40. Work Stealing Web Server
Re-structure:
readURL, lookUp, addToCache, output
myQueue.put(new readURL (url))
while (true) {
wait for connection;
read from socket & parse URL;
look up URL contents in cache;
if (!in cache) {
fetch from disk / execute CGI;
put in cache;
}
send data to client;
}
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 40
41. Work Stealing Web Server
Re-structure:
readURL, lookUp, addToCache, output
myQueue.put(new readURL (url))
readURL(url) {
wait for connection;
read from socket & parse URL;
myQueue.put (new lookUp (URL));
}
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 41
42. Work Stealing Web Server
Re-structure:
readURL, lookUp, addToCache, output
myQueue.put(new readURL (url))
readURL(url) { lookUp(url) {
wait for connection; look up URL contents in cache;
read from socket & parse URL; if (!in cache) {
myQueue.put (new lookUp (URL)); myQueue.put (new addToCache (URL));
} } else {
myQueue.put (new output(contents));
}
addToCache(URL) {
}
fetch from disk / execute CGI;
put in cache;
myQueue.put (new
output(contents));
}
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 42
43. Work Stealing
Works great for heterogeneous tasks
Convert addWork and worker into units of
work (different colors)
Flexible: can easily re-define tasks
Coarse, fine-grained, anything in-between
Automatic load balancing
Separates thread logic from functionality
Popular model for structuring servers
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 43
44. The End
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 44