SlideShare ist ein Scribd-Unternehmen logo
1 von 94
Downloaden Sie, um offline zu lesen
@crichardson
Map(), ïŹ‚atMap() and reduce() are your
new best friends:
Simpler collections, concurrency, and big
data
Chris Richardson
Author of POJOs in Action
Founder of the original CloudFoundry.com
@crichardson
chris@chrisrichardson.net
http://plainoldobjects.com
@crichardson
Presentation goal
How functional programming simpliïŹes
your code
Show that
map(), ïŹ‚atMap() and reduce()
are remarkably versatile functions
@crichardson
About Chris
@crichardson
About Chris
Founder of a buzzword compliant (stealthy, social, mobile,
big data, machine learning, ...) startup
Consultant helping organizations improve how they
architect and deploy applications using cloud, micro
services, polyglot applications, NoSQL, ...
@crichardson
Agenda
Why functional programming?
Simplifying collection processing
Eliminating NullPointerExceptions
Simplifying concurrency with Futures and Rx Observables
Tackling big data problems with functional programming
@crichardson
What’s functional
programming?
@crichardson
It’s a programming paradigm
@crichardson
It’s a kind of programming
language
@crichardson
Functions as the building
blocks of the application
@crichardson
Functions as ïŹrst class
citizens
Assign functions to variables
Store functions in ïŹelds
Use and write higher-order functions:
Pass functions as arguments
Return functions as values
@crichardson
Avoids mutable state
Use:
Immutable data structures
Single assignment variables
Some functional languages such as Haskell don’t side-effects
There are beneïŹts to immutability
Easier concurrency
More reliable code
But be pragmatic
@crichardson
Why functional programming?
"the highest goal of
programming-language
design to enable good
ideas to be elegantly
expressed"
http://en.wikipedia.org/wiki/Tony_Hoare
@crichardson
Why functional programming?
More expressive
More concise
More intuitive - solution matches problem deïŹnition
Elimination of error-prone mutable state
Easy parallelization
@crichardson
An ancient idea that has
recently become popular
@crichardson
Mathematical foundation:
λ-calculus
Introduced by
Alonzo Church in the 1930s
@crichardson
Lisp = an early functional
language invented in 1958
http://en.wikipedia.org/wiki/
Lisp_(programming_language)
1940
1950
1960
1970
1980
1990
2000
2010
garbage collection
dynamic typing
self-hosting compiler
tree data structures
(defun factorial (n)
(if (<= n 1)
1
(* n (factorial (- n 1)))))
@crichardson
My ïŹnal year project in 1985:
Implementing SASL
sieve (p:xs) =
p : sieve [x | x <- xs, rem x p > 0];
primes = sieve [2..]
A list of integers starting with 2
Filter out multiples of p
Mostly an Ivory Tower
technology
Lisp was used for AI
FP languages: Miranda,
ML, Haskell, ...
“Side-eïŹ€ects
kills kittens and
puppies”
@crichardson
http://steve-yegge.blogspot.com/2010/12/haskell-researchers-announce-discovery.html
!*
!*
!*
@crichardson
But today FP is mainstream
Clojure - a dialect of Lisp
A hybrid OO/functional language
A hybrid OO/FP language for .NET
Java 8 has lambda expressions
@crichardson
Java 8 lambda expressions
are functions
x -> x * x
x -> {
for (int i = 2; i < Math.sqrt(x); i = i + 1) {
if (x % i == 0)
return false;
}
return true;
};
(x, y) -> x * x + y * y
@crichardson
Java 8 lambdas are a
shorthand* for an anonymous
inner class
* not exactly. See http://programmers.stackexchange.com/questions/
177879/type-inference-in-java-8
@crichardson
Java 8 functional interfaces
Interface with a single abstract method
e.g. Runnable, Callable, Spring’s TransactionCallback
A lambda expression is an instance of a functional
interface.
You can use a lambda wherever a function interface
“value” is expected
The type of the lambda expression is determined from it’s
context
@crichardson
Example Functional Interface
Function<Integer, Integer> square = x -> x * x;
BiFunction<Integer, Integer, Integer>
sumSquares = (x, y) -> x * x + y * y;
Predicate<Integer> makeIsDivisibleBy(int y) {
return x -> x % y == 0;
}
Predicate<Integer> isEven = makeIsDivisibleBy(2);
Assert.assertTrue(isEven.test(8));
Assert.assertFalse(isEven.test(11));
@crichardson
Example Functional Interface
ExecutorService executor = ...;
final int x = 999
Future<Boolean> outcome = executor.submit(() -> {
for (int i = 2; i < Math.sqrt(x); i = i + 1) {
if (x % i == 0)
return false;
}
return true;
}
This lambda is
a Callable
@crichardson
Agenda
Why functional programming?
Simplifying collection processing
Eliminating NullPointerExceptions
Simplifying concurrency with Futures and Rx Observables
Tackling big data problems with functional programming
@crichardson
Lot’s of application code
=
collection processing:
Mapping, ïŹltering, and reducing
@crichardson
Social network example
public class Person {
enum Gender { MALE, FEMALE }
private Name name;
private LocalDate birthday;
private Gender gender;
private Hometown hometown;
private Set<Friend> friends = new HashSet<Friend>();
....
public class Friend {
private Person friend;
private LocalDate becameFriends;
...
}
public class SocialNetwork {
private Set<Person> people;
...
@crichardson
Mapping, ïŹltering, and
reducing
public class Person {
public Set<Hometown> hometownsOfFriends() {
Set<Hometown> result = new HashSet<>();
for (Friend friend : friends) {
result.add(friend.getPerson().getHometown());
}
return result;
}
@crichardson
Mapping, ïŹltering, and
reducing
public class Person {
public Set<Person> friendOfFriends() {
Set<Person> result = new HashSet();
for (Friend friend : friends)
for (Friend friendOfFriend : friend.getPerson().friends)
if (friendOfFriend.getPerson() != this)
result.add(friendOfFriend.getPerson());
return result;
}
@crichardson
Mapping, ïŹltering, and
reducing
public class SocialNetwork {
private Set<Person> people;
...
public Set<Person> lonelyPeople() {
Set<Person> result = new HashSet<Person>();
for (Person p : people) {
if (p.getFriends().isEmpty())
result.add(p);
}
return result;
}
@crichardson
Mapping, ïŹltering, and
reducing
public class SocialNetwork {
private Set<Person> people;
...
public int averageNumberOfFriends() {
int sum = 0;
for (Person p : people) {
sum += p.getFriends().size();
}
return sum / people.size();
}
@crichardson
Problems with this style of
programming
Low level
Imperative (how to do it) NOT declarative (what to do)
Verbose
Mutable variables are potentially error prone
DifïŹcult to parallelize
@crichardson
Java 8 streams to the rescue
A sequence of elements
“Wrapper” around a collection
Streams can also be inïŹnite
Provides a functional/lambda-based API for transforming,
ïŹltering and aggregating elements
Much simpler, cleaner code
@crichardson
Using Java 8 streams -
mapping
class Person ..
private Set<Friend> friends = ...;
public Set<Hometown> hometownsOfFriends() {
return friends.stream()
.map(f -> f.getPerson().getHometown())
.collect(Collectors.toSet());
}
@crichardson
The map() function
s1 a b c d e ...
s2 f(a) f(b) f(c) f(d) f(e) ...
s2 = s1.map(f)
@crichardson
public class SocialNetwork {
private Set<Person> people;
...
public Set<Person> peopleWithNoFriends() {
Set<Person> result = new HashSet<Person>();
for (Person p : people) {
if (p.getFriends().isEmpty())
result.add(p);
}
return result;
}
Using Java 8 streams -
ïŹltering
public class SocialNetwork {
private Set<Person> people;
...
public Set<Person> lonelyPeople() {
return people.stream()
.filter(p -> p.getFriends().isEmpty())
.collect(Collectors.toSet());
}
@crichardson
Using Java 8 streams - friend
of friends V1
class Person ..
public Set<Person> friendOfFriends() {
Set<Set<Friend>> fof = friends.stream()
.map(friend -> friend.getPerson().friends)
.collect(Collectors.toSet());
...
}
Using map()
=> Set of Sets :-(
Somehow we need to flatten
@crichardson
Using Java 8 streams -
mapping
class Person ..
public Set<Person> friendOfFriends() {
return friends.stream()
.flatMap(friend -> friend.getPerson().friends.stream())
.map(Friend::getPerson)
.filter(f -> f != this)
.collect(Collectors.toSet());
}
maps and ïŹ‚attens
@crichardson
The ïŹ‚atMap() function
s1 a b ...
s2 f(a)0 f(a)1 f(b)0 f(b)1 f(b)2 ...
s2 = s1.ïŹ‚atMap(f)
@crichardson
Using Java 8 streams -
reducing
public class SocialNetwork {
private Set<Person> people;
...
public long averageNumberOfFriends() {
return people.stream()
.map ( p -> p.getFriends().size() )
.reduce(0, (x, y) -> x + y)
/ people.size();
} int x = 0;
for (int y : inputStream)
x = x + y
return x;
@crichardson
The reduce() function
s1 a b c d e ...
x = s1.reduce(initial, f)
f(f(f(f(f(f(initial, a), b), c), d), e), ...)
@crichardson
Newton's method for ïŹnding square
roots
public class SquareRootCalculator {
public double squareRoot(double input, double precision) {
return
Stream.iterate(
new Result(1.0),
current -> refine(current, input, precision))
.filter(r -> r.done)
.findFirst().get().value;
}
private static Result refine(Result current,
double input, double precision) {
double value = current.value;
double newCurrent =
value - (value * value - input) / (2 * value);
boolean done = Math.abs(value - newCurrent) < precision;
return new Result(newCurrent, done);
}
class Result { boolean done; double value; }
Creates an inïŹnite stream:
seed, f(seed), f(f(seed)), .....
Don’t panic!
Streams are lazy
@crichardson
Agenda
Why functional programming?
Simplifying collection processing
Eliminating NullPointerExceptions
Simplifying concurrency with Futures and Rx Observables
Tackling big data problems with functional programming
@crichardson
Tony’s $1B mistake
“I call it my billion-dollar mistake.
It was the invention of the null
reference in 1965....But I couldn't
resist the temptation to put in a
null reference, simply because it
was so easy to implement...”
http://qconlondon.com/london-2009/presentation/
Null+References:+The+Billion+Dollar+Mistake
@crichardson
Coding with null pointers
class Person
public Friend longestFriendship() {
Friend result = null;
for (Friend friend : friends) {
if (result == null ||
friend.getBecameFriends()
.isBefore(result.getBecameFriends()))
result = friend;
}
return result;
}
Friend oldestFriend = person.longestFriendship();
if (oldestFriend != null) {
...
} else {
...
}
Null check is essential yet
easily forgotten
@crichardson
Java 8 Optional<T>
A wrapper for nullable references
It has two states:
empty throws an exception if you try to get the reference
non-empty contain a non-null reference
Provides methods for:
testing whether it has a value
getting the value
...
Return reference wrapped in an instance of this type instead of null
@crichardson
Coding with optionals
class Person
public Optional<Friend> longestFriendship() {
Friend result = null;
for (Friend friend : friends) {
if (result == null ||
friend.getBecameFriends().isBefore(result.getBecameFriends()))
result = friend;
}
return Optional.ofNullable(result);
}
Optional<Friend> oldestFriend = person.longestFriendship();
// Might throw java.util.NoSuchElementException: No value present
// Person dangerous = popularPerson.get();
if (oldestFriend.isPresent) {
...oldestFriend.get()
} else {
...
}
@crichardson
Using Optionals - better
Optional<Friend> oldestFriendship = ...;
Friend whoToCall1 = oldestFriendship.orElse(mother);
Avoid calling isPresent() and get()
Friend whoToCall3 =
oldestFriendship.orElseThrow(
() -> new LonelyPersonException());
Friend whoToCall2 =
oldestFriendship.orElseGet(() -> lazilyFindSomeoneElse());
@crichardson
Using Optional.map()
public class Person {
public Optional<Friend> longestFriendship() {
return ...;
}
public Optional<Long> ageDifferenceWithOldestFriend() {
Optional<Friend> oldestFriend = longestFriendship();
return oldestFriend.map ( of ->
Math.abs(of.getPerson().getAge() - getAge())) );
}
Eliminates messy conditional logic
@crichardson
Using ïŹ‚atMap()
class Person
public Optional<Friend> longestFriendship() {...}
public Optional<Friend> longestFriendshipOfLongestFriend() {
return
longestFriendship()
.flatMap(friend ->
friend.getPerson().longestFriendship());
}
not always a symmetric
relationship. :-)
@crichardson
Agenda
Why functional programming?
Simplifying collection processing
Eliminating NullPointerExceptions
Simplifying concurrency with Futures and Rx Observables
Tackling big data problems with functional programming
@crichardson
Let’s imagine you are performing
a CPU intensive operation
class Person ..
public Set<Hometown> hometownsOfFriends() {
return friends.stream()
.map(f -> cpuIntensiveOperation())
.collect(Collectors.toSet());
}
@crichardson
class Person ..
public Set<Hometown> hometownsOfFriends() {
return friends.parallelStream()
.map(f -> cpuIntensiveOperation())
.collect(Collectors.toSet());
}
Parallel streams = simple
concurrency Potentially uses N cores
Nx speed up
@crichardson
Let’s imagine that you are
writing code to display the
products in a user’s wish list
@crichardson
The need for concurrency
Step #1
Web service request to get the user proïŹle including wish
list (list of product Ids)
Step #2
For each productId: web service request to get product info
Sequentially terrible response time
Need fetch productInfo concurrently
@crichardson
Futures are a great
concurrency abstraction
http://en.wikipedia.org/wiki/Futures_and_promises
@crichardson
Worker thread or
event-driven
Main thread
How futures work
Outcome
Future
Client
get
Asynchronous
operation
set
initiates
@crichardson
BeneïŹts
Simple way for two concurrent activities to communicate safely
Abstraction:
Client does not know how the asynchronous operation is
implemented
Easy to implement scatter/gather:
Scatter: Client can invoke multiple asynchronous operations
and gets a Future for each one.
Gather: Get values from the futures
@crichardson
Example wish list service
public interface UserService {
Future<UserProfile> getUserProfile(long userId);
}
public class UserServiceProxy implements UserService {
private ExecutorService executorService;
@Override
public Future<UserProfile> getUserProfile(long userId) {
return executorService.submit(() ->
restfulGet("http://uservice/user/" + userId,
UserProfile.class));
}
...
}
public interface ProductInfoService {
Future<ProductInfo> getProductInfo(long productId);
}
@crichardson
public class WishlistService {
private UserService userService;
private ProductInfoService productInfoService;
public Wishlist getWishlistDetails(long userId) throws Exception {
Future<UserProfile> userProfileFuture = userService.getUserProfile(userId);
UserProfile userProfile = userProfileFuture.get(300, TimeUnit.MILLISECONDS);
Example wish list service
get user
info
List<Future<ProductInfo>> productInfoFutures =
userProfile.getWishListProductIds().stream()
.map(productInfoService::getProductInfo)
.collect(Collectors.toList());
long deadline = System.currentTimeMillis() + 300;
List<ProductInfo> products = new ArrayList<ProductInfo>();
for (Future<ProductInfo> pif : productInfoFutures) {
long timeout = deadline - System.currentTimeMillis();
if (timeout <= 0) throw new TimeoutException(...);
products.add(pif.get(timeout, TimeUnit.MILLISECONDS));
}
...
return new Wishlist(products);
}
asynchronously
get all products
wait for
product
info
@crichardson
It works BUT
Code is very low-level and
messy
And, it’s blocking
@crichardson
Better: Futures with callbacks
no blocking!
def asyncSquare(x : Int)
: Future[Int] = ... x * x...
val f = asyncSquare(25)
Guava ListenableFutures, Spring 4 ListenableFuture
Java 8 CompletableFuture, Scala Futures
f onSuccess {
case x : Int => println(x)
}
f onFailure {
case e : Exception => println("exception thrown")
}
Partial function applied to
successful outcome
Applied to failed outcome
@crichardson
But
callback-based scatter/gather
Messy, tangled code
(aka. callback hell)
@crichardson
Functional futures - map
def asyncPlus(x : Int, y : Int) = ... x + y ...
val future2 = asyncPlus(4, 5).map{ _ * 3 }
assertEquals(27, Await.result(future2, 1 second))
Scala, Java 8 CompletableFuture
Asynchronously transforms
future
@crichardson
Functional futures - ïŹ‚atMap()
val f2 = asyncPlus(5, 8).flatMap { x => asyncSquare(x) }
assertEquals(169, Await.result(f2, 1 second))
Scala, Java 8 CompletableFuture (partially)
Calls asyncSquare() with the eventual
outcome of asyncPlus()
@crichardson
ïŹ‚atMap() is asynchronous
Outcome3f3
Outcome3
f2
f2 = f1 ïŹ‚atMap (someFn)
Outcome1
f1
Implemented using callbacks
someFn(outcome1)
@crichardson
class WishListService(...) {
def getWishList(userId : Long) : Future[WishList] = {
userService.getUserProfile(userId) flatMap { userProfile =>
Scala wishlist service
val futureOfProductsList : Future[List[ProductInfo]] =
Future.sequence(listOfProductFutures)
val timeoutFuture = ...
Future.firstCompletedOf(Seq(wishlist, timeoutFuture))
}
}
val wishlist =
futureOfProductsList.map { products =>
WishList(products) }
val listOfProductFutures : List[Future[ProductInfo]] =
userProfile.wishListProductIds
.map { productInfoService.getProductInfo }
@crichardson
Using Java 8 CompletableFutures
public class UserServiceImpl implements UserService {
@Override
public CompletableFuture<UserInfo> getUserInfo(long userId) {
return CompletableFuture.supplyAsync(
() -> httpGetRequest("http://myuservice/user" + userId,
UserInfo.class));
}
Runs in ExecutorService
@crichardson
Using Java 8 CompletableFutures
public CompletableFuture<Wishlist> getWishlistDetails(long userId) {
return userService.getUserProfile(userId).thenComposeAsync(userProfile -> {
Stream<CompletableFuture<ProductInfo>> s1 =
userProfile.getWishListProductIds()
.stream()
.map(productInfoService::getProductInfo);
Stream<CompletableFuture<List<ProductInfo>>> s2 =
s1.map(fOfPi -> fOfPi.thenApplyAsync(pi -> Arrays.asList(pi)));
CompletableFuture<List<ProductInfo>> productInfos = s2
.reduce((f1, f2) -> f1.thenCombine(f2, ListUtils::union))
.orElse(CompletableFuture.completedFuture(Collections.emptyList()));
return productInfos.thenApply(list -> new Wishlist());
});
}
Java 8 is missing Future.sequence()
ïŹ‚atMap()!
map()!
@crichardson
Introducing Reactive
Extensions (Rx)
The Reactive Extensions (Rx) is a library for composing
asynchronous and event-based programs using
observable sequences and LINQ-style query operators.
Using Rx, developers represent asynchronous data
streams with Observables , query asynchronous
data streams using LINQ operators , and .....
https://rx.codeplex.com/
@crichardson
About RxJava
Reactive Extensions (Rx) for the JVM
Original motivation for NetïŹ‚ix was to provide rich Futures
Implemented in Java
Adaptors for Scala, Groovy and Clojure
https://github.com/NetïŹ‚ix/RxJava
@crichardson
RxJava core concepts
trait Observable[T] {
def subscribe(observer : Observer[T]) : Subscription
...
}
trait Observer[T] {
def onNext(value : T)
def onCompleted()
def onError(e : Throwable)
}
NotiïŹes
An asynchronous stream of items
Used to
unsubscribe
Comparing Observable to...
Observer pattern - similar but
adds
Observer.onComplete()
Observer.onError()
Iterator pattern - mirror image
Push rather than pull
Futures - similar
Can be used as Futures
But Observables = a stream
of multiple values
Collections and Streams -
similar
Functional API supporting
map(), ïŹ‚atMap(), ...
But Observables are
asynchronous
@crichardson
Fun with observables
val every10Seconds = Observable.interval(10 seconds)
-1 0 1 ...
t=0 t=10 t=20 ...
val oneItem = Observable.items(-1L)
val ticker = oneItem ++ every10Seconds
val subscription = ticker.subscribe { (value: Long) => println("value=" + value) }
...
subscription.unsubscribe()
@crichardson
def getTableStatus(tableName: String) : Observable[DynamoDbStatus]=
Observable { subscriber: Subscriber[DynamoDbMessage] =>
}
Connecting observables to the
outside world
amazonDynamoDBAsyncClient.describeTableAsync(
new DescribeTableRequest(tableName),
new AsyncHandler[DescribeTableRequest, DescribeTableResult] {
override def onSuccess(request: DescribeTableRequest,
result: DescribeTableResult) = {
subscriber.onNext(DynamoDbStatus(result.getTable.getTableStatus))
subscriber.onCompleted()
}
override def onError(exception: Exception) = exception match {
case t: ResourceNotFoundException =>
subscriber.onNext(DynamoDbStatus("NOT_FOUND"))
subscriber.onCompleted()
case _ =>
subscriber.onError(exception)
}
})
}
@crichardson
Transforming observables
val tableStatus = ticker.flatMap { i =>
logger.info("{}th describe table", i + 1)
getTableStatus(name)
}
Status1 Status2 Status3 ...
t=0 t=10 t=20 ...
+ Usual collection methods: map(), filter(), take(), drop(), ...
@crichardson
Calculating rolling average
class AverageTradePriceCalculator {
def calculateAverages(trades: Observable[Trade]):
Observable[AveragePrice] = {
...
}
case class Trade(
symbol : String,
price : Double,
quantity : Int
...
)
case class AveragePrice(
symbol : String,
price : Double,
...)
@crichardson
Calculating average pricesdef calculateAverages(trades: Observable[Trade]): Observable[AveragePrice] = {
trades.groupBy(_.symbol).map { symbolAndTrades =>
val (symbol, tradesForSymbol) = symbolAndTrades
val openingEverySecond =
Observable.items(-1L) ++ Observable.interval(1 seconds)
def closingAfterSixSeconds(opening: Any) =
Observable.interval(6 seconds).take(1)
tradesForSymbol.window(...).map {
windowOfTradesForSymbol =>
windowOfTradesForSymbol.fold((0.0, 0, List[Double]())) { (soFar, trade) =>
val (sum, count, prices) = soFar
(sum + trade.price, count + trade.quantity, trade.price +: prices)
} map { x =>
val (sum, length, prices) = x
AveragePrice(symbol, sum / length, prices)
}
}.flatten
}.flatten
}
@crichardson
Agenda
Why functional programming?
Simplifying collection processing
Eliminating NullPointerExceptions
Simplifying concurrency with Futures and Rx Observables
Tackling big data problems with functional programming
@crichardson
Let’s imagine that you want
to count word frequencies
@crichardson
Scala Word Count
val frequency : Map[String, Int] =
Source.fromFile("gettysburgaddress.txt").getLines()
.flatMap { _.split(" ") }.toList
frequency("THE") should be(11)
frequency("LIBERTY") should be(1)
.groupBy(identity)
.mapValues(_.length))
Map
Reduce
@crichardson
But how to scale to a cluster
of machines?
@crichardson
Apache Hadoop
Open-source software for reliable, scalable, distributed computing
Hadoop Distributed File System (HDFS)
EfïŹciently stores very large amounts of data
Files are partitioned and replicated across multiple machines
Hadoop MapReduce
Batch processing system
Provides plumbing for writing distributed jobs
Handles failures
...
@crichardson
Overview of MapReduce
Input
Data
Mapper
Mapper
Mapper
Reducer
Reducer
Reducer
Out
put
Data
ShufïŹ‚e
(K,V)
(K,V)
(K,V)
(K,V)*
(K,V)*
(K,V)*
(K1,V, ....)*
(K2,V, ....)*
(K3,V, ....)*
(K,V)
(K,V)
(K,V)
@crichardson
MapReduce Word count -
mapperclass Map extends Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, Context context) {
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
context.write(word, one);
}
}
}
(“Four”, 1), (“score”, 1), (“and”, 1), (“seven”, 1), ...
Four score and seven years
http://wiki.apache.org/hadoop/WordCount
@crichardson
Hadoop then shufïŹ‚es the
key-value pairs...
@crichardson
MapReduce Word count -
reducer
class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key,
Iterable<IntWritable> values, Context context) {
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
context.write(key, new IntWritable(sum));
}
}
(“the”, 11)
(“the”, (1, 1, 1, 1, 1, 1, ...))
http://wiki.apache.org/hadoop/WordCount
@crichardson
About MapReduce
Very simple programming abstract yet incredibly powerful
By chaining together multiple map/reduce jobs you can process
very large amounts of data
e.g. Apache Mahout for machine learning
But
Mappers and Reducers = verbose code
Development is challenging, e.g. unit testing is difïŹcult
It’s disk-based, batch processing slow
@crichardson
Scalding: Scala DSL for
MapReduce
class WordCountJob(args : Args) extends Job(args) {
TextLine( args("input") )
.flatMap('line -> 'word) { line : String => tokenize(line) }
.groupBy('word) { _.size }
.write( Tsv( args("output") ) )
def tokenize(text : String) : Array[String] = {
text.toLowerCase.replaceAll("[^a-zA-Z0-9s]", "")
.split("s+")
}
}
https://github.com/twitter/scalding
Expressive and unit testable
Each row is a map of
named ïŹelds
@crichardson
Apache Spark
Part of the Hadoop ecosystem
Key abstraction = Resilient Distributed Datasets (RDD)
Collection that is partitioned across cluster members
Operations are parallelized
Created from either a Scala collection or a Hadoop
supported datasource - HDFS, S3 etc
Can be cached in-memory for super-fast performance
Can be replicated for fault-tolerance
http://spark.apache.org
@crichardson
Spark Word Count
val sc = new SparkContext(...)
sc.textFile(“s3n://mybucket/...”)
.flatMap { _.split(" ")}
.groupBy(identity)
.mapValues(_.length)
.toArray.toMap
}
}
Expressive, unit testable and very fast
@crichardson
Summary
Functional programming enables the elegant expression of
good ideas in a wide variety of domains
map(), ïŹ‚atMap() and reduce() are remarkably versatile
higher-order functions
Use FP and OOP together
Java 8 has taken a good ïŹrst step towards supporting FP
@crichardson
Questions?
@crichardson chris@chrisrichardson.net
http://plainoldobjects.com

Weitere Àhnliche Inhalte

Was ist angesagt?

Clean code slide
Clean code slideClean code slide
Clean code slideAnh Huan Miu
 
Monadic Java
Monadic JavaMonadic Java
Monadic JavaMario Fusco
 
Design functional solutions in Java, a practical example
Design functional solutions in Java, a practical exampleDesign functional solutions in Java, a practical example
Design functional solutions in Java, a practical exampleMarian Wamsiedel
 
Clean Code: Chapter 3 Function
Clean Code: Chapter 3 FunctionClean Code: Chapter 3 Function
Clean Code: Chapter 3 FunctionKent Huang
 
Pragmatic functional refactoring with java 8
Pragmatic functional refactoring with java 8Pragmatic functional refactoring with java 8
Pragmatic functional refactoring with java 8RichardWarburton
 
Clean Code I - Best Practices
Clean Code I - Best PracticesClean Code I - Best Practices
Clean Code I - Best PracticesTheo Jungeblut
 
Java 8 Lambda and Streams
Java 8 Lambda and StreamsJava 8 Lambda and Streams
Java 8 Lambda and StreamsVenkata Naga Ravi
 
Solid C++ by Example
Solid C++ by ExampleSolid C++ by Example
Solid C++ by ExampleOlve Maudal
 
Dependency injection in Java, from naive to functional
Dependency injection in Java, from naive to functionalDependency injection in Java, from naive to functional
Dependency injection in Java, from naive to functionalMarian Wamsiedel
 
Why TypeScript?
Why TypeScript?Why TypeScript?
Why TypeScript?FITC
 
JavaScript Inheritance
JavaScript InheritanceJavaScript Inheritance
JavaScript InheritanceJussi Pohjolainen
 
Advanced Javascript
Advanced JavascriptAdvanced Javascript
Advanced JavascriptAdieu
 
JavaScript - Chapter 8 - Objects
 JavaScript - Chapter 8 - Objects JavaScript - Chapter 8 - Objects
JavaScript - Chapter 8 - ObjectsWebStackAcademy
 
A Deeper look into Javascript Basics
A Deeper look into Javascript BasicsA Deeper look into Javascript Basics
A Deeper look into Javascript BasicsMindfire Solutions
 
Fundamental JavaScript [UTC, March 2014]
Fundamental JavaScript [UTC, March 2014]Fundamental JavaScript [UTC, March 2014]
Fundamental JavaScript [UTC, March 2014]Aaron Gustafson
 

Was ist angesagt? (20)

Clean code slide
Clean code slideClean code slide
Clean code slide
 
Clean code
Clean codeClean code
Clean code
 
Monadic Java
Monadic JavaMonadic Java
Monadic Java
 
Design functional solutions in Java, a practical example
Design functional solutions in Java, a practical exampleDesign functional solutions in Java, a practical example
Design functional solutions in Java, a practical example
 
Clean Code: Chapter 3 Function
Clean Code: Chapter 3 FunctionClean Code: Chapter 3 Function
Clean Code: Chapter 3 Function
 
Pragmatic functional refactoring with java 8
Pragmatic functional refactoring with java 8Pragmatic functional refactoring with java 8
Pragmatic functional refactoring with java 8
 
Clean Code I - Best Practices
Clean Code I - Best PracticesClean Code I - Best Practices
Clean Code I - Best Practices
 
Clean code
Clean codeClean code
Clean code
 
Java 8 Lambda and Streams
Java 8 Lambda and StreamsJava 8 Lambda and Streams
Java 8 Lambda and Streams
 
Solid C++ by Example
Solid C++ by ExampleSolid C++ by Example
Solid C++ by Example
 
Clean code
Clean codeClean code
Clean code
 
JavaScript Promises
JavaScript PromisesJavaScript Promises
JavaScript Promises
 
Clean Code
Clean CodeClean Code
Clean Code
 
Dependency injection in Java, from naive to functional
Dependency injection in Java, from naive to functionalDependency injection in Java, from naive to functional
Dependency injection in Java, from naive to functional
 
Why TypeScript?
Why TypeScript?Why TypeScript?
Why TypeScript?
 
JavaScript Inheritance
JavaScript InheritanceJavaScript Inheritance
JavaScript Inheritance
 
Advanced Javascript
Advanced JavascriptAdvanced Javascript
Advanced Javascript
 
JavaScript - Chapter 8 - Objects
 JavaScript - Chapter 8 - Objects JavaScript - Chapter 8 - Objects
JavaScript - Chapter 8 - Objects
 
A Deeper look into Javascript Basics
A Deeper look into Javascript BasicsA Deeper look into Javascript Basics
A Deeper look into Javascript Basics
 
Fundamental JavaScript [UTC, March 2014]
Fundamental JavaScript [UTC, March 2014]Fundamental JavaScript [UTC, March 2014]
Fundamental JavaScript [UTC, March 2014]
 

Andere mochten auch

Map, Flatmap and Reduce are Your New Best Friends: Simpler Collections, Concu...
Map, Flatmap and Reduce are Your New Best Friends: Simpler Collections, Concu...Map, Flatmap and Reduce are Your New Best Friends: Simpler Collections, Concu...
Map, Flatmap and Reduce are Your New Best Friends: Simpler Collections, Concu...Chris Richardson
 
Why we cannot ignore Functional Programming
Why we cannot ignore Functional ProgrammingWhy we cannot ignore Functional Programming
Why we cannot ignore Functional ProgrammingMario Fusco
 
OOP and FP - Become a Better Programmer
OOP and FP - Become a Better ProgrammerOOP and FP - Become a Better Programmer
OOP and FP - Become a Better ProgrammerMario Fusco
 
Reactive Programming for a demanding world: building event-driven and respons...
Reactive Programming for a demanding world: building event-driven and respons...Reactive Programming for a demanding world: building event-driven and respons...
Reactive Programming for a demanding world: building event-driven and respons...Mario Fusco
 
Developing microservices with aggregates (devnexus2017)
Developing microservices with aggregates (devnexus2017)Developing microservices with aggregates (devnexus2017)
Developing microservices with aggregates (devnexus2017)Chris Richardson
 
WebCamp:Front-end Developers Day. АлДĐșŃĐ°ĐœĐŽŃ€ ĐœĐŸŃŃ‚ĐŸĐČĐ”ĐœĐșĐŸ "Rx.js - ĐŽĐ”Đ»Đ°Đ”ĐŒ Đ°ŃĐžĐœŃ…Ń€...
WebCamp:Front-end Developers Day. АлДĐșŃĐ°ĐœĐŽŃ€ ĐœĐŸŃŃ‚ĐŸĐČĐ”ĐœĐșĐŸ "Rx.js - ĐŽĐ”Đ»Đ°Đ”ĐŒ Đ°ŃĐžĐœŃ…Ń€...WebCamp:Front-end Developers Day. АлДĐșŃĐ°ĐœĐŽŃ€ ĐœĐŸŃŃ‚ĐŸĐČĐ”ĐœĐșĐŸ "Rx.js - ĐŽĐ”Đ»Đ°Đ”ĐŒ Đ°ŃĐžĐœŃ…Ń€...
WebCamp:Front-end Developers Day. АлДĐșŃĐ°ĐœĐŽŃ€ ĐœĐŸŃŃ‚ĐŸĐČĐ”ĐœĐșĐŸ "Rx.js - ĐŽĐ”Đ»Đ°Đ”ĐŒ Đ°ŃĐžĐœŃ…Ń€...GeeksLab Odessa
 
GKAC 2014 Nov. - ê·žëŁšëč„ëĄœ ì•ˆë“œëĄœìŽë“œ 앱 개발하Ʞ
GKAC 2014 Nov. - ê·žëŁšëč„ëĄœ ì•ˆë“œëĄœìŽë“œ 앱 개발하ꞰGKAC 2014 Nov. - ê·žëŁšëč„ëĄœ ì•ˆë“œëĄœìŽë“œ 앱 개발하Ʞ
GKAC 2014 Nov. - ê·žëŁšëč„ëĄœ ì•ˆë“œëĄœìŽë“œ 앱 개발하ꞰGDG Korea
 
Solit 2014, РДаĐșтоĐČĐœŃ‹Đč Javascript. ĐŸĐŸĐ±Đ”ĐŽĐ° ĐœĐ°ĐŽ Đ°ŃĐžĐœŃ…Ń€ĐŸĐœĐœĐŸŃŃ‚ŃŒŃŽ Đž ĐČĐ»ĐŸĐ¶Đ”ĐœĐœĐŸŃŃ‚ŃŒŃŽ, ...
Solit 2014, РДаĐșтоĐČĐœŃ‹Đč Javascript. ĐŸĐŸĐ±Đ”ĐŽĐ° ĐœĐ°ĐŽ Đ°ŃĐžĐœŃ…Ń€ĐŸĐœĐœĐŸŃŃ‚ŃŒŃŽ Đž ĐČĐ»ĐŸĐ¶Đ”ĐœĐœĐŸŃŃ‚ŃŒŃŽ, ...Solit 2014, РДаĐșтоĐČĐœŃ‹Đč Javascript. ĐŸĐŸĐ±Đ”ĐŽĐ° ĐœĐ°ĐŽ Đ°ŃĐžĐœŃ…Ń€ĐŸĐœĐœĐŸŃŃ‚ŃŒŃŽ Đž ĐČĐ»ĐŸĐ¶Đ”ĐœĐœĐŸŃŃ‚ŃŒŃŽ, ...
Solit 2014, РДаĐșтоĐČĐœŃ‹Đč Javascript. ĐŸĐŸĐ±Đ”ĐŽĐ° ĐœĐ°ĐŽ Đ°ŃĐžĐœŃ…Ń€ĐŸĐœĐœĐŸŃŃ‚ŃŒŃŽ Đž ĐČĐ»ĐŸĐ¶Đ”ĐœĐœĐŸŃŃ‚ŃŒŃŽ, ...solit
 
Decomposing applications for deployability and scalability (SpringOne China 2...
Decomposing applications for deployability and scalability (SpringOne China 2...Decomposing applications for deployability and scalability (SpringOne China 2...
Decomposing applications for deployability and scalability (SpringOne China 2...Chris Richardson
 
Polygot persistence for Java Developers - August 2011 / @Oakjug
Polygot persistence for Java Developers - August 2011 / @OakjugPolygot persistence for Java Developers - August 2011 / @Oakjug
Polygot persistence for Java Developers - August 2011 / @OakjugChris Richardson
 
Developing polyglot persistence applications (devnexus 2013)
Developing polyglot persistence applications (devnexus 2013)Developing polyglot persistence applications (devnexus 2013)
Developing polyglot persistence applications (devnexus 2013)Chris Richardson
 
Developing applications with Cloud Services (Devnexus 2013)
Developing applications with Cloud Services (Devnexus 2013)Developing applications with Cloud Services (Devnexus 2013)
Developing applications with Cloud Services (Devnexus 2013)Chris Richardson
 
Developing applications with Cloud Services #javaone 2012
Developing applications with Cloud Services  #javaone 2012Developing applications with Cloud Services  #javaone 2012
Developing applications with Cloud Services #javaone 2012Chris Richardson
 
Developing polyglot applications on Cloud Foundry (#oredev 2012)
Developing polyglot applications on Cloud Foundry (#oredev 2012)Developing polyglot applications on Cloud Foundry (#oredev 2012)
Developing polyglot applications on Cloud Foundry (#oredev 2012)Chris Richardson
 
Developing polyglot persistence applications #javaone 2012
Developing polyglot persistence applications  #javaone 2012Developing polyglot persistence applications  #javaone 2012
Developing polyglot persistence applications #javaone 2012Chris Richardson
 
Decomposing applications for scalability and deployability - svcc sv_code_ca...
Decomposing applications for scalability and deployability  - svcc sv_code_ca...Decomposing applications for scalability and deployability  - svcc sv_code_ca...
Decomposing applications for scalability and deployability - svcc sv_code_ca...Chris Richardson
 
Developing polyglot persistence applications (SpringOne India 2012)
Developing polyglot persistence applications (SpringOne India 2012)Developing polyglot persistence applications (SpringOne India 2012)
Developing polyglot persistence applications (SpringOne India 2012)Chris Richardson
 
Jumping-with-java8
Jumping-with-java8Jumping-with-java8
Jumping-with-java8Dhaval Dalal
 
Using Spring with NoSQL databases (SpringOne China 2012)
Using Spring with NoSQL databases (SpringOne China 2012)Using Spring with NoSQL databases (SpringOne China 2012)
Using Spring with NoSQL databases (SpringOne China 2012)Chris Richardson
 

Andere mochten auch (20)

Map, Flatmap and Reduce are Your New Best Friends: Simpler Collections, Concu...
Map, Flatmap and Reduce are Your New Best Friends: Simpler Collections, Concu...Map, Flatmap and Reduce are Your New Best Friends: Simpler Collections, Concu...
Map, Flatmap and Reduce are Your New Best Friends: Simpler Collections, Concu...
 
Carta perĂș sabor
Carta perĂș saborCarta perĂș sabor
Carta perĂș sabor
 
Why we cannot ignore Functional Programming
Why we cannot ignore Functional ProgrammingWhy we cannot ignore Functional Programming
Why we cannot ignore Functional Programming
 
OOP and FP - Become a Better Programmer
OOP and FP - Become a Better ProgrammerOOP and FP - Become a Better Programmer
OOP and FP - Become a Better Programmer
 
Reactive Programming for a demanding world: building event-driven and respons...
Reactive Programming for a demanding world: building event-driven and respons...Reactive Programming for a demanding world: building event-driven and respons...
Reactive Programming for a demanding world: building event-driven and respons...
 
Developing microservices with aggregates (devnexus2017)
Developing microservices with aggregates (devnexus2017)Developing microservices with aggregates (devnexus2017)
Developing microservices with aggregates (devnexus2017)
 
WebCamp:Front-end Developers Day. АлДĐșŃĐ°ĐœĐŽŃ€ ĐœĐŸŃŃ‚ĐŸĐČĐ”ĐœĐșĐŸ "Rx.js - ĐŽĐ”Đ»Đ°Đ”ĐŒ Đ°ŃĐžĐœŃ…Ń€...
WebCamp:Front-end Developers Day. АлДĐșŃĐ°ĐœĐŽŃ€ ĐœĐŸŃŃ‚ĐŸĐČĐ”ĐœĐșĐŸ "Rx.js - ĐŽĐ”Đ»Đ°Đ”ĐŒ Đ°ŃĐžĐœŃ…Ń€...WebCamp:Front-end Developers Day. АлДĐșŃĐ°ĐœĐŽŃ€ ĐœĐŸŃŃ‚ĐŸĐČĐ”ĐœĐșĐŸ "Rx.js - ĐŽĐ”Đ»Đ°Đ”ĐŒ Đ°ŃĐžĐœŃ…Ń€...
WebCamp:Front-end Developers Day. АлДĐșŃĐ°ĐœĐŽŃ€ ĐœĐŸŃŃ‚ĐŸĐČĐ”ĐœĐșĐŸ "Rx.js - ĐŽĐ”Đ»Đ°Đ”ĐŒ Đ°ŃĐžĐœŃ…Ń€...
 
GKAC 2014 Nov. - ê·žëŁšëč„ëĄœ ì•ˆë“œëĄœìŽë“œ 앱 개발하Ʞ
GKAC 2014 Nov. - ê·žëŁšëč„ëĄœ ì•ˆë“œëĄœìŽë“œ 앱 개발하ꞰGKAC 2014 Nov. - ê·žëŁšëč„ëĄœ ì•ˆë“œëĄœìŽë“œ 앱 개발하Ʞ
GKAC 2014 Nov. - ê·žëŁšëč„ëĄœ ì•ˆë“œëĄœìŽë“œ 앱 개발하Ʞ
 
Solit 2014, РДаĐșтоĐČĐœŃ‹Đč Javascript. ĐŸĐŸĐ±Đ”ĐŽĐ° ĐœĐ°ĐŽ Đ°ŃĐžĐœŃ…Ń€ĐŸĐœĐœĐŸŃŃ‚ŃŒŃŽ Đž ĐČĐ»ĐŸĐ¶Đ”ĐœĐœĐŸŃŃ‚ŃŒŃŽ, ...
Solit 2014, РДаĐșтоĐČĐœŃ‹Đč Javascript. ĐŸĐŸĐ±Đ”ĐŽĐ° ĐœĐ°ĐŽ Đ°ŃĐžĐœŃ…Ń€ĐŸĐœĐœĐŸŃŃ‚ŃŒŃŽ Đž ĐČĐ»ĐŸĐ¶Đ”ĐœĐœĐŸŃŃ‚ŃŒŃŽ, ...Solit 2014, РДаĐșтоĐČĐœŃ‹Đč Javascript. ĐŸĐŸĐ±Đ”ĐŽĐ° ĐœĐ°ĐŽ Đ°ŃĐžĐœŃ…Ń€ĐŸĐœĐœĐŸŃŃ‚ŃŒŃŽ Đž ĐČĐ»ĐŸĐ¶Đ”ĐœĐœĐŸŃŃ‚ŃŒŃŽ, ...
Solit 2014, РДаĐșтоĐČĐœŃ‹Đč Javascript. ĐŸĐŸĐ±Đ”ĐŽĐ° ĐœĐ°ĐŽ Đ°ŃĐžĐœŃ…Ń€ĐŸĐœĐœĐŸŃŃ‚ŃŒŃŽ Đž ĐČĐ»ĐŸĐ¶Đ”ĐœĐœĐŸŃŃ‚ŃŒŃŽ, ...
 
Decomposing applications for deployability and scalability (SpringOne China 2...
Decomposing applications for deployability and scalability (SpringOne China 2...Decomposing applications for deployability and scalability (SpringOne China 2...
Decomposing applications for deployability and scalability (SpringOne China 2...
 
Polygot persistence for Java Developers - August 2011 / @Oakjug
Polygot persistence for Java Developers - August 2011 / @OakjugPolygot persistence for Java Developers - August 2011 / @Oakjug
Polygot persistence for Java Developers - August 2011 / @Oakjug
 
Developing polyglot persistence applications (devnexus 2013)
Developing polyglot persistence applications (devnexus 2013)Developing polyglot persistence applications (devnexus 2013)
Developing polyglot persistence applications (devnexus 2013)
 
Developing applications with Cloud Services (Devnexus 2013)
Developing applications with Cloud Services (Devnexus 2013)Developing applications with Cloud Services (Devnexus 2013)
Developing applications with Cloud Services (Devnexus 2013)
 
Developing applications with Cloud Services #javaone 2012
Developing applications with Cloud Services  #javaone 2012Developing applications with Cloud Services  #javaone 2012
Developing applications with Cloud Services #javaone 2012
 
Developing polyglot applications on Cloud Foundry (#oredev 2012)
Developing polyglot applications on Cloud Foundry (#oredev 2012)Developing polyglot applications on Cloud Foundry (#oredev 2012)
Developing polyglot applications on Cloud Foundry (#oredev 2012)
 
Developing polyglot persistence applications #javaone 2012
Developing polyglot persistence applications  #javaone 2012Developing polyglot persistence applications  #javaone 2012
Developing polyglot persistence applications #javaone 2012
 
Decomposing applications for scalability and deployability - svcc sv_code_ca...
Decomposing applications for scalability and deployability  - svcc sv_code_ca...Decomposing applications for scalability and deployability  - svcc sv_code_ca...
Decomposing applications for scalability and deployability - svcc sv_code_ca...
 
Developing polyglot persistence applications (SpringOne India 2012)
Developing polyglot persistence applications (SpringOne India 2012)Developing polyglot persistence applications (SpringOne India 2012)
Developing polyglot persistence applications (SpringOne India 2012)
 
Jumping-with-java8
Jumping-with-java8Jumping-with-java8
Jumping-with-java8
 
Using Spring with NoSQL databases (SpringOne China 2012)
Using Spring with NoSQL databases (SpringOne China 2012)Using Spring with NoSQL databases (SpringOne China 2012)
Using Spring with NoSQL databases (SpringOne China 2012)
 

Ähnlich wie MAP, FILTER, REDUCE: Functional programming simplifies collections

Map, flatmap and reduce are your new best friends (javaone, svcc)
Map, flatmap and reduce are your new best friends (javaone, svcc)Map, flatmap and reduce are your new best friends (javaone, svcc)
Map, flatmap and reduce are your new best friends (javaone, svcc)Chris Richardson
 
Transformations and actions a visual guide training
Transformations and actions a visual guide trainingTransformations and actions a visual guide training
Transformations and actions a visual guide trainingSpark Summit
 
R for Pythonistas (PyData NYC 2017)
R for Pythonistas (PyData NYC 2017)R for Pythonistas (PyData NYC 2017)
R for Pythonistas (PyData NYC 2017)Christopher Roach
 
NodeJS: the good parts? A skeptic’s view (jax jax2013)
NodeJS: the good parts? A skeptic’s view (jax jax2013)NodeJS: the good parts? A skeptic’s view (jax jax2013)
NodeJS: the good parts? A skeptic’s view (jax jax2013)Chris Richardson
 
What's New In C# 7
What's New In C# 7What's New In C# 7
What's New In C# 7Paulo Morgado
 
End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017
End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017
End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017StampedeCon
 
Cpp tutorial
Cpp tutorialCpp tutorial
Cpp tutorialVikas Sharma
 
CppTutorial.ppt
CppTutorial.pptCppTutorial.ppt
CppTutorial.pptHODZoology3
 
C++ oop
C++ oopC++ oop
C++ oopSunil OS
 
How to deal with nested lists in R?
How to deal with nested lists in R? How to deal with nested lists in R?
How to deal with nested lists in R? Sotrender
 
Poetry with R -- Dissecting the code
Poetry with R -- Dissecting the codePoetry with R -- Dissecting the code
Poetry with R -- Dissecting the codePeter Solymos
 
SparkR - Play Spark Using R (20160909 HadoopCon)
SparkR - Play Spark Using R (20160909 HadoopCon)SparkR - Play Spark Using R (20160909 HadoopCon)
SparkR - Play Spark Using R (20160909 HadoopCon)wqchen
 
Spark - Philly JUG
Spark  - Philly JUGSpark  - Philly JUG
Spark - Philly JUGBrian O'Neill
 
Kotlin Developer Starter in Android projects
Kotlin Developer Starter in Android projectsKotlin Developer Starter in Android projects
Kotlin Developer Starter in Android projectsBartosz Kosarzycki
 
Kotlin Developer Starter in Android - STX Next Lightning Talks - Feb 12, 2016
Kotlin Developer Starter in Android - STX Next Lightning Talks - Feb 12, 2016Kotlin Developer Starter in Android - STX Next Lightning Talks - Feb 12, 2016
Kotlin Developer Starter in Android - STX Next Lightning Talks - Feb 12, 2016STX Next
 
Going reactive in java
Going reactive in javaGoing reactive in java
Going reactive in javaJosé Paumard
 
Mastering Hadoop Map Reduce - Custom Types and Other Optimizations
Mastering Hadoop Map Reduce - Custom Types and Other OptimizationsMastering Hadoop Map Reduce - Custom Types and Other Optimizations
Mastering Hadoop Map Reduce - Custom Types and Other Optimizationsscottcrespo
 
Data Analysis in Python
Data Analysis in PythonData Analysis in Python
Data Analysis in PythonRichard Herrell
 
All I know about rsc.io/c2go
All I know about rsc.io/c2goAll I know about rsc.io/c2go
All I know about rsc.io/c2goMoriyoshi Koizumi
 
Soft Shake Event / A soft introduction to Neo4J
Soft Shake Event / A soft introduction to Neo4JSoft Shake Event / A soft introduction to Neo4J
Soft Shake Event / A soft introduction to Neo4JFlorent Biville
 

Ähnlich wie MAP, FILTER, REDUCE: Functional programming simplifies collections (20)

Map, flatmap and reduce are your new best friends (javaone, svcc)
Map, flatmap and reduce are your new best friends (javaone, svcc)Map, flatmap and reduce are your new best friends (javaone, svcc)
Map, flatmap and reduce are your new best friends (javaone, svcc)
 
Transformations and actions a visual guide training
Transformations and actions a visual guide trainingTransformations and actions a visual guide training
Transformations and actions a visual guide training
 
R for Pythonistas (PyData NYC 2017)
R for Pythonistas (PyData NYC 2017)R for Pythonistas (PyData NYC 2017)
R for Pythonistas (PyData NYC 2017)
 
NodeJS: the good parts? A skeptic’s view (jax jax2013)
NodeJS: the good parts? A skeptic’s view (jax jax2013)NodeJS: the good parts? A skeptic’s view (jax jax2013)
NodeJS: the good parts? A skeptic’s view (jax jax2013)
 
What's New In C# 7
What's New In C# 7What's New In C# 7
What's New In C# 7
 
End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017
End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017
End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017
 
Cpp tutorial
Cpp tutorialCpp tutorial
Cpp tutorial
 
CppTutorial.ppt
CppTutorial.pptCppTutorial.ppt
CppTutorial.ppt
 
C++ oop
C++ oopC++ oop
C++ oop
 
How to deal with nested lists in R?
How to deal with nested lists in R? How to deal with nested lists in R?
How to deal with nested lists in R?
 
Poetry with R -- Dissecting the code
Poetry with R -- Dissecting the codePoetry with R -- Dissecting the code
Poetry with R -- Dissecting the code
 
SparkR - Play Spark Using R (20160909 HadoopCon)
SparkR - Play Spark Using R (20160909 HadoopCon)SparkR - Play Spark Using R (20160909 HadoopCon)
SparkR - Play Spark Using R (20160909 HadoopCon)
 
Spark - Philly JUG
Spark  - Philly JUGSpark  - Philly JUG
Spark - Philly JUG
 
Kotlin Developer Starter in Android projects
Kotlin Developer Starter in Android projectsKotlin Developer Starter in Android projects
Kotlin Developer Starter in Android projects
 
Kotlin Developer Starter in Android - STX Next Lightning Talks - Feb 12, 2016
Kotlin Developer Starter in Android - STX Next Lightning Talks - Feb 12, 2016Kotlin Developer Starter in Android - STX Next Lightning Talks - Feb 12, 2016
Kotlin Developer Starter in Android - STX Next Lightning Talks - Feb 12, 2016
 
Going reactive in java
Going reactive in javaGoing reactive in java
Going reactive in java
 
Mastering Hadoop Map Reduce - Custom Types and Other Optimizations
Mastering Hadoop Map Reduce - Custom Types and Other OptimizationsMastering Hadoop Map Reduce - Custom Types and Other Optimizations
Mastering Hadoop Map Reduce - Custom Types and Other Optimizations
 
Data Analysis in Python
Data Analysis in PythonData Analysis in Python
Data Analysis in Python
 
All I know about rsc.io/c2go
All I know about rsc.io/c2goAll I know about rsc.io/c2go
All I know about rsc.io/c2go
 
Soft Shake Event / A soft introduction to Neo4J
Soft Shake Event / A soft introduction to Neo4JSoft Shake Event / A soft introduction to Neo4J
Soft Shake Event / A soft introduction to Neo4J
 

Mehr von Chris Richardson

The microservice architecture: what, why, when and how?
The microservice architecture: what, why, when and how?The microservice architecture: what, why, when and how?
The microservice architecture: what, why, when and how?Chris Richardson
 
More the merrier: a microservices anti-pattern
More the merrier: a microservices anti-patternMore the merrier: a microservices anti-pattern
More the merrier: a microservices anti-patternChris Richardson
 
YOW London - Considering Migrating a Monolith to Microservices? A Dark Energy...
YOW London - Considering Migrating a Monolith to Microservices? A Dark Energy...YOW London - Considering Migrating a Monolith to Microservices? A Dark Energy...
YOW London - Considering Migrating a Monolith to Microservices? A Dark Energy...Chris Richardson
 
Dark Energy, Dark Matter and the Microservices Patterns?!
Dark Energy, Dark Matter and the Microservices Patterns?!Dark Energy, Dark Matter and the Microservices Patterns?!
Dark Energy, Dark Matter and the Microservices Patterns?!Chris Richardson
 
Dark energy, dark matter and microservice architecture collaboration patterns
Dark energy, dark matter and microservice architecture collaboration patternsDark energy, dark matter and microservice architecture collaboration patterns
Dark energy, dark matter and microservice architecture collaboration patternsChris Richardson
 
Scenarios_and_Architecture_SkillsMatter_April_2022.pdf
Scenarios_and_Architecture_SkillsMatter_April_2022.pdfScenarios_and_Architecture_SkillsMatter_April_2022.pdf
Scenarios_and_Architecture_SkillsMatter_April_2022.pdfChris Richardson
 
Using patterns and pattern languages to make better architectural decisions
Using patterns and pattern languages to make better architectural decisions Using patterns and pattern languages to make better architectural decisions
Using patterns and pattern languages to make better architectural decisions Chris Richardson
 
iSAQB gathering 2021 keynote - Architectural patterns for rapid, reliable, fr...
iSAQB gathering 2021 keynote - Architectural patterns for rapid, reliable, fr...iSAQB gathering 2021 keynote - Architectural patterns for rapid, reliable, fr...
iSAQB gathering 2021 keynote - Architectural patterns for rapid, reliable, fr...Chris Richardson
 
Events to the rescue: solving distributed data problems in a microservice arc...
Events to the rescue: solving distributed data problems in a microservice arc...Events to the rescue: solving distributed data problems in a microservice arc...
Events to the rescue: solving distributed data problems in a microservice arc...Chris Richardson
 
A pattern language for microservices - June 2021
A pattern language for microservices - June 2021 A pattern language for microservices - June 2021
A pattern language for microservices - June 2021 Chris Richardson
 
QConPlus 2021: Minimizing Design Time Coupling in a Microservice Architecture
QConPlus 2021: Minimizing Design Time Coupling in a Microservice ArchitectureQConPlus 2021: Minimizing Design Time Coupling in a Microservice Architecture
QConPlus 2021: Minimizing Design Time Coupling in a Microservice ArchitectureChris Richardson
 
Mucon 2021 - Dark energy, dark matter: imperfect metaphors for designing micr...
Mucon 2021 - Dark energy, dark matter: imperfect metaphors for designing micr...Mucon 2021 - Dark energy, dark matter: imperfect metaphors for designing micr...
Mucon 2021 - Dark energy, dark matter: imperfect metaphors for designing micr...Chris Richardson
 
Designing loosely coupled services
Designing loosely coupled servicesDesigning loosely coupled services
Designing loosely coupled servicesChris Richardson
 
Microservices - an architecture that enables DevOps (T Systems DevOps day)
Microservices - an architecture that enables DevOps (T Systems DevOps day)Microservices - an architecture that enables DevOps (T Systems DevOps day)
Microservices - an architecture that enables DevOps (T Systems DevOps day)Chris Richardson
 
DDD SoCal: Decompose your monolith: Ten principles for refactoring a monolith...
DDD SoCal: Decompose your monolith: Ten principles for refactoring a monolith...DDD SoCal: Decompose your monolith: Ten principles for refactoring a monolith...
DDD SoCal: Decompose your monolith: Ten principles for refactoring a monolith...Chris Richardson
 
Decompose your monolith: Six principles for refactoring a monolith to microse...
Decompose your monolith: Six principles for refactoring a monolith to microse...Decompose your monolith: Six principles for refactoring a monolith to microse...
Decompose your monolith: Six principles for refactoring a monolith to microse...Chris Richardson
 
TDC2020 - The microservice architecture: enabling rapid, reliable, frequent a...
TDC2020 - The microservice architecture: enabling rapid, reliable, frequent a...TDC2020 - The microservice architecture: enabling rapid, reliable, frequent a...
TDC2020 - The microservice architecture: enabling rapid, reliable, frequent a...Chris Richardson
 
Overview of the Eventuate Tram Customers and Orders application
Overview of the Eventuate Tram Customers and Orders applicationOverview of the Eventuate Tram Customers and Orders application
Overview of the Eventuate Tram Customers and Orders applicationChris Richardson
 
An overview of the Eventuate Platform
An overview of the Eventuate PlatformAn overview of the Eventuate Platform
An overview of the Eventuate PlatformChris Richardson
 
#DevNexus202 Decompose your monolith
#DevNexus202 Decompose your monolith#DevNexus202 Decompose your monolith
#DevNexus202 Decompose your monolithChris Richardson
 

Mehr von Chris Richardson (20)

The microservice architecture: what, why, when and how?
The microservice architecture: what, why, when and how?The microservice architecture: what, why, when and how?
The microservice architecture: what, why, when and how?
 
More the merrier: a microservices anti-pattern
More the merrier: a microservices anti-patternMore the merrier: a microservices anti-pattern
More the merrier: a microservices anti-pattern
 
YOW London - Considering Migrating a Monolith to Microservices? A Dark Energy...
YOW London - Considering Migrating a Monolith to Microservices? A Dark Energy...YOW London - Considering Migrating a Monolith to Microservices? A Dark Energy...
YOW London - Considering Migrating a Monolith to Microservices? A Dark Energy...
 
Dark Energy, Dark Matter and the Microservices Patterns?!
Dark Energy, Dark Matter and the Microservices Patterns?!Dark Energy, Dark Matter and the Microservices Patterns?!
Dark Energy, Dark Matter and the Microservices Patterns?!
 
Dark energy, dark matter and microservice architecture collaboration patterns
Dark energy, dark matter and microservice architecture collaboration patternsDark energy, dark matter and microservice architecture collaboration patterns
Dark energy, dark matter and microservice architecture collaboration patterns
 
Scenarios_and_Architecture_SkillsMatter_April_2022.pdf
Scenarios_and_Architecture_SkillsMatter_April_2022.pdfScenarios_and_Architecture_SkillsMatter_April_2022.pdf
Scenarios_and_Architecture_SkillsMatter_April_2022.pdf
 
Using patterns and pattern languages to make better architectural decisions
Using patterns and pattern languages to make better architectural decisions Using patterns and pattern languages to make better architectural decisions
Using patterns and pattern languages to make better architectural decisions
 
iSAQB gathering 2021 keynote - Architectural patterns for rapid, reliable, fr...
iSAQB gathering 2021 keynote - Architectural patterns for rapid, reliable, fr...iSAQB gathering 2021 keynote - Architectural patterns for rapid, reliable, fr...
iSAQB gathering 2021 keynote - Architectural patterns for rapid, reliable, fr...
 
Events to the rescue: solving distributed data problems in a microservice arc...
Events to the rescue: solving distributed data problems in a microservice arc...Events to the rescue: solving distributed data problems in a microservice arc...
Events to the rescue: solving distributed data problems in a microservice arc...
 
A pattern language for microservices - June 2021
A pattern language for microservices - June 2021 A pattern language for microservices - June 2021
A pattern language for microservices - June 2021
 
QConPlus 2021: Minimizing Design Time Coupling in a Microservice Architecture
QConPlus 2021: Minimizing Design Time Coupling in a Microservice ArchitectureQConPlus 2021: Minimizing Design Time Coupling in a Microservice Architecture
QConPlus 2021: Minimizing Design Time Coupling in a Microservice Architecture
 
Mucon 2021 - Dark energy, dark matter: imperfect metaphors for designing micr...
Mucon 2021 - Dark energy, dark matter: imperfect metaphors for designing micr...Mucon 2021 - Dark energy, dark matter: imperfect metaphors for designing micr...
Mucon 2021 - Dark energy, dark matter: imperfect metaphors for designing micr...
 
Designing loosely coupled services
Designing loosely coupled servicesDesigning loosely coupled services
Designing loosely coupled services
 
Microservices - an architecture that enables DevOps (T Systems DevOps day)
Microservices - an architecture that enables DevOps (T Systems DevOps day)Microservices - an architecture that enables DevOps (T Systems DevOps day)
Microservices - an architecture that enables DevOps (T Systems DevOps day)
 
DDD SoCal: Decompose your monolith: Ten principles for refactoring a monolith...
DDD SoCal: Decompose your monolith: Ten principles for refactoring a monolith...DDD SoCal: Decompose your monolith: Ten principles for refactoring a monolith...
DDD SoCal: Decompose your monolith: Ten principles for refactoring a monolith...
 
Decompose your monolith: Six principles for refactoring a monolith to microse...
Decompose your monolith: Six principles for refactoring a monolith to microse...Decompose your monolith: Six principles for refactoring a monolith to microse...
Decompose your monolith: Six principles for refactoring a monolith to microse...
 
TDC2020 - The microservice architecture: enabling rapid, reliable, frequent a...
TDC2020 - The microservice architecture: enabling rapid, reliable, frequent a...TDC2020 - The microservice architecture: enabling rapid, reliable, frequent a...
TDC2020 - The microservice architecture: enabling rapid, reliable, frequent a...
 
Overview of the Eventuate Tram Customers and Orders application
Overview of the Eventuate Tram Customers and Orders applicationOverview of the Eventuate Tram Customers and Orders application
Overview of the Eventuate Tram Customers and Orders application
 
An overview of the Eventuate Platform
An overview of the Eventuate PlatformAn overview of the Eventuate Platform
An overview of the Eventuate Platform
 
#DevNexus202 Decompose your monolith
#DevNexus202 Decompose your monolith#DevNexus202 Decompose your monolith
#DevNexus202 Decompose your monolith
 

KĂŒrzlich hochgeladen

Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsTop 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsRoshan Dwivedi
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processorsdebabhi2
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdfhans926745
 
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking MenDelhi Call girls
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationRadu Cotescu
 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptxHampshireHUG
 
Developing An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of BrazilDeveloping An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of BrazilV3cube
 
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking MenDelhi Call girls
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024The Digital Insurer
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonetsnaman860154
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc
 
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍾 8923113531 🎰 Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍾 8923113531 🎰 Avail...Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍾 8923113531 🎰 Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍾 8923113531 🎰 Avail...gurkirankumar98700
 
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure serviceWhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure servicePooja Nehwal
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityPrincipled Technologies
 
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfThe Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfEnterprise Knowledge
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)Gabriella Davis
 
Salesforce Community Group Quito, Salesforce 101
Salesforce Community Group Quito, Salesforce 101Salesforce Community Group Quito, Salesforce 101
Salesforce Community Group Quito, Salesforce 101Paola De la Torre
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking MenDelhi Call girls
 
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxMalak Abu Hammad
 

KĂŒrzlich hochgeladen (20)

Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsTop 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
 
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
 
Developing An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of BrazilDeveloping An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of Brazil
 
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonets
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
 
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍾 8923113531 🎰 Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍾 8923113531 🎰 Avail...Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍾 8923113531 🎰 Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍾 8923113531 🎰 Avail...
 
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure serviceWhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfThe Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 
Salesforce Community Group Quito, Salesforce 101
Salesforce Community Group Quito, Salesforce 101Salesforce Community Group Quito, Salesforce 101
Salesforce Community Group Quito, Salesforce 101
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
 
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptx
 

MAP, FILTER, REDUCE: Functional programming simplifies collections

  • 1. @crichardson Map(), ïŹ‚atMap() and reduce() are your new best friends: Simpler collections, concurrency, and big data Chris Richardson Author of POJOs in Action Founder of the original CloudFoundry.com @crichardson chris@chrisrichardson.net http://plainoldobjects.com
  • 2. @crichardson Presentation goal How functional programming simpliïŹes your code Show that map(), ïŹ‚atMap() and reduce() are remarkably versatile functions
  • 4. @crichardson About Chris Founder of a buzzword compliant (stealthy, social, mobile, big data, machine learning, ...) startup Consultant helping organizations improve how they architect and deploy applications using cloud, micro services, polyglot applications, NoSQL, ...
  • 5. @crichardson Agenda Why functional programming? Simplifying collection processing Eliminating NullPointerExceptions Simplifying concurrency with Futures and Rx Observables Tackling big data problems with functional programming
  • 8. @crichardson It’s a kind of programming language
  • 9. @crichardson Functions as the building blocks of the application
  • 10. @crichardson Functions as ïŹrst class citizens Assign functions to variables Store functions in ïŹelds Use and write higher-order functions: Pass functions as arguments Return functions as values
  • 11. @crichardson Avoids mutable state Use: Immutable data structures Single assignment variables Some functional languages such as Haskell don’t side-effects There are beneïŹts to immutability Easier concurrency More reliable code But be pragmatic
  • 12. @crichardson Why functional programming? "the highest goal of programming-language design to enable good ideas to be elegantly expressed" http://en.wikipedia.org/wiki/Tony_Hoare
  • 13. @crichardson Why functional programming? More expressive More concise More intuitive - solution matches problem deïŹnition Elimination of error-prone mutable state Easy parallelization
  • 14. @crichardson An ancient idea that has recently become popular
  • 16. @crichardson Lisp = an early functional language invented in 1958 http://en.wikipedia.org/wiki/ Lisp_(programming_language) 1940 1950 1960 1970 1980 1990 2000 2010 garbage collection dynamic typing self-hosting compiler tree data structures (defun factorial (n) (if (<= n 1) 1 (* n (factorial (- n 1)))))
  • 17. @crichardson My ïŹnal year project in 1985: Implementing SASL sieve (p:xs) = p : sieve [x | x <- xs, rem x p > 0]; primes = sieve [2..] A list of integers starting with 2 Filter out multiples of p
  • 18. Mostly an Ivory Tower technology Lisp was used for AI FP languages: Miranda, ML, Haskell, ... “Side-eïŹ€ects kills kittens and puppies”
  • 20. @crichardson But today FP is mainstream Clojure - a dialect of Lisp A hybrid OO/functional language A hybrid OO/FP language for .NET Java 8 has lambda expressions
  • 21. @crichardson Java 8 lambda expressions are functions x -> x * x x -> { for (int i = 2; i < Math.sqrt(x); i = i + 1) { if (x % i == 0) return false; } return true; }; (x, y) -> x * x + y * y
  • 22. @crichardson Java 8 lambdas are a shorthand* for an anonymous inner class * not exactly. See http://programmers.stackexchange.com/questions/ 177879/type-inference-in-java-8
  • 23. @crichardson Java 8 functional interfaces Interface with a single abstract method e.g. Runnable, Callable, Spring’s TransactionCallback A lambda expression is an instance of a functional interface. You can use a lambda wherever a function interface “value” is expected The type of the lambda expression is determined from it’s context
  • 24. @crichardson Example Functional Interface Function<Integer, Integer> square = x -> x * x; BiFunction<Integer, Integer, Integer> sumSquares = (x, y) -> x * x + y * y; Predicate<Integer> makeIsDivisibleBy(int y) { return x -> x % y == 0; } Predicate<Integer> isEven = makeIsDivisibleBy(2); Assert.assertTrue(isEven.test(8)); Assert.assertFalse(isEven.test(11));
  • 25. @crichardson Example Functional Interface ExecutorService executor = ...; final int x = 999 Future<Boolean> outcome = executor.submit(() -> { for (int i = 2; i < Math.sqrt(x); i = i + 1) { if (x % i == 0) return false; } return true; } This lambda is a Callable
  • 26. @crichardson Agenda Why functional programming? Simplifying collection processing Eliminating NullPointerExceptions Simplifying concurrency with Futures and Rx Observables Tackling big data problems with functional programming
  • 27. @crichardson Lot’s of application code = collection processing: Mapping, ïŹltering, and reducing
  • 28. @crichardson Social network example public class Person { enum Gender { MALE, FEMALE } private Name name; private LocalDate birthday; private Gender gender; private Hometown hometown; private Set<Friend> friends = new HashSet<Friend>(); .... public class Friend { private Person friend; private LocalDate becameFriends; ... } public class SocialNetwork { private Set<Person> people; ...
  • 29. @crichardson Mapping, ïŹltering, and reducing public class Person { public Set<Hometown> hometownsOfFriends() { Set<Hometown> result = new HashSet<>(); for (Friend friend : friends) { result.add(friend.getPerson().getHometown()); } return result; }
  • 30. @crichardson Mapping, ïŹltering, and reducing public class Person { public Set<Person> friendOfFriends() { Set<Person> result = new HashSet(); for (Friend friend : friends) for (Friend friendOfFriend : friend.getPerson().friends) if (friendOfFriend.getPerson() != this) result.add(friendOfFriend.getPerson()); return result; }
  • 31. @crichardson Mapping, ïŹltering, and reducing public class SocialNetwork { private Set<Person> people; ... public Set<Person> lonelyPeople() { Set<Person> result = new HashSet<Person>(); for (Person p : people) { if (p.getFriends().isEmpty()) result.add(p); } return result; }
  • 32. @crichardson Mapping, ïŹltering, and reducing public class SocialNetwork { private Set<Person> people; ... public int averageNumberOfFriends() { int sum = 0; for (Person p : people) { sum += p.getFriends().size(); } return sum / people.size(); }
  • 33. @crichardson Problems with this style of programming Low level Imperative (how to do it) NOT declarative (what to do) Verbose Mutable variables are potentially error prone DifïŹcult to parallelize
  • 34. @crichardson Java 8 streams to the rescue A sequence of elements “Wrapper” around a collection Streams can also be inïŹnite Provides a functional/lambda-based API for transforming, ïŹltering and aggregating elements Much simpler, cleaner code
  • 35. @crichardson Using Java 8 streams - mapping class Person .. private Set<Friend> friends = ...; public Set<Hometown> hometownsOfFriends() { return friends.stream() .map(f -> f.getPerson().getHometown()) .collect(Collectors.toSet()); }
  • 36. @crichardson The map() function s1 a b c d e ... s2 f(a) f(b) f(c) f(d) f(e) ... s2 = s1.map(f)
  • 37. @crichardson public class SocialNetwork { private Set<Person> people; ... public Set<Person> peopleWithNoFriends() { Set<Person> result = new HashSet<Person>(); for (Person p : people) { if (p.getFriends().isEmpty()) result.add(p); } return result; } Using Java 8 streams - ïŹltering public class SocialNetwork { private Set<Person> people; ... public Set<Person> lonelyPeople() { return people.stream() .filter(p -> p.getFriends().isEmpty()) .collect(Collectors.toSet()); }
  • 38. @crichardson Using Java 8 streams - friend of friends V1 class Person .. public Set<Person> friendOfFriends() { Set<Set<Friend>> fof = friends.stream() .map(friend -> friend.getPerson().friends) .collect(Collectors.toSet()); ... } Using map() => Set of Sets :-( Somehow we need to flatten
  • 39. @crichardson Using Java 8 streams - mapping class Person .. public Set<Person> friendOfFriends() { return friends.stream() .flatMap(friend -> friend.getPerson().friends.stream()) .map(Friend::getPerson) .filter(f -> f != this) .collect(Collectors.toSet()); } maps and ïŹ‚attens
  • 40. @crichardson The ïŹ‚atMap() function s1 a b ... s2 f(a)0 f(a)1 f(b)0 f(b)1 f(b)2 ... s2 = s1.ïŹ‚atMap(f)
  • 41. @crichardson Using Java 8 streams - reducing public class SocialNetwork { private Set<Person> people; ... public long averageNumberOfFriends() { return people.stream() .map ( p -> p.getFriends().size() ) .reduce(0, (x, y) -> x + y) / people.size(); } int x = 0; for (int y : inputStream) x = x + y return x;
  • 42. @crichardson The reduce() function s1 a b c d e ... x = s1.reduce(initial, f) f(f(f(f(f(f(initial, a), b), c), d), e), ...)
  • 43. @crichardson Newton's method for ïŹnding square roots public class SquareRootCalculator { public double squareRoot(double input, double precision) { return Stream.iterate( new Result(1.0), current -> refine(current, input, precision)) .filter(r -> r.done) .findFirst().get().value; } private static Result refine(Result current, double input, double precision) { double value = current.value; double newCurrent = value - (value * value - input) / (2 * value); boolean done = Math.abs(value - newCurrent) < precision; return new Result(newCurrent, done); } class Result { boolean done; double value; } Creates an inïŹnite stream: seed, f(seed), f(f(seed)), ..... Don’t panic! Streams are lazy
  • 44. @crichardson Agenda Why functional programming? Simplifying collection processing Eliminating NullPointerExceptions Simplifying concurrency with Futures and Rx Observables Tackling big data problems with functional programming
  • 45. @crichardson Tony’s $1B mistake “I call it my billion-dollar mistake. It was the invention of the null reference in 1965....But I couldn't resist the temptation to put in a null reference, simply because it was so easy to implement...” http://qconlondon.com/london-2009/presentation/ Null+References:+The+Billion+Dollar+Mistake
  • 46. @crichardson Coding with null pointers class Person public Friend longestFriendship() { Friend result = null; for (Friend friend : friends) { if (result == null || friend.getBecameFriends() .isBefore(result.getBecameFriends())) result = friend; } return result; } Friend oldestFriend = person.longestFriendship(); if (oldestFriend != null) { ... } else { ... } Null check is essential yet easily forgotten
  • 47. @crichardson Java 8 Optional<T> A wrapper for nullable references It has two states: empty throws an exception if you try to get the reference non-empty contain a non-null reference Provides methods for: testing whether it has a value getting the value ... Return reference wrapped in an instance of this type instead of null
  • 48. @crichardson Coding with optionals class Person public Optional<Friend> longestFriendship() { Friend result = null; for (Friend friend : friends) { if (result == null || friend.getBecameFriends().isBefore(result.getBecameFriends())) result = friend; } return Optional.ofNullable(result); } Optional<Friend> oldestFriend = person.longestFriendship(); // Might throw java.util.NoSuchElementException: No value present // Person dangerous = popularPerson.get(); if (oldestFriend.isPresent) { ...oldestFriend.get() } else { ... }
  • 49. @crichardson Using Optionals - better Optional<Friend> oldestFriendship = ...; Friend whoToCall1 = oldestFriendship.orElse(mother); Avoid calling isPresent() and get() Friend whoToCall3 = oldestFriendship.orElseThrow( () -> new LonelyPersonException()); Friend whoToCall2 = oldestFriendship.orElseGet(() -> lazilyFindSomeoneElse());
  • 50. @crichardson Using Optional.map() public class Person { public Optional<Friend> longestFriendship() { return ...; } public Optional<Long> ageDifferenceWithOldestFriend() { Optional<Friend> oldestFriend = longestFriendship(); return oldestFriend.map ( of -> Math.abs(of.getPerson().getAge() - getAge())) ); } Eliminates messy conditional logic
  • 51. @crichardson Using ïŹ‚atMap() class Person public Optional<Friend> longestFriendship() {...} public Optional<Friend> longestFriendshipOfLongestFriend() { return longestFriendship() .flatMap(friend -> friend.getPerson().longestFriendship()); } not always a symmetric relationship. :-)
  • 52. @crichardson Agenda Why functional programming? Simplifying collection processing Eliminating NullPointerExceptions Simplifying concurrency with Futures and Rx Observables Tackling big data problems with functional programming
  • 53. @crichardson Let’s imagine you are performing a CPU intensive operation class Person .. public Set<Hometown> hometownsOfFriends() { return friends.stream() .map(f -> cpuIntensiveOperation()) .collect(Collectors.toSet()); }
  • 54. @crichardson class Person .. public Set<Hometown> hometownsOfFriends() { return friends.parallelStream() .map(f -> cpuIntensiveOperation()) .collect(Collectors.toSet()); } Parallel streams = simple concurrency Potentially uses N cores Nx speed up
  • 55. @crichardson Let’s imagine that you are writing code to display the products in a user’s wish list
  • 56. @crichardson The need for concurrency Step #1 Web service request to get the user proïŹle including wish list (list of product Ids) Step #2 For each productId: web service request to get product info Sequentially terrible response time Need fetch productInfo concurrently
  • 57. @crichardson Futures are a great concurrency abstraction http://en.wikipedia.org/wiki/Futures_and_promises
  • 58. @crichardson Worker thread or event-driven Main thread How futures work Outcome Future Client get Asynchronous operation set initiates
  • 59. @crichardson BeneïŹts Simple way for two concurrent activities to communicate safely Abstraction: Client does not know how the asynchronous operation is implemented Easy to implement scatter/gather: Scatter: Client can invoke multiple asynchronous operations and gets a Future for each one. Gather: Get values from the futures
  • 60. @crichardson Example wish list service public interface UserService { Future<UserProfile> getUserProfile(long userId); } public class UserServiceProxy implements UserService { private ExecutorService executorService; @Override public Future<UserProfile> getUserProfile(long userId) { return executorService.submit(() -> restfulGet("http://uservice/user/" + userId, UserProfile.class)); } ... } public interface ProductInfoService { Future<ProductInfo> getProductInfo(long productId); }
  • 61. @crichardson public class WishlistService { private UserService userService; private ProductInfoService productInfoService; public Wishlist getWishlistDetails(long userId) throws Exception { Future<UserProfile> userProfileFuture = userService.getUserProfile(userId); UserProfile userProfile = userProfileFuture.get(300, TimeUnit.MILLISECONDS); Example wish list service get user info List<Future<ProductInfo>> productInfoFutures = userProfile.getWishListProductIds().stream() .map(productInfoService::getProductInfo) .collect(Collectors.toList()); long deadline = System.currentTimeMillis() + 300; List<ProductInfo> products = new ArrayList<ProductInfo>(); for (Future<ProductInfo> pif : productInfoFutures) { long timeout = deadline - System.currentTimeMillis(); if (timeout <= 0) throw new TimeoutException(...); products.add(pif.get(timeout, TimeUnit.MILLISECONDS)); } ... return new Wishlist(products); } asynchronously get all products wait for product info
  • 62. @crichardson It works BUT Code is very low-level and messy And, it’s blocking
  • 63. @crichardson Better: Futures with callbacks no blocking! def asyncSquare(x : Int) : Future[Int] = ... x * x... val f = asyncSquare(25) Guava ListenableFutures, Spring 4 ListenableFuture Java 8 CompletableFuture, Scala Futures f onSuccess { case x : Int => println(x) } f onFailure { case e : Exception => println("exception thrown") } Partial function applied to successful outcome Applied to failed outcome
  • 65. @crichardson Functional futures - map def asyncPlus(x : Int, y : Int) = ... x + y ... val future2 = asyncPlus(4, 5).map{ _ * 3 } assertEquals(27, Await.result(future2, 1 second)) Scala, Java 8 CompletableFuture Asynchronously transforms future
  • 66. @crichardson Functional futures - ïŹ‚atMap() val f2 = asyncPlus(5, 8).flatMap { x => asyncSquare(x) } assertEquals(169, Await.result(f2, 1 second)) Scala, Java 8 CompletableFuture (partially) Calls asyncSquare() with the eventual outcome of asyncPlus()
  • 67. @crichardson ïŹ‚atMap() is asynchronous Outcome3f3 Outcome3 f2 f2 = f1 ïŹ‚atMap (someFn) Outcome1 f1 Implemented using callbacks someFn(outcome1)
  • 68. @crichardson class WishListService(...) { def getWishList(userId : Long) : Future[WishList] = { userService.getUserProfile(userId) flatMap { userProfile => Scala wishlist service val futureOfProductsList : Future[List[ProductInfo]] = Future.sequence(listOfProductFutures) val timeoutFuture = ... Future.firstCompletedOf(Seq(wishlist, timeoutFuture)) } } val wishlist = futureOfProductsList.map { products => WishList(products) } val listOfProductFutures : List[Future[ProductInfo]] = userProfile.wishListProductIds .map { productInfoService.getProductInfo }
  • 69. @crichardson Using Java 8 CompletableFutures public class UserServiceImpl implements UserService { @Override public CompletableFuture<UserInfo> getUserInfo(long userId) { return CompletableFuture.supplyAsync( () -> httpGetRequest("http://myuservice/user" + userId, UserInfo.class)); } Runs in ExecutorService
  • 70. @crichardson Using Java 8 CompletableFutures public CompletableFuture<Wishlist> getWishlistDetails(long userId) { return userService.getUserProfile(userId).thenComposeAsync(userProfile -> { Stream<CompletableFuture<ProductInfo>> s1 = userProfile.getWishListProductIds() .stream() .map(productInfoService::getProductInfo); Stream<CompletableFuture<List<ProductInfo>>> s2 = s1.map(fOfPi -> fOfPi.thenApplyAsync(pi -> Arrays.asList(pi))); CompletableFuture<List<ProductInfo>> productInfos = s2 .reduce((f1, f2) -> f1.thenCombine(f2, ListUtils::union)) .orElse(CompletableFuture.completedFuture(Collections.emptyList())); return productInfos.thenApply(list -> new Wishlist()); }); } Java 8 is missing Future.sequence() ïŹ‚atMap()! map()!
  • 71. @crichardson Introducing Reactive Extensions (Rx) The Reactive Extensions (Rx) is a library for composing asynchronous and event-based programs using observable sequences and LINQ-style query operators. Using Rx, developers represent asynchronous data streams with Observables , query asynchronous data streams using LINQ operators , and ..... https://rx.codeplex.com/
  • 72. @crichardson About RxJava Reactive Extensions (Rx) for the JVM Original motivation for NetïŹ‚ix was to provide rich Futures Implemented in Java Adaptors for Scala, Groovy and Clojure https://github.com/NetïŹ‚ix/RxJava
  • 73. @crichardson RxJava core concepts trait Observable[T] { def subscribe(observer : Observer[T]) : Subscription ... } trait Observer[T] { def onNext(value : T) def onCompleted() def onError(e : Throwable) } NotiïŹes An asynchronous stream of items Used to unsubscribe
  • 74. Comparing Observable to... Observer pattern - similar but adds Observer.onComplete() Observer.onError() Iterator pattern - mirror image Push rather than pull Futures - similar Can be used as Futures But Observables = a stream of multiple values Collections and Streams - similar Functional API supporting map(), ïŹ‚atMap(), ... But Observables are asynchronous
  • 75. @crichardson Fun with observables val every10Seconds = Observable.interval(10 seconds) -1 0 1 ... t=0 t=10 t=20 ... val oneItem = Observable.items(-1L) val ticker = oneItem ++ every10Seconds val subscription = ticker.subscribe { (value: Long) => println("value=" + value) } ... subscription.unsubscribe()
  • 76. @crichardson def getTableStatus(tableName: String) : Observable[DynamoDbStatus]= Observable { subscriber: Subscriber[DynamoDbMessage] => } Connecting observables to the outside world amazonDynamoDBAsyncClient.describeTableAsync( new DescribeTableRequest(tableName), new AsyncHandler[DescribeTableRequest, DescribeTableResult] { override def onSuccess(request: DescribeTableRequest, result: DescribeTableResult) = { subscriber.onNext(DynamoDbStatus(result.getTable.getTableStatus)) subscriber.onCompleted() } override def onError(exception: Exception) = exception match { case t: ResourceNotFoundException => subscriber.onNext(DynamoDbStatus("NOT_FOUND")) subscriber.onCompleted() case _ => subscriber.onError(exception) } }) }
  • 77. @crichardson Transforming observables val tableStatus = ticker.flatMap { i => logger.info("{}th describe table", i + 1) getTableStatus(name) } Status1 Status2 Status3 ... t=0 t=10 t=20 ... + Usual collection methods: map(), filter(), take(), drop(), ...
  • 78. @crichardson Calculating rolling average class AverageTradePriceCalculator { def calculateAverages(trades: Observable[Trade]): Observable[AveragePrice] = { ... } case class Trade( symbol : String, price : Double, quantity : Int ... ) case class AveragePrice( symbol : String, price : Double, ...)
  • 79. @crichardson Calculating average pricesdef calculateAverages(trades: Observable[Trade]): Observable[AveragePrice] = { trades.groupBy(_.symbol).map { symbolAndTrades => val (symbol, tradesForSymbol) = symbolAndTrades val openingEverySecond = Observable.items(-1L) ++ Observable.interval(1 seconds) def closingAfterSixSeconds(opening: Any) = Observable.interval(6 seconds).take(1) tradesForSymbol.window(...).map { windowOfTradesForSymbol => windowOfTradesForSymbol.fold((0.0, 0, List[Double]())) { (soFar, trade) => val (sum, count, prices) = soFar (sum + trade.price, count + trade.quantity, trade.price +: prices) } map { x => val (sum, length, prices) = x AveragePrice(symbol, sum / length, prices) } }.flatten }.flatten }
  • 80. @crichardson Agenda Why functional programming? Simplifying collection processing Eliminating NullPointerExceptions Simplifying concurrency with Futures and Rx Observables Tackling big data problems with functional programming
  • 81. @crichardson Let’s imagine that you want to count word frequencies
  • 82. @crichardson Scala Word Count val frequency : Map[String, Int] = Source.fromFile("gettysburgaddress.txt").getLines() .flatMap { _.split(" ") }.toList frequency("THE") should be(11) frequency("LIBERTY") should be(1) .groupBy(identity) .mapValues(_.length)) Map Reduce
  • 83. @crichardson But how to scale to a cluster of machines?
  • 84. @crichardson Apache Hadoop Open-source software for reliable, scalable, distributed computing Hadoop Distributed File System (HDFS) EfïŹciently stores very large amounts of data Files are partitioned and replicated across multiple machines Hadoop MapReduce Batch processing system Provides plumbing for writing distributed jobs Handles failures ...
  • 86. @crichardson MapReduce Word count - mapperclass Map extends Mapper<LongWritable, Text, Text, IntWritable> { private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(LongWritable key, Text value, Context context) { String line = value.toString(); StringTokenizer tokenizer = new StringTokenizer(line); while (tokenizer.hasMoreTokens()) { word.set(tokenizer.nextToken()); context.write(word, one); } } } (“Four”, 1), (“score”, 1), (“and”, 1), (“seven”, 1), ... Four score and seven years http://wiki.apache.org/hadoop/WordCount
  • 87. @crichardson Hadoop then shufïŹ‚es the key-value pairs...
  • 88. @crichardson MapReduce Word count - reducer class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> { public void reduce(Text key, Iterable<IntWritable> values, Context context) { int sum = 0; for (IntWritable val : values) { sum += val.get(); } context.write(key, new IntWritable(sum)); } } (“the”, 11) (“the”, (1, 1, 1, 1, 1, 1, ...)) http://wiki.apache.org/hadoop/WordCount
  • 89. @crichardson About MapReduce Very simple programming abstract yet incredibly powerful By chaining together multiple map/reduce jobs you can process very large amounts of data e.g. Apache Mahout for machine learning But Mappers and Reducers = verbose code Development is challenging, e.g. unit testing is difïŹcult It’s disk-based, batch processing slow
  • 90. @crichardson Scalding: Scala DSL for MapReduce class WordCountJob(args : Args) extends Job(args) { TextLine( args("input") ) .flatMap('line -> 'word) { line : String => tokenize(line) } .groupBy('word) { _.size } .write( Tsv( args("output") ) ) def tokenize(text : String) : Array[String] = { text.toLowerCase.replaceAll("[^a-zA-Z0-9s]", "") .split("s+") } } https://github.com/twitter/scalding Expressive and unit testable Each row is a map of named ïŹelds
  • 91. @crichardson Apache Spark Part of the Hadoop ecosystem Key abstraction = Resilient Distributed Datasets (RDD) Collection that is partitioned across cluster members Operations are parallelized Created from either a Scala collection or a Hadoop supported datasource - HDFS, S3 etc Can be cached in-memory for super-fast performance Can be replicated for fault-tolerance http://spark.apache.org
  • 92. @crichardson Spark Word Count val sc = new SparkContext(...) sc.textFile(“s3n://mybucket/...”) .flatMap { _.split(" ")} .groupBy(identity) .mapValues(_.length) .toArray.toMap } } Expressive, unit testable and very fast
  • 93. @crichardson Summary Functional programming enables the elegant expression of good ideas in a wide variety of domains map(), ïŹ‚atMap() and reduce() are remarkably versatile higher-order functions Use FP and OOP together Java 8 has taken a good ïŹrst step towards supporting FP