spark 是个分布式弹性数据计算框架

1. spark
2015.12.25

2. spark
Apache Spark
in Hadoop and Spark

3. 1 text_file = spark.textFile("hdfs://...")
2 text_file.flatMap(lambda line: line.split())
3 .map(lambda word: (word, 1))
4 .reduceByKey(lambda a, b: a+b)
Spark 80
Scala, Python and R shells .

4. SQL, streaming, and
complex analytics.
Spark
including SQL and
DataFrames, MLlib for
machine learning, GraphX,
and Spark Streaming.

5. Spark Hadoop, Mesos,
standalone, or in the cloud.
HDFS, Cassandra, HBase, and S3
.
Spark
on EC2, on Hadoop
YARN, or on Apache Mesos .
HDFS, Cassandra, HBase, Hive,
Tachyon, and any Hadoop data
source.

6. (RDDs)
Spark Resilient Distributed Dataset (RDD)
2 RDDs
HDFS HBase Hadoop

7. RDDs SparkContext textFile
URI ( hdfs:// s3n://
)
1 val distFile = sc.textFile("/usr/local/Cellar/
apache-spark/1.5.2/README.md")
2 distFile.count()
3 distFile.map(s => s.length).reduce((a, b) => a+b)
4 val wordCounts = distFile.flatMap(line =>
line.split(" ")).map(word => (word, 1)).reduceByKey((a,
b) => a + b)

8. 1 /* SimpleApp.scala */
2 import org.apache.spark.SparkContext
3 import org.apache.spark.SparkContext._
4 import org.apache.spark.SparkConf
5
6 object SimpleApp {
7 def main(args: Array[String]) {
8 val logFile = "YOUR_SPARK_HOME/README.md"
// Should be some file on your system
9 val conf = new SparkConf().setAppName(
"Simple Application")
10 val sc = new SparkContext(conf)
11 val logData = sc.textFile(logFile, 2).cache()
12 val numAs = logData.filter(line =>
line.contains("a")).count()
13 val numBs = logData.filter(line =>
line.contains("b")).count()
14 println("Lines with a: %s, Lines with b:
%s".format(numAs, numBs))
15 }
16 }

9. 1 """SimpleApp.py"""
2 from pyspark import SparkContext
3
4 logFile = "YOUR_SPARK_HOME/README.md"
# Should be some file on your system
5 sc = SparkContext("local", "Simple App")
6 logData = sc.textFile(logFile).cache()
7
8 numAs = logData.filter(lambda s: 'a' in s).count()
9 numBs = logData.filter(lambda s: 'b' in s).count()
10
11 print("Lines with a: %i, lines with b: %i" % (numAs,
numBs))

10. 1 /* SimpleApp.java */
2 import org.apache.spark.api.java.*;
3 import org.apache.spark.SparkConf;
4 import org.apache.spark.api.java.function.Function;
5
6 public class SimpleApp {
7 public static void main(String[] args) {
8 String logFile = "YOUR_SPARK_HOME/README.md";
// Should be some file on your system
9 SparkConf conf = new SparkConf().setAppName("Simple Application");
10 JavaSparkContext sc = new JavaSparkContext(conf);
11 JavaRDD<String> logData = sc.textFile(logFile).cache();
12
13 long numAs = logData.filter(new Function<String, Boolean>() {
14 public Boolean call(String s) { return s.contains("a"); }
15 }).count();
16
17 long numBs = logData.filter(new Function<String, Boolean>() {
18 public Boolean call(String s) { return s.contains("b"); }
19 }).count();
20
21 System.out.println("Lines with a: " + numAs + ", lines with b: " +
numBs);
22 }
23 }

11. Spark Streaming
Spark streaming Spark API
kafka flume Twitter ZeroMQ Kinesis
map reduce join window
Spark

12. 1 Source Artifact
2 Kafka spark-streaming-kafka_2.10
3 Flume spark-streaming-flume_2.10
4 Kinesis spark-streaming-kinesis-asl_2.10
5 Twitter spark-streaming-twitter_2.10
6 ZeroMQ spark-streaming-zeromq_2.10
7 MQTT spark-streaming-mqtt_2.10

13. Spark SQL
Spark SQL Spark SQL, HiveQL Scala
RDD-SchemaRDD
SchemaRDDs scheme
SchemaRDD
RDD Parquet JSON
Apache Hive HiveSQL

14. DataFrame
Spark DataFrame RDD
DataFrame RDD schema
DataFrame

15. RDD RDD DAG
DataFrame lazy
RDD DAG

16. https://endymecy.gitbooks.io/spark-programming-guide-zh-cn/content/
spark-sql/data-sources/rdds.html
Spark RDDs SchemaRDDs
1. RDD
(schema)
2. SchemaRDDs
RDDs