Two documents produce seven tokens and five distinct word counts
Word Count: The Full Data Flowdid identifies a document, v is its text, and w is a word
Map emits (word, 1); shuffle groups words; reduce adds each group
Map Tasks and Reduce TasksA task is one unit of work; a worker process can run multiple tasks over time
Map tasks split the input; reduce tasks split the grouped keys
A Worker Can Fail
Retry needs available source data and a repeatable computation
Demo: Follow the Grouped CountSensor example: count readings at or above 80°C by machine
Track M1 from three input partitions to one result
Answer: Follow the Grouped CountMap emits five pairs, shuffle forms three keys, reduce returns three counts
The output counts sum to five retained readings
§ Relational Operators
Relational Operations in MapReduceSQL names the result; MapReduce requires you to write how records produce it
SQL operation
What the MapReduce program must decide
WHERE
which rows map emits
GROUP BY + SUM
grouping key and values to add
JOIN
join key, source tags and matching pairs
The same familiar SQL operation requires explicit record-processing code
The Records for Our SQL ExamplesThe query determines the intermediate key; a tuple position does not determine its meaning
R = [(123,21), (456,10), (123,4)]
S = [(21,67), (21,80), (4,9), (5,99)]
R has three rows; S has two different rows with C = 21
Selection in MapReduceSELECT * FROM R WHERE A = 123 keeps only the two matching rows
The mapper checks the condition and emits the retained rows
A Map-Only SelectionA filter can finish in map; Hadoop supports map-only jobs by setting reducers to zero
Only rows passing A = 123 need to be emitted
Selection: Code and Retained RowsA WHERE condition decides whether each input row emits an output row
# SELECT * FROM R WHERE A = 123
def select_row(a, b):
if a == 123:
yield (a, b)
Filtering needs no cross-partition grouping
Grouping in MapReduceSELECT A, SUM(B) FROM R GROUP BY A adds B values within each A
Key 123 groups values 21 and 4; reduce returns 25
GROUP BY: Choose the Key and ValueThe grouping column becomes the key; the quantity to sum becomes the value
# SELECT A, SUM(B) FROM R GROUP BY A
def map_sum(a, b):
yield (a, b)
def reduce_sum(a, values):
yield (a, sum(values))
Changing emitted values from B to 1 changes SUM into COUNT
Shuffle Join: Move Both InputsMatching join keys must meet even when the two input tables start on different servers
Redistribute both inputs by the join key, then join locally
Shuffle Join: Match Rows Within a KeyA source tag tells reduce which rows came from R and which came from S
Within one key, reduce pairs each R row with each matching S row
JOIN: Emit the Join Key and a Source TagBoth mappers emit the same kind of key: R.B on the left and S.C on the right
# SELECT R.A, R.B, S.D# FROM R JOIN S ON R.B = S.C
def map_r(a, b):
yield (b, ("R", a))
def map_s(c, d):
yield (c, ("S", d))
Key 21 brings one R value and two S values together
JOIN: Pair the Values Within One KeyThe nested loops produce every R × S pair within one key
def reduce_join(key, tagged):
rows = list(tagged)
rs = [v for t, v in rows if t == "R"]
ss = [v for t, v in rows if t == "S"]
for a in rs:
for d in ss:
yield (a, key, d)
One reducer call can emit zero, one or many joined rows
JOIN: All Keys and the Final ResultAn inner join emits only keys with rows on both sides
The three output rows preserve both matches for key 21
Caveat: Duplicate Join KeysExample change: add (789, 21) to R; inspect key 21 only
Two rows on each side create four combinations
Broadcast Join: Copy the Small InputIf S is small enough, each worker can receive a copy while R stays partitioned
Copy S to the workers instead of shuffling the large input
Broadcast Join: Work Inside MapEach mapper builds a lookup table for S and probes it with its own R rows
The join can finish in map when each worker holds the small input
Broadcast JOIN: Keep Every MatchEach worker probes a local copy of the small table using its own R rows
from collections import defaultdict
lookup = defaultdict(list)
for c, d in S: # The small input
lookup[c].append(d)
def join_local(a, b):
for d in lookup.get(b, []):
yield (a, b, d)
A list per key preserves the two matches; a single-value dictionary loses one
Two Join StrategiesThe data sizes help determine which input movement is practical
Shuffle join
Broadcast join
Input movement
Redistribute both inputs by key
Copy the small input to each worker
Useful when
Both inputs are large
One input fits in each worker’s memory
Main cost
Network transfer and uneven keys
Copies and memory for the small input
Why Move Beyond MapReduceMore complex calculations require more job code and intermediate data handling
Complex queries need several connected jobs
You write the steps and the intermediate data flow
02 · Spark
01
MapReduce
write map and reduce functions for distributed jobs
02
Spark
compose RDD operations, then describe table queries with DataFrames
§ RDDs
Spark Begins at BerkeleyA university research project grew into an open-source processing engine
2009: research at UC Berkeley
2010: open-source release and the Spark paper
2013: moved to Apache; 2014: became a top-level Apache project
Spark: A Brief HistoryThe MapReduce paper appeared in 2004; the Spark paper followed in 2010
A research idea became an open-source project and a broader community
A Shorter Word Count ProgramThe same word-count task needs much less job wiring in the Spark example
Compose operations instead of wiring a new job around each pair of functions
Word Count with Spark RDDsThe same Word Count now uses a chain of distributed collection operations
sc = spark.sparkContext
lines = sc.parallelize([
"spark makes data simple",
"spark processes data"], 2)
words = lines.flatMap(lambda s: s.split())
pairs = words.map(lambda w: (w, 1))
counts = pairs.reduceByKey(
lambda a, b: a + b)
result = sorted(counts.collect())
Variable names identify the intermediate datasets in the graph
flatMap: Lines Become Wordsmap preserves one output element per line; flatMap flattens each line’s word list
nested = lines.map(lambda s: s.split())
words = lines.flatMap(lambda s: s.split())
The function is the same; the resulting element type changes
map and reduceByKey: Word CountsreduceByKey combines values for each word; it may combine locally before the shuffle
pairs = words.map(lambda w: (w, 1))
counts = pairs.reduceByKey(
lambda a, b: a + b)
The result has five keys, and the counts still sum to seven
Spark’s 2014 Sorting RecordHistorical sorting records drew attention to Spark as a practical processing engine
Different clusters and runs: 72 minutes in 2013, 23 minutes in 2014
Spark Libraries and LanguagesOne execution engine supports several kinds of work and several programming languages
Today’s route uses Python, DataFrames and Spark SQL
Driver, Executors and StorageThe driver coordinates the program; executor processes run tasks on the data
Spark handles the execution machinery behind the shorter program
Resilient Distributed DatasetsAn RDD gives distributed data a programmable collection interface
RDD: an immutable collection divided into partitions, with a record of how to rebuild it
Distributed: partitions can run on different workers
Resilient: lineage records how lost partitions can be recomputed
Transformations Build Another RDDA transformation such as filter describes a new distributed collection
The program builds the calculation before requesting its result
Actions Request ResultsAn action such as count() requests a value and triggers the required computation
Transformations return RDDs; actions return values or save output
Lineage: Code Becomes a Dependency GraphLineage records parent RDDs and operations; the DAG shows these dependencies
# Reuse lines from the Word Count example
words = lines.flatMap(lambda s: s.split())
pairs = words.map(lambda w: (w, 1))
counts = pairs.reduceByKey(
lambda a, b: a + b)
The graph uses the same variable names and functions as the code
Lineage: File, Filter and MapA file, a filter and a map form the same kind of dependency chain
Parent data and recorded functions describe how to produce each derived RDD
An Action Executes the Required DAGAn action requests the answer and runs the calculation defined by the transformations
Transformations describe the calculation; an action requests its result
Cache and Two Actions on One RDDTwo actions reuse the same intermediate words; cache and lineage serve different roles
words = lines.flatMap(lambda s: s.split()).cache()
n = words.count() # 7, fills the cache
k = words.filter(lambda w: w == "spark").count()
# k == 2; reuse available cached words
words.unpersist()
The first action counts all seven words; the second counts two spark occurrences
Cache: Repeated Log QueriesThe filtered log records can serve more than one later query
