Open main menu

CDOT Wiki β

Changes

GPU621/Apache Spark

2 bytes removed, 15:51, 30 November 2020
Comparison: Spark vs Hadoop MapReduce
= Comparison: Spark vs Hadoop MapReduce =
[[File: Comparison.PNG|thumb|upright=1.12|right|alt=Spark cluster|5.1 Spark vs Hadoop MapReduce]]
=== Performance ===
Spark processes data in RAM while Hadoop persists data back to the disk after a map or reduce action. Spark has been found to run '''100 times faster in-memory''', and '''10 times faster on disk'''. Spark won the 2014 Gray Sort Benchmark where it sorted 100TB of data using 206 machines in 23 minutes beating a Hadoop MapReduce cluster's previous world record of 72 minutes using 2100 nodes.