1
edit
Changes
→Assignment
The creation step is the step in which the data is loaded into the first RDD. This data can be loaded from multiple sources, but for the purpose of this assignment it is loaded from a local text file.
In scala, the following line will load a text file into a variable.
<codenowiki>val test = sc.textFile("food.txt")</codenowiki>
====Transformation====
The next step is to transform the data that is in the first RDD into something you want to use. In this example we would like to count the number of times each word is used in a piece of text.
The following code snip-it will split the RDD by spaces, and map each word with a key value. The value is set to 1 to count each word as 1.
<codenowiki>test.flatMap { line =>
line.split(" ")
{
.map { word =>
(word,1)
}</codenowiki>
====Action====
Finally an action is taken to reduce the RDD. In this example we want to reduce each word by adding the value of the word. To do this we can use:
<codenowiki>.reduceByKey(_ + _)</codenowiki>
====Full Program====
To run the program on windows, the following is the main.
<codenowiki>
object WordCount {
def main(args: Array[String]) ={
}
}
</codenowiki>
