1
edit
Changes
→Assignment
==Assignment==
For the assignment, I wanted to try and create a program that would compare two pieces of text and try to determine if they were written by the same person. To do this I will take the two pieces of text and compare how they were written. This includes looking at the words per sentence, sentences per paragraph, characters per word, number of commas, and the number of colons.
===Full Program===
<nowiki>
object WordCount {
def main(args: Array[String]) ={
System.setProperty("hadoop.home.dir", "c:\\winutil\\")
val conf = new SparkConf()
.setAppName("TextCompare")
.setMaster("local")
val sc = new SparkContext(conf)
val text1 = sc.textFile("text1.txt")
val text2 = sc.textFile("text2.txt")
text1.flatMap { line =>
line.split(" ")
}
.map { word =>
(word,1)
}
.reduceByKey(_ + _)
.saveAsTextFile("food.count.txt")
sc.stop
}
}
</nowiki>
