1
edit
Changes
→Introduction To Scala
===Introduction To Scala===
Scala is a programming language that is based off of Java, but is meant for more scalable applications. Information on scala can be found at: http://www.scala-lang.org/
====IDE Used====
For this asignment the Scala IDE was used. It can be found here: http://scala-ide.org/
The IDE is a version of eclipse with Scala built in.
====Setting Up The Scala Environment====
To run an application on windows, using the Scala IDE, you need to create a Marven project. You do this by clicking File > new > Project > Marven Project. Once the project is created, to use Scala instead of Java, the name of your source file should be refacotred from src/main/java to src/main/scala.
From here you need to edit the pom.xml file to include Spark.
You can do this by adding the following code:
<nowiki>
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.6.0</version>
</dependency>
</dependencies>
</nowiki>
When the project is built it will include the Apache Spark files.
====Imports In Object====
Like Java, Scala uses both classes and objects, to use Spark inside these you must import it by adding the following two lines to the top of the file.
<nowiki>
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
</nowiki>
====Windows Bug====
There is a known bug on windows for Scala where it will not run unless hadoop is installed. This happens even though hadoop is not required for Scala or Spark. To fix this issue, a workaround is to add one of the hadoop exe files to your program.
To do this, I had to download the pre-compiled hadoop file and put it in my C directory. From there I had to add the following line to my file to tell Spark where to find the exe.
<nowiki>
System.setProperty("hadoop.home.dir", "c:\\winutil\\")
</nowiki>
===Introduction To Spark===