Difference between revisions of "GPU621/Apache Spark Fall 2022"
(→Apache Spark Core API) |
(→Spark Installation Using Maven) |
||
| Line 5: | Line 5: | ||
===Spark Installation Using Maven=== | ===Spark Installation Using Maven=== | ||
| − | An Apache Spark application can be easily instantiated using Maven. To add the required libraries, you can copy and paste the following code | + | An Apache Spark application can be easily instantiated using Maven. To add the required libraries, you can copy and paste the following code into the "pom.xml". |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
<properties> | <properties> | ||
| Line 38: | Line 28: | ||
</dependency> | </dependency> | ||
</dependencies> | </dependencies> | ||
| − | |||
==Deploy Apache Spark Application On AWS== | ==Deploy Apache Spark Application On AWS== | ||
Revision as of 22:15, 29 November 2022
Contents
Apache Spark
Apache Spark Core API
RDD Overview
One of the most important concepts in Spark is a resilient distributed dataset (RDD). RDD is a collection of elements partitioned across the nodes of the cluster that can be operated in parallel. RDDs are created by starting with a file, or an existing Java collection in the driver program, and transforming it.
Spark Installation Using Maven
An Apache Spark application can be easily instantiated using Maven. To add the required libraries, you can copy and paste the following code into the "pom.xml".
<properties>
<maven.compiler.source>11</maven.compiler.source>
<maven.compiler.target>11</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.10</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>2.2.0</version>
</dependency>
</dependencies>
