Install Apache Spark on Ubuntu

Its been long time since I have written a blog. Meanwhile i was exploring the work of Apache spark. so i thought i will share my knowledge on how to Install spark on Ubuntu.
    As usual I am going to share commands for both Ubuntu and Lubuntu. Both are very similar. First of all we have to make sure that we have java installed on our machine. I am using java 7 for spark. If you already have java installed on your machine, then you don't need to run following command. If you do not have java installed on your machine, then run following command to install java 7.

       sudo apt-get install openjdk-7-jdk

This will install jdk 7 on your machine. if you do not know which version of java is already installed, use following command

      java -version.

if you have multiple versions of java and want to change the default version. then use following command and select specific version.

     sudo update-alternatives --config java

After installing java, we have to make sure that Scala is installed on our machine.I am using scala version 2.10.4. you can download it from here
 
for all the next steps i am assuming that home directory in your machine is "/home/hduser" and we are going to install spark and scala in this directory. you can install scala and spark some where else also but just to keep things simple i am going to install every thing in "/home/hduser".

1. Download and copy scala in /home/hduser directory, you will get a tar compressed file.

2. Extract scala tar and change the name of folder from "scala-2.10.4" to "scala".

3. Set scala home in environment properties. Edit /etc/bash.bashrc file

In Ubuntu

      sudo gedit /etc/bash.bashrc

In Lubuntu

     sudo leafpad /etc/bash.bashrc

4. Copy following lines at the end of file.

      export SCALA_HOME=/home/hduser/scala
      export PATH=$PATH:$SCALA_HOME/bin

save the file and close.

5. Download Spark 1.3.1  from here

6. Untar the spark and copy it in "/home/hduser" directory

7. change the name of directory from "spark-1.3.1" to "spark"

8. open terminal and type scala and then enter. just to verify if scala path is set properly. It will show shell as below.

       scala>

    after this come out of scala shell.

9. Change your current folder to spark folder with following command

      cd /home/hduser/spark

10. Now we have to build spark with the hadoop version that we are using. In my case i am using hadoop 2.4.0 , so i will run following command.

      SPARK_HADOOP_VERSION=2.4.0 sbt/sbt assembly -mem 512

you should change this command for the version that you are using. I have assigned 512 memory while running this build process as some times scala will throw error while running this and show it is short of memory.

11. You can get a prebuild version of spark also and use it with your specific version of Hadoop from here

12. Once your installation is complete you can run following command to check the installation

       ./bin/run-example SparkPi 10

this command will run a already existing example and if it completes successfully then you can assume that your spark is also installed and built successfully.

thank you for going through the steps, please feel free to drop a comment if you face any issues.

Comments

Post a Comment

Popular posts from this blog

Hive UDF Example

UDF(User Defined Function) is a Very important functionality provided by Hive. It is very simple to create a UDF for Hive. In this tutorial we will learn creating UDF and how to use it with hive. There are two possible ways to create UDF. 1. using org.apache.hadoop.hive.ql.exec.UDF 2. using  org.apache.hadoop.hive.ql.udf.generic.GenericUDF        If input and output to your custom function is basic type eg. Text, FloatWritable, DoubleWritable,IntWritable etc the use  org.apache.hadoop.hive.ql.exec.UDF.        If yor input and output can be Map, set, list type of data structure the use  using  org.apache.hadoop.hive.ql.udf.generic.GenericUDF. We will discuss the first type of UDF here. I will write one more post to discuss the second approach. First of all lets assume I want to create a hive function called toUpper which will convert a string to uppercase. Follow the following steps to achieve it. 1. Down...
Read more

Custom UDF in Apache Spark

Image
Apache Spark has become very widely used framework to build Big data application. Spark SQL has made adhoc analysis on structured data very easy, So it is very popular among users who deal with huge amount of structured data. However many times you will realise that some functionality is not available in Spark SQL. For example in Spark 1.3.1 , when you are using sqlContext basic functions like variance , standard deviation and percentile are not available(You can access these using HiveContext). In a similar situation, you may like to do some string operation on an input column but functionality may not be available in sqlContext. In such situations , we can use spark udf to add new constructs in sqlContext. I have found them very handy, specially while processing string type columns. In this post we will discuss how to create UDFs in Spark and  how to use them. Use Case : Let us keep this example very simple. We will create a udf which will take a string as input and it will c...
Read more

Enterprise Kafka and Spark : Kerberos based Integration

In previous posts we have seen how to integrate Kafka with Spark streaming. However in a typical enterprise environment the connection between Kafka and spark should be secure. If you are using cloudera or hortonworks distribution of Hadoop/Spark, then most likely it will be Kerberos enabled connection. In this post we will see what are the configurations we have to make to enable kerberized kafka spark integration.  Before getting into details of integration, i wanted to make it clear that I assume you are using cloudera/hortonworks/MapR provided kafka and spark kerberos enablled installation. we will see that as a developer what additional things has to be performed by you. For this we have to first select security protocol provided by kafka. if you see kafka documentation, security.protocol property is used for this. we can select following values 1. PLAINTEXT 2. SASL_PLAINTEXT 3. SASL_SSL We will select SASL_PLAINTEXT for kerberos.  This property has to be set ...
Read more