Harjeet's blog

Its been long time since I have written a blog. Meanwhile i was exploring the work of Apache spark. so i thought i will share my knowledge on how to Install spark on Ubuntu.     As usual I am going to share commands for both Ubuntu and Lubuntu. Both are very similar. First of all we have to make sure that we have java installed on our machine. I am using java 7 for spark. If you already have java installed on your machine, then you don't need to run following command. If you do not have java installed on your machine, then run following command to install java 7.        sudo apt-get install openjdk-7-jdk This will install jdk 7 on your machine. if you do not know which version of java is already installed, use following command       java -version. if you have multiple versions of java and want to change the default version. then use following command and select specific version.      sudo u...

It been some time, since i have written any blog. I just managed to get some time to put these things on blog.   In my day to day work, I come across many people who are trying their hands on machine learning. Machine Learning is amazing. Once you fall in love with it, You will enjoy doing it all day. Availability of lot of libraries have made it very easy for us to develop machine learning solutions. many people who are who are new to machine learning use these  API's directly and skip important steps, which will lead to not getting good results. So I thought of sharing some of important steps in this post. Following are the steps followed to create a good machine learning solution.  1. Data collection 2. Data preprocessing     1) Data cleaning     2) Feature creation and feature selection 3) feature scaling and Normalization     4) Divide data into training and testing sets(You can create cross validation set also) 3. Build a mo...

Hi There, people has been asking me to put some light on safemode of Hadoop. So lets see what is it. Many time when you start Hadoop, it gets stuck in Safemode. Now what exactly is this safemode? when Hadoop starts, Normally it puts itself in Safemode. In this mode, you cannot write any new data to hadoop. it is a read only mode. Hadoop says untill  i am going to get heartbeat from some fix number of datanodes, i will keep myself in Safemode. I have seen this happening even when you are in Pseudo distributed mode. So if your Hadoop is in safemode. you will not be able to write any new data or create any folder on Hadoop. So you have to bring Hadoop out of safemode. Following is the command for bringing hadoop out of safemode.       hadoop dfsadmin -safemode leave After this, you will be able to use Hadoop as normal. Enjoy, Keep Coding, keep facing issues, keep learning :)

My Major area of  work is Text Analytic and Machine Learning. I always get excited to solve the problems in this area. So i thought i will share some of my knowledge on this also :). We will start with Naive Bayes algorithm. It is a supervised learning, classification algorithm. Supervised learning means, before running on actual data, we have to train this algorithm with some training set and explain it that which records are acceptable and which records are not acceptable. Eg. before trying my NB(Naive Bayes algo) on test data, I will  show some  examples to algo, that these how does a spam message look and how does a non-spam message look. once it is ready we can go ahead with trying it on real world data. Before Trying NB, we need to know some basics about Probability . Lets go through that. Lets assume. we have a dice. A dice have six faces, each face marks a distinct number between 1 to six. If the dice is not biased, what is the probability that ...

Today we will learn how to install Pig. Installing pig is very simple and straight forward. Following are the steps to install pig. Pig requires Hadoop and java to be already installed. If you have not installed it, follow the link  here 1. Download Pig from  Apache Pig website . Check the compatibility of pig you are downloading with already installed hadoop on your machine. I am going to download  Pig 0.10.1 . 2. After the download is complete, go to download directory and extract Pig 0.10.1.tar.gz 3. Copy the extracted folder in $HOME/pig directory. 4. Edit /etc/bash.bashrc and set PIG_HOME with following command     sudo leafpad /etc/bash.bashrc   Then go to last line and copy following      export PIG_HOME=$HOME/pig     export PATH=$PATH:PIG_HOME/bin also set JAVA_HOME if not set earlier using following command. In my case my java is installed at /usr/lib/jvm/java-6-jdk-i386 ...

Hi Friends... As promised earlier i have created few videos for Hadoop Installation tutorial. Currently these videos are about Pseudo distributed mode installation of hadoop. I will create few videos for fully distributed mode installation also. For now please find the videos below.  1. Video for installation of Lubuntu on windows. For this you should download vmware player from  vmware site  and install on windows machine. you will also need .iso file for lubuntu.  You can download it from  Lubuntu website . 2. After you have installed Lubuntu, you can go through following video and install Hadoop in Pseudo distributed mode. Stay tuned for more stuff on this blog. Please share your feedback and let me know if you want a post on any specific topic.

In Previous posts we learned how to install hadoop , Introductionto hadoop etc. today we will learn about HDFS (Hadoop Distributed file system). HDFS is component of Hadoop. It handles storage part of Hadoop. HDFS follows master slave architecture. Let us discuss Master slave Arch. Let us discuss, what is Master slave Arch.      In Master slave Arch. we have two kind of machines. First set is Master other is slaves. Master does following two things. 1. Plan 2. Monitor     Master is like Manager of your team, He will plan. If he has some work to do, Master will plan whom to assign that work.    Slaves do Following two things. 1. Work 2. Report     Slave is like developer of your team( :P  Please dont feel offended it is just for analogy). Slave does the actual work. If Master assign work to slaves and slaves works and complete the work. Similar to Manager of your team who wants to develop some software, he will plan who...