Harjeet's blog

Spark has evolved over the years and Spark Provide Multiple ways to write applications. You Can develop a spark application using following API. You can watch Video on This topic as part of Free Spark Course RDD DataFrame Dataset As of Today, There are 3 base Data structures in Spark to write applications. As part Of This Post we will focus on Dataframe API. When Spark was developed, In beginning it had only RDD API to develop application. RDD api were completely functional in nature. People who were from java background or SQL backround used to find it difficult to write huge applications in Spark using RDD. Currently, when i Take interviews, If i ask candidates to write code in rdd, Very few of them seem confident and try to solve the problem using RDD API. So Spark guys, got inspired from world of pandas and R and brought in concept of dataframe. Dataframe helps you Imagine your data as a table. you can do your traditional operations on dataframe like grouby , cou...

This is very famous interview question in spark interviews. What is difference between RDD and dataframe and datasets. If the person answers this question then generally next question is what is type safety in Dataset. How it is helpful. Even though this is very simple to explain,  people do get confused to explain this. So as part of following videos i have tried to explain these concepts in very simple way. I hope it will be useful for you Please subscribe to this channel and share videos . let us knwo what other topics you want us to cover

Kafka is very well suited and acceptable tool for events processing in Tech world today. Most of Streaming applications use kafka because of its simplicity , scalabilty and ease of use. As part of this video I am planning to cover basics of Kafka. I am also going to explain why people are prefering kafka over traditional queue based systems like activemq , rabbitmq etc. Then we will discuss about kafka architecture. as part of that we will also understand what is broker, what is kafka producer and what is kafka topic and what are kafka partitions. Please subscribe to this youtube channel and share your feedback

Streaming applications are new Hot Thing in data processing world today. everyone wants to process data as soon as it is available. To cater to this need , there are new set of tools in market. Spark Tries to provide streaming using its basic batch processing API. It has been very successful and accepted well in industry for this. Spark Treats input stream of messages as mini batch. These batches run as frequently as possible. Minimum time interval is 1 sec. after every one second a new RDD is generated from stream of input messages and data is processed in batch way. It gives a feeling to client that he is using batch processing system. As part of the following video i have covered basic concepts of spark streaming . Please watch this video and share your thoughts. Please subscribe to channel and share what other topics you would like me to cover.  

I am back with one more interesting Post on Spark. One of my Youtube channel's viewer requested a Video post on Dynamic Resource allocation in spark. I have seen in my experience that lot of folks enable dynamic resource allocation in spark , but dont start external shuffle service. so eventually they are not using dynamic resource allocation as it is mandatory. External shuffle service takes away responsibility of shuffling data from executors and let them focus on execution of tasks. This helps us make optimum utilization of resources and enables dynamic resource allocation. External shuffle service is also implemented in Yarn using auxiliary api provided by yarn. You can enable dynamic resource allocation and external shuffle in yarn by setting properties in yarn-site.xml. I have shared the details of all these things in the video.  Here are the links      I hope this video was useful. Please subscribe to our channel. Please also let me know if you ha...

Hi Friends, I am back with one more video. I hope you are enjoying the series of the videos on spark and hadoop interview questions. as part of today's video we are going to explain  1. what is Architecture of spark. 2. what is role of Driver 3. what is role of cluster manager 4. what is role of spark context 5. how a job is executed on spark I hope you will enjoy this video. please share feed back on this video Please subscribe our channel and share video with your friends

I have been creating videos related to Hadoop and spark from some time now. So as part of the series i am sharing video about why spark is better than Hadoop. As part of this video i have explained follwoing 1. How spark avoids creating jvms again and again. Spark reuses the resources 2. Caching in spark 3. When spark gives far more faster results than Hadoop. 4. Code in spark is very compact It must be noted that there are lot of cases when Hadoop is better than spark.  I hope you will like the video. Please subscribe to my channel. Please also let us know. what are some other topics that you want a video on. 

This post talks about a way to find out if a tree is binary search tree. this is very common interview question for data structures.  following is the link for video to find out What is Binary search Tree. Binary search Tree is also called BST Thank you for watching the video. Please share and subscribe our channel. Please also share your thoughts that what other topics you want us to cover as part of video and post.

In This post we are going to find out that how we can write a algorithm to find if a Binary tree is Binary Search Tree or not. This is a very common data structure based interview question. I have tried to keep things simple while explaining the solution.  Here is the video of solution Thank You for watching this video. Please subscribe and share our channel. Please let us know, what other type of videos you want us to make.

In This post, we have covered introductory concepts of Hadoop. following are the topics covered. 1. What is Big data 2. Introduction to Hadoop and origin. 3. Master Slave Architecture 4. What is Hive 5. What is Pig 6. what is HDFS 7. What is Map Reduce Following is the link of Video Thank You For watching the Video. Please subscribe and share our channel. Please also share your thoughts on what other big data concepts and hadoop concepts video we should prepare for you

This Post Discuss about basics of HDFS. following are the topics covered 1. What is HDFS 2. Architecture of HDFS 3. How data is accessed in HDFS 4. Concept of Namenode, Datanode and Secondary Namenode 5. advantages of Using HDFS 6. Replication Factor in HDFS 7. Block size in HDFS 8. Why HDFS is better Filesystem for Big Data Following is the link for video Thank you for watching Video. Please subscribe to out channel and share your thoughts. Please also tell us on what other big data topics you want videos.

This Post covers basics of Hive. It covers following concepts of Hive 1. What is Hive 2. Hive Architecture 3. Hive Tables  4. Internal Table and external table 5. How to Create Internal table 6. How to create external table 7. How to load data in a Hive table data can be loaded from local filesystem or from hdfs file system. Please watch the following video  Please share your comments and views

As Past of This Post , I am sharing my thoughts on Some Hadoop Interview questions. Please share your comments and thoughts Hadoop Combiner Thanks For watching This. Please subscribe channel for more Hadoop Interview questions

In previous posts we have seen how to integrate Kafka with Spark streaming. However in a typical enterprise environment the connection between Kafka and spark should be secure. If you are using cloudera or hortonworks distribution of Hadoop/Spark, then most likely it will be Kerberos enabled connection. In this post we will see what are the configurations we have to make to enable kerberized kafka spark integration.  Before getting into details of integration, i wanted to make it clear that I assume you are using cloudera/hortonworks/MapR provided kafka and spark kerberos enablled installation. we will see that as a developer what additional things has to be performed by you. For this we have to first select security protocol provided by kafka. if you see kafka documentation, security.protocol property is used for this. we can select following values 1. PLAINTEXT 2. SASL_PLAINTEXT 3. SASL_SSL We will select SASL_PLAINTEXT for kerberos.  This property has to be set ...

Apache Spark has become very widely used tool in Big data world. It provides us a one stop shop to do lot of activities with data. You can do data warehouse activities , build Machine Learning application, Build Reporting solutions, create Stream processing Applications. Spark Streaming provides API for connecting different frameworks. As the big data world is growing fast, there are always some new tools that we may need to use in our projects and integrate with spark or spark streaming. In this post , we will look see how can we integrate any system with spark streaming using custom receiver API. There can be many reasons one has to do this. I am just listing few below for your understanding. 1. Spark streaming does not provide integration with your data source 2. Spark streaming integration provided is using different version of API that you source is using 3. You want to handle lower level details of how spark connects with source. 4. A situation where you are using o...

When you Integrate Kafka with spark streaming. There are some very small things that needs to be taken care. These things are often ignored and we end up wasting lot of time in resolving issues which occur because of this. As part of this post , i will discuss these small small issues with you. 1. Task Not Serializable:  If you have tried posting messages from spark streaming to kafka, there is very high probability that you will face this issues. Let us understand the problem first.   when does it occur - when you create a object of a class and try to use it in any of transformation logic for dstream or rdd.   Example  - example code is below. If KafkaSink class is not declared as serializable, and you are using it as below, You will definitely face Task not serializable issue.  kafkaStream.foreachRDD{ rdd => rdd.foreachPartition{ part => part.foreach(msg => kafkaSink.value.send("kafka_out_topic",msg.toSt ri...

Spark Streaming and Kafka are two talked about technologies these days. Kafka is a distributed queue. we can post message to kafka queue and also read message from kafka queue. Spark Streaming can keep on listening to a source and process the incoming data. Let us understand how can we integrate these two technologies. As part of this post, we will learn following 1. Read Message from Kafka queue 2. Write Message to Kafka Queue Reading Message from Kafka queue :  First of all we should create a stream which will keep listening to Kafka topic. Following is code for this. import org.apache.spark.streaming.kafka._ @transient val kafkaStream = KafkaUtils.createStream(streamingContext,"zookpr-host:2181","kafka-grp", Map("Kafka_input" -> 1.toInt)) There is another way to read messages from Kafka topics, i.e. Kafka Direct. Kafka Direct is out of scope for this scope. We will discuss about that in another post.         once kafkaStrea...

If you are using spark on Yarn, then you must have observed that there are different ways a job can be run on yarn cluster. In this post we will try to explore this. Spark Job to yarn can be submitted in two modes. 1. yarn-client 2. yarn cluster Yarn-Client : When we submit spark job to Yarn and mode is set to yarn-client, then spark driver runs on client machine. Let me elaborate on that, as we know spark driver is kind of controller of the job. When a spark job is submitted in client mode , driver runs on local machine and while spark job runs we can see logs of spark job on client machine. This will not allow you to run anything else on client untill job copletes. you can do this as following spark-submit -mode yarn-client com.mycomp.Example ./test-code.jar here -mode yarn-client is the important parameter that needs to be added. You can also use nohop and diver all logs to a log file and still run this spark job in  client mode.  Yarn-Cluster : Yarn-cluster...

Apache Spark has become very widely used framework to build Big data application. Spark SQL has made adhoc analysis on structured data very easy, So it is very popular among users who deal with huge amount of structured data. However many times you will realise that some functionality is not available in Spark SQL. For example in Spark 1.3.1 , when you are using sqlContext basic functions like variance , standard deviation and percentile are not available(You can access these using HiveContext). In a similar situation, you may like to do some string operation on an input column but functionality may not be available in sqlContext. In such situations , we can use spark udf to add new constructs in sqlContext. I have found them very handy, specially while processing string type columns. In this post we will discuss how to create UDFs in Spark and  how to use them. Use Case : Let us keep this example very simple. We will create a udf which will take a string as input and it will c...

Hadoop support many File formats. These include plain text files in Hadoop and storing files in Hadoop specific format like Sequence Files. There some more sophisticated file formats like Avro and Parquet. Every File format in Hadoop brings its own strengths. In this blog post we will discuss what is Parquet File Format and how is it useful for us. Parquet File format was created by Twitter and Cloudera to make a efficient file format HDFS.                     Parquet File format comes from class of columnar file formats. Columnar File formats are more usable when you plan to access only few columns of data. These kind of formats are very useful for columnar databases. It has following advantages. 1. Columnar File formats are more compression friendly, because probability having common values     in a column is more as compare to at row level. 2. While reading only those columns are read which are required. so you en...