Enterprise Kafka and Spark : Kerberos based Integration

In previous posts we have seen how to integrate Kafka with Spark streaming. However in a typical enterprise environment the connection between Kafka and spark should be secure. If you are using cloudera or hortonworks distribution of Hadoop/Spark, then most likely it will be Kerberos enabled connection. In this post we will see what are the configurations we have to make to enable kerberized kafka spark integration.
 Before getting into details of integration, i wanted to make it clear that I assume you are using cloudera/hortonworks/MapR provided kafka and spark kerberos enablled installation. we will see that as a developer what additional things has to be performed by you. For this we have to first select security protocol provided by kafka. if you see kafka documentation, security.protocol property is used for this. we can select following values
1. PLAINTEXT
2. SASL_PLAINTEXT
3. SASL_SSL
We will select SASL_PLAINTEXT for kerberos. This property has to be set while creating kafka consumer object.
Once this is done you should create following jaas file(name it as kafka_jass.conf). 
If you are using keytab file for authentication
KafkaClient {
   com.sun.security.auth.module.Krb5LoginModule required
   useKeyTab=true
   keyTab="./keytab"
   storeKey=true
   useTicketCache=false
   serviceName="kafka"
   principal="my_id/_HOST@EXAMPLE.COM";

};
if you are using kerberos ticket for authentication , use following jaas configuration

 KafkaClient {
   com.sun.security.auth.module.Krb5LoginModule required
   useKeyTab=false
   storeKey=true
   useTicketCache=true
   serviceName="kafka"
   principal="my_id/_HOST@EXAMPLE.COM";

};
Once this is done use following command to run the spark code. 
spark-submit --master yarn-client \
--files "kafka_jaas.conf,/path/to/keytab" \
--driver-java-options "-Djava.security.auth.login.config=./kafka_jaas.conf" \
--conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=./kafka_jaas.conf" \
--class <user-main-class> \
<user-application.jar> 

When using kerberos tickets make sure you are setting SPARK_YARN_USER_ENV property so that every executer knows where to pic kerberos ticket from
export SPARK_YARN_USER_ENV="KRB5CCNAME=/path/to/kerberos/ticket"
then run following
spark-submit --master yarn-client \
--files "kafka_jaas.conf" \
--driver-java-options "-Djava.security.auth.login.config=./kafka_jaas.conf" \
--conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=./kafka_jaas.conf" \
--class <user-main-class> \
<user-application.jar>
Now Let us understand some of important properties.
--files : helps you pass files to all executors. Files are copied in application container folder.
--driver-java-options : helps you set java properties for driver
spark.executor.extraJavaOptions : helps you set java properties for executors.
SPARK_YARN_USER_ENV : helps you set environment properties before any executor is started.
If you want to know about jaas files, go through this link   
If you want to see how to create custom receiver the follow this link.
Thats all i had for this post. Please feel free to share your thoughts on this.

Comments

Post a Comment

Popular posts from this blog

Hive UDF Example

UDF(User Defined Function) is a Very important functionality provided by Hive. It is very simple to create a UDF for Hive. In this tutorial we will learn creating UDF and how to use it with hive. There are two possible ways to create UDF. 1. using org.apache.hadoop.hive.ql.exec.UDF 2. using  org.apache.hadoop.hive.ql.udf.generic.GenericUDF        If input and output to your custom function is basic type eg. Text, FloatWritable, DoubleWritable,IntWritable etc the use  org.apache.hadoop.hive.ql.exec.UDF.        If yor input and output can be Map, set, list type of data structure the use  using  org.apache.hadoop.hive.ql.udf.generic.GenericUDF. We will discuss the first type of UDF here. I will write one more post to discuss the second approach. First of all lets assume I want to create a hive function called toUpper which will convert a string to uppercase. Follow the following steps to achieve it. 1. Download and install eclipse from  here 2. Hive should be installed, if
Read more

Custom UDF in Apache Spark

Image
Apache Spark has become very widely used framework to build Big data application. Spark SQL has made adhoc analysis on structured data very easy, So it is very popular among users who deal with huge amount of structured data. However many times you will realise that some functionality is not available in Spark SQL. For example in Spark 1.3.1 , when you are using sqlContext basic functions like variance , standard deviation and percentile are not available(You can access these using HiveContext). In a similar situation, you may like to do some string operation on an input column but functionality may not be available in sqlContext. In such situations , we can use spark udf to add new constructs in sqlContext. I have found them very handy, specially while processing string type columns. In this post we will discuss how to create UDFs in Spark and  how to use them. Use Case : Let us keep this example very simple. We will create a udf which will take a string as input and it will conve
Read more

Hadoop series : Pseudo Distributed Mode Hadoop Instalation

In this tutorial, we will learn required steps of setting up Hadoop on single node also called pseudo distributed mode. How ever as part of this series we will also setup hadoop on multiple machines. We will also learn Map Reduce, hive , Pig etc.So stay tuned, Here it comes .... :) We need Following this to start with. Ubuntu: I always prefer ubuntu as my linux flavour. However if you are using a very low end machine with very small ram, you can install Lubuntu also. Lubuntu is very light weight and any low end machine should work. If you have a windows machine. you can install hadoop using cygwin or VirtualBox or Vmware player.  Here we go... Steps for installation of Hadoop. If you are using Windows follow steps from 1 to 4 1. Download  and install Virtual box from https://www.virtualbox.org/wiki/Downloads   2. Download ubuntu from  http://www.ubuntu.com/download/desktop . If you have low  configuration machine then you can use Lubuntu. download Lubuntu from 
Read more