Parquet File Format

Hadoop support many File formats. These include plain text files in Hadoop and storing files in Hadoop specific format like Sequence Files. There some more sophisticated file formats like Avro and Parquet. Every File format in Hadoop brings its own strengths. In this blog post we will discuss what is Parquet File Format and how is it useful for us. Parquet File format was created by Twitter and Cloudera to make a efficient file format HDFS.
        
           Parquet File format comes from class of columnar file formats. Columnar File formats are more usable when you plan to access only few columns of data. These kind of formats are very useful for columnar databases. It has following advantages.

 1. Columnar File formats are more compression friendly, because probability having common values     in a column is more as compare to at row level.
2. While reading only those columns are read which are required. so you end up saving a lot of time       on IO and decompression on data.

      Columnar storage has evolved for Hadoop over period of time. Hive has something called RCFile format, which is very popular columnar file format.One another file format is ORC(Optimized Row Columnar) File Format.
I would say ORC and Parquet File Formats are very similar , they are mature version of RCFile format. Parquet File format following advantages over RCFile format.

 1. Parquet is more compression efficient because you can specify column specific compression.
2. Meta data of parquet files is stored as part of file at the end. so this format is self documenting.
3. Parquet file format can be used with Avro and Thrift also.
4. Parquet file are tuned for better query performance than RCFile format. 

           Spark provides with very good support for parquet files. you can load parquet files very easily. In fact spark provided support for parquet file very early than avro. Avro support was added in spark 1.6, while parquet support existed for long. 

 Eg. val data = sqlContext.parquet("testFile")

 Advantage of using parquet file with spark is that, you do not need to bother about schema of file. spark takes schema from parquet file and creates a data frame and gives you. 

 To Be Continued...

Comments

Post a Comment

Popular posts from this blog

Hive UDF Example

UDF(User Defined Function) is a Very important functionality provided by Hive. It is very simple to create a UDF for Hive. In this tutorial we will learn creating UDF and how to use it with hive. There are two possible ways to create UDF. 1. using org.apache.hadoop.hive.ql.exec.UDF 2. using  org.apache.hadoop.hive.ql.udf.generic.GenericUDF        If input and output to your custom function is basic type eg. Text, FloatWritable, DoubleWritable,IntWritable etc the use  org.apache.hadoop.hive.ql.exec.UDF.        If yor input and output can be Map, set, list type of data structure the use  using  org.apache.hadoop.hive.ql.udf.generic.GenericUDF. We will discuss the first type of UDF here. I will write one more post to discuss the second approach. First of all lets assume I want to create a hive function called toUpper which will convert a string to uppercase. Follow the following steps to achieve it. 1. Download and install eclipse from  here 2. Hive should be installed, if
Read more

Custom UDF in Apache Spark

Image
Apache Spark has become very widely used framework to build Big data application. Spark SQL has made adhoc analysis on structured data very easy, So it is very popular among users who deal with huge amount of structured data. However many times you will realise that some functionality is not available in Spark SQL. For example in Spark 1.3.1 , when you are using sqlContext basic functions like variance , standard deviation and percentile are not available(You can access these using HiveContext). In a similar situation, you may like to do some string operation on an input column but functionality may not be available in sqlContext. In such situations , we can use spark udf to add new constructs in sqlContext. I have found them very handy, specially while processing string type columns. In this post we will discuss how to create UDFs in Spark and  how to use them. Use Case : Let us keep this example very simple. We will create a udf which will take a string as input and it will conve
Read more

Enterprise Kafka and Spark : Kerberos based Integration

In previous posts we have seen how to integrate Kafka with Spark streaming. However in a typical enterprise environment the connection between Kafka and spark should be secure. If you are using cloudera or hortonworks distribution of Hadoop/Spark, then most likely it will be Kerberos enabled connection. In this post we will see what are the configurations we have to make to enable kerberized kafka spark integration.  Before getting into details of integration, i wanted to make it clear that I assume you are using cloudera/hortonworks/MapR provided kafka and spark kerberos enablled installation. we will see that as a developer what additional things has to be performed by you. For this we have to first select security protocol provided by kafka. if you see kafka documentation, security.protocol property is used for this. we can select following values 1. PLAINTEXT 2. SASL_PLAINTEXT 3. SASL_SSL We will select SASL_PLAINTEXT for kerberos.  This property has to be set while creating
Read more