Season II: 1. Prepare the data for analysis with Pig and Python UDF

Season II: 1. Prepare the data for analysis with Pig and Python UDF

Welcome to season 2 of the Hue video series. In this new chapter we are going to demonstrate how Hue can simplify Hadoop usage and lets you focus on the business and less about the underlying technology. In a real life scenario, we will use various Hadoop tools within the Hue UI and explore some data and extract some competitive advantage insights from it.


Let’s go surf the Big Data wave, directly from your Browser!

We want to open a new restaurant. In order to optimize our future business we would like to learn more about the existing restaurants, which tastes are trending, what food eaters are looking for or are positive/negative about… In order to answer these questions, we are going to need some data.

Luckily, Yelp is providing some datasets of restaurants and reviews and we download them. What’s next? Let’s move the data into Hadoop and make it queryable!

Convert Json data with Pig

The current format is Json, which is easy to save but difficult to query as it consist in one big record for each row and requires a more sophisticated loader. We are also going to cleanup the data a bit in the process.

In order to do this in a scalable way, we are going to use the query tool Apache Pig and to make it easy, the Pig Editor in Hue. We explain two ways to do it.

All the code is available on the Hadoop Tutorial github.

Method 1: Pig JsonLoader/JsonStorage

Pig natively provides a JsonLoader. We load our data and map it to a schema, then explode the votes into 3 columns. Notice the clean-up of the text of the reviews.

Here is the script:

reviews = 
  LOAD 'yelp_academic_dataset_review.json'
  USING JsonLoader('votes:map[],user_id:chararray,review_id:chararray,stars:int,date:chararray,text:chararray,type:chararray,business_id:chararray');

tabs = 
  FOREACH reviews
     (INT) votes#'funny', (INT) votes#'useful', (INT) votes#'cool', user_id, review_id, stars, REPLACE(REPLACE(text, 'n', ''), 't', ''), date, type, business_id;

STORE tabs INTO 'yelp_academic_dataset_review.tsv';


Note: if the script fails with a ClassNotFound exception, you might need to logging as ‘oozie’ or ‘hdfs’ and upload /usr/lib/pig/lib/json-simple-1.1.jar into /user/oozie/share/lib/pig on HDFS with File Browser.

Method 2: Pig Python UDF

Let’s convert the business data to TSV with a great Pig features: Python UDF. We are going to process each row with with a UDF loading the Json records one by one and printing them with tabs as delimiter.

As Pig is currently using Jython 2.5 for executing Python UDF and there is no builtin json lib, we need to download jyson from Grab the jyson-1.0.2 version, extract it and upload jyson-1.0.2.jar to /user/oozie/share/lib/pig with FileBrowser.

We need to import our Python UDF into Pig. Open up the Pig Editor and upload a file resource named You can also create the file directly on HDFS with FileBrowser, then edit it and add this script:

from com.xhaus.jyson import JysonCodec as json

def tsvify(line):
 business_json = json.loads(line)
 business = map(unicode, business_json.values())
 return 't'.join(business).replace('n', ' ').encode('utf-8')

Go to ‘Properties’, ‘Resource’ and specify the path to on HDFS.

You are then ready to type the following Pig script:

REGISTER '' USING jython AS converter;

reviews = 
  LOAD '/user/romain/yelp/yelp_academic_dataset_business.json' AS (line:CHARARRAY);

tsv = 
  FOREACH reviews
  GENERATE converter.tsvify(line);

STORE tsv INTO 'yelp_academic_dataset_business.tsv'

What’s next?

Pig is a powerful tool for processing terabytes of data and Hue Pig Editor makes it easier to play around. Python UDF will become part of the editor when HUE-1136 is finished. In episode 3, we will see how to convert to even better formats.

In the next episode, let’s see how to query the data and learn more about the restaurant market!


  1. Kirthi Raman 1 year ago

    This is awesome. I will try to play around with it.

  2. hani 1 year ago

    I am using chd5 quick start virtual machine, the path /user/oozie/share/lib/pig is not a vailable. And I face errors when running the udf.

  3. Moussa 1 year ago

    I’m also using cdh5 quickstart and in cloudera manager i can see that oozie is installed. But when I run a scipt with JsonLoader ,it works without “STORE” statement… but if I add it, script will fail… Do I have to change something in the installation?

  4. Des 1 year ago

    I tried to upload the json to Cloudera Live demo. It says upload is disable in live demo. How can we POC if it is disabled?

  5. Lee 1 year ago

    Thanks! this is a very useful article.

    I’m using the new CDH5 distribution – there is an issue with oozie and the sharedliblist (which displays as empty).

    Be sure to:
    $ export OOZIE_URL=http://localhost:11000/oozie
    the move the hdfs share directory for oozie to old-share:
    $ hadoop fs -mv /user/oozie/share /user/oozie/old-share
    and re-create the share like this:
    $ hadoop fs -mkdir /user/oozie/share/lib
    $ sudo oozie-setup sharelib create -fs hdfs://localhost:8020 -locallib /usr/lib/oozie/oozie-sharelib-yarn.tar.gz
    your share will now be using the new timestamp format e.g. lib_20140918162047
    so to put the jyson-1.0.2.jar i did this:
    $ hadoop fs -put jyson-1.0.2/lib/jyson-1.0.2.jar /user/oozie/share/lib/lib_20140918162047/pig/
    and then restart oozie:
    $ sudo service oozie stop
    $ sudo service oozie start

    now i can see the shareliblist (and run the code in this article):
    $ oozie admin -shareliblist pig
    [Available ShareLib]

    • Author
      Hue Team 1 year ago

      Indeed, this is correct, in CDH5 the sharelib changed and need to be updated like you did!

  6. Srinivas 1 year ago

    I am getting the below error when I run the Pig Script ‘’. Can you please help me how to fix this?

    Apache Pig version 0.12.0-cdh5.1.0 (rexported)
    compiled Jul 12 2014, 08:41:26

    Run pig script using for Pig version 0.8+
    2014-10-14 18:39:53,744 [main] INFO org.apache.pig.Main – Apache Pig version 0.12.0-cdh5.1.0 (rexported) compiled Jul 12 2014, 08:41:26
    2014-10-14 18:39:53,745 [main] INFO org.apache.pig.Main – Logging error messages to: /var/lib/hadoop-yarn/cache/yarn/nm-local-dir/usercache/cloudera/appcache/application_1413332198888_0004/container_1413332198888_0004_01_000002/pig-job_1413332198888_0004.log
    2014-10-14 18:39:54,067 [main] INFO org.apache.pig.impl.util.Utils – Default bootup file /var/lib/hadoop-yarn/.pigbootup not found
    2014-10-14 18:39:54,419 [main] INFO org.apache.hadoop.conf.Configuration.deprecation – mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
    2014-10-14 18:39:54,419 [main] INFO org.apache.hadoop.conf.Configuration.deprecation – is deprecated. Instead, use fs.defaultFS
    2014-10-14 18:39:54,420 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine – Connecting to hadoop file system at: hdfs://quickstart.cloudera:8020
    2014-10-14 18:39:54,436 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine – Connecting to map-reduce job tracker at: localhost:8032
    2014-10-14 18:39:54,594 [main] INFO org.apache.hadoop.conf.Configuration.deprecation – is deprecated. Instead, use fs.defaultFS
    2014-10-14 18:39:54,594 [main] INFO org.apache.hadoop.conf.Configuration.deprecation – mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
    2014-10-14 18:39:54,656 [main] INFO org.apache.pig.scripting.jython.JythonScriptEngine – created tmp python.cachedir=/var/lib/hadoop-yarn/cache/yarn/nm-local-dir/usercache/cloudera/appcache/application_1413332198888_0004/container_1413332198888_0004_01_000002/tmp/pig_jython_6753505879123046001
    2014-10-14 18:40:05,207 [main] WARN org.apache.pig.scripting.jython.JythonScriptEngine – pig.cmd.args.remainders is empty. This is not expected unless on testing.
    2014-10-14 18:40:08,185 [main] INFO org.apache.pig.scripting.jython.JythonScriptEngine – Register scripting UDF: converter.tsvify
    2014-10-14 18:40:08,996 [main] INFO org.apache.pig.scripting.jython.JythonFunction – Schema ‘business:chararray’ defined for func tsvify
    2014-10-14 18:40:09,119 [main] ERROR – ERROR 1000: Error during parsing. Lexical error at line 8, column 0. Encountered: after : “”

  7. Srinivas 1 year ago

    Thanks for your mail. I agree with your above comments. I followed the steps as you mentioned above but still unable to process the data when I run a python UDF that will be used by a Pig script.

    2014-10-20 10:25:03,155 [main] INFO org.apache.pig.scripting.jython.JythonFunction – Schema ‘business:chararray’ defined for func tsvify
    2014-10-20 10:25:03,595 [main] ERROR – ERROR 1000: Error during parsing. Lexical error at line 11, column 0. Encountered: after : “”

    Can you please help me to fix the issue

    • Clark 1 year ago

      Srinivas, the last line, currently:

      STORE tsv INTO ‘yelp_academic_dataset_business.tsv’

      just needs a semicolon at the end:

      STORE tsv INTO ‘yelp_academic_dataset_business.tsv’;

Leave a reply

Your email address will not be published. Required fields are marked *