Season II: 1. Prepare the data for analysis with Pig and Python UDF

Season II: 1. Prepare the data for analysis with Pig and Python UDF

Welcome to season 2 of the Hue video series. In this new chapter we are going to demonstrate how Hue can simplify Hadoop usage and lets you focus on the business and less about the underlying technology. In a real life scenario, we will use various Hadoop tools within the Hue UI and explore some data and extract some competitive advantage insights from it.

 

Let’s go surf the Big Data wave, directly from your Browser!

We want to open a new restaurant. In order to optimize our future business we would like to learn more about the existing restaurants, which tastes are trending, what food eaters are looking for or are positive/negative about… In order to answer these questions, we are going to need some data.

Luckily, Yelp is providing some datasets of restaurants and reviews and we download them. What’s next? Let’s move the data into Hadoop and make it queryable!

Convert Json data with Pig

The current format is Json, which is easy to save but difficult to query as it consist in one big record for each row and requires a more sophisticated loader. We are also going to cleanup the data a bit in the process.

In order to do this in a scalable way, we are going to use the query tool Apache Pig and to make it easy, the Pig Editor in Hue. We explain two ways to do it.

All the code is available on the Hadoop Tutorial github.

Method 1: Pig JsonLoader/JsonStorage

Pig natively provides a JsonLoader. We load our data and map it to a schema, then explode the votes into 3 columns. Notice the clean-up of the text of the reviews.

Here is the script:

reviews = 
  LOAD 'yelp_academic_dataset_review.json'
  USING JsonLoader('votes:map[],user_id:chararray,review_id:chararray,stars:int,date:chararray,text:chararray,type:chararray,business_id:chararray');

tabs = 
  FOREACH reviews
  GENERATE
     (INT) votes#'funny', (INT) votes#'useful', (INT) votes#'cool', user_id, review_id, stars, REPLACE(REPLACE(text, 'n', ''), 't', ''), date, type, business_id;

STORE tabs INTO 'yelp_academic_dataset_review.tsv';

 

Note: if the script fails with a ClassNotFound exception, you might need to logging as ‘oozie’ or ‘hdfs’ and upload /usr/lib/pig/lib/json-simple-1.1.jar into /user/oozie/share/lib/pig on HDFS with File Browser.

Method 2: Pig Python UDF

Let’s convert the business data to TSV with a great Pig features: Python UDF. We are going to process each row with with a UDF loading the Json records one by one and printing them with tabs as delimiter.

As Pig is currently using Jython 2.5 for executing Python UDF and there is no builtin json lib, we need to download jyson from http://downloads.xhaus.com/jyson/. Grab the jyson-1.0.2 version, extract it and upload jyson-1.0.2.jar to /user/oozie/share/lib/pig with FileBrowser.

We need to import our Python UDF into Pig. Open up the Pig Editor and upload a file resource named converter.py [link]. You can also create the file directly on HDFS with FileBrowser, then edit it and add this script:

from com.xhaus.jyson import JysonCodec as json

@outputSchema("business:chararray")
def tsvify(line):
 business_json = json.loads(line)
 business = map(unicode, business_json.values())
 return 't'.join(business).replace('n', ' ').encode('utf-8')

Go to ‘Properties’, ‘Resource’ and specify the path to converter.py on HDFS.

You are then ready to type the following Pig script:

REGISTER 'converter.py' USING jython AS converter;

reviews = 
  LOAD '/user/romain/yelp/yelp_academic_dataset_business.json' AS (line:CHARARRAY);

tsv = 
  FOREACH reviews
  GENERATE converter.tsvify(line);

STORE tsv INTO 'yelp_academic_dataset_business.tsv'

 

What’s next?

Pig is a powerful tool for processing terabytes of data and Hue Pig Editor makes it easier to play around. Python UDF will become part of the editor when HUE-1136 is finished. In episode 3, we will see how to convert to even better formats.

In the next episode, let’s see how to query the data and learn more about the restaurant market!

9 Comments

  1. Kirthi Raman 4 months ago

    This is awesome. I will try to play around with it.

  2. hani 4 months ago

    I am using chd5 quick start virtual machine, the path /user/oozie/share/lib/pig is not a vailable. And I face errors when running the udf.

  3. Moussa 3 months ago

    I’m also using cdh5 quickstart and in cloudera manager i can see that oozie is installed. But when I run a scipt with JsonLoader ,it works without “STORE” statement… but if I add it, script will fail… Do I have to change something in the installation?

  4. Des 4 weeks ago

    I tried to upload the json to Cloudera Live demo. It says upload is disable in live demo. How can we POC if it is disabled?

  5. Lee 2 weeks ago

    Thanks! this is a very useful article.

    I’m using the new CDH5 distribution – there is an issue with oozie and the sharedliblist (which displays as empty).

    Be sure to:
    $ export OOZIE_URL=http://localhost:11000/oozie
    the move the hdfs share directory for oozie to old-share:
    $ hadoop fs -mv /user/oozie/share /user/oozie/old-share
    and re-create the share like this:
    $ hadoop fs -mkdir /user/oozie/share/lib
    $ sudo oozie-setup sharelib create -fs hdfs://localhost:8020 -locallib /usr/lib/oozie/oozie-sharelib-yarn.tar.gz
    your share will now be using the new timestamp format e.g. lib_20140918162047
    so to put the jyson-1.0.2.jar i did this:
    $ hadoop fs -put jyson-1.0.2/lib/jyson-1.0.2.jar /user/oozie/share/lib/lib_20140918162047/pig/
    and then restart oozie:
    $ sudo service oozie stop
    $ sudo service oozie start

    now i can see the shareliblist (and run the code in this article):
    $ oozie admin -shareliblist pig
    [Available ShareLib]
    pig
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/ant-1.6.5.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/antlr-2.7.7.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/antlr-runtime-3.4.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/automaton-1.11-8.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/commons-collections-3.2.1.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/commons-el-1.0.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/commons-httpclient-3.1.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/commons-io-2.1.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/core-3.1.1.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/findbugs-annotations-1.3.9-1.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/guava-11.0.2.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/hbase-client-0.98.1-cdh5.1.0-tests.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/hbase-client-0.98.1-cdh5.1.0.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/hbase-common-0.98.1-cdh5.1.0.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/hbase-protocol-0.98.1-cdh5.1.0.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/hsqldb-1.8.0.10.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/htrace-core-2.04.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/jansi-1.9.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/jasper-compiler-5.5.23.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/jasper-runtime-5.5.23.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/jets3t-0.6.1.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/jetty-6.1.14.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/jetty-util-6.1.26.cloudera.2.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/jline-0.9.94.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/joda-time-1.6.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/jsch-0.1.42.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/jsp-2.1-6.1.14.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/jsp-api-2.1-6.1.14.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/jsr305-1.3.9.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/jyson-1.0.2.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/jython-standalone-2.5.3.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/kfs-0.3.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/netty-3.6.6.Final.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/oozie-sharelib-pig-4.0.0-cdh5.1.0.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/oro-2.0.8.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/parquet-pig-bundle-1.2.5-cdh5.1.0.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/pig-0.12.0-cdh5.1.0.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/protobuf-java-2.5.0.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/servlet-api-2.5-6.1.14.jar
    hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20140918162047/pig/stringtemplate-3.2.1.jar

    • Author
      Hue Team 2 weeks ago

      Indeed, this is correct, in CDH5 the sharelib changed and need to be updated like you did!

Leave a reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>