Tutorial: How to run the Hue unit and integration tests?

Published on 24 July 2014 in Development - 3 minutes read - Last modified on 19 April 2021

December 2020: GitHub pull requests are now prefered https://docs.gethue.com/developer/development/#development-process

After the Review Board post, here is another tutorial about becoming an awesome Hue developer: how to run the Hue tests!

Hue has two types of tests:

  • unit tests
  • integration tests

First, clone the Hue repository and make sure that you have all the pre-requisite packages:

git clone git@github.com:cloudera/hue.git

Unit Tests

The regular unit tests do not require all this setup! Just run them directly:

build/env/bin/hue test specific beeswax.tests:test_split_statements &> a; vim a

Note

This requires to have done an initial build of Hue with:

make apps

Integration Tests

Live cluster

The tests will run against the cluster configured in your hue.ini one if you specify:

export LIVE_CLUSTER=true

Mini cluster

The test will run in a mini cluster (mini Hadoop, Oozie, Sqoop2 and Hive) created and destroyed after each run.

Here is how to get started:

./tools/jenkins/jenkins.sh slow

Note

You might have lost all the changes in your local pseudo hue.ini because of the `git clean` done by the script.

In order to avoid this, add this into your `~/.bashrc`:

export SKIP_CLEAN=true

Note

To point to an Impalad and trigger the Impala tests:

export TEST_IMPALAD_HOST=impalad-01.gethue.com

or

./build/env/bin/hue test impalaimpalad-01.gethue.com

 

It is then going to download the 4 latest Hadoop, Oozie, Sqoop2 and Hive and pre-install them for you.

You can CTRL+C and kill the script when you see:

INFO: Oozie webconsole disabled, ExtJS library not specified

New Oozie WAR file with added 'JARs' at /home/romain/projects/hue-master/ext/oozie/oozie-4.0.0-cdh5.1.0/oozie-server/webapps/oozie.war

INFO: Oozie is ready to be started

+ /home/romain/projects/hue-master/ext/oozie/oozie-4.0.0-cdh5.1.0/bin/ooziedb.sh create -sqlfile oozie.sql -run

setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"

Validate DB Connection

DONE

Check DB schema does not exist

DONE

Check OOZIE_SYS table does not exist

DONE

Create SQL schema

DONE

Create OOZIE_SYS table

DONE

Oozie DB has been created for Oozie version '4.0.0-cdh5.1.0'

The SQL commands have been written to: oozie.sql

+ build_sqoop

+ check_mtime /home/romain/.hue_cache/.sqoop_mtime http://archive.cloudera.com/cdh5/cdh/5/sqoop2-1.99.3-cdh5.1.0.tar.gz

+ MTIME_FILE=/home/romain/.hue_cache/.sqoop_mtime

++ curl -Is http://archive.cloudera.com/cdh5/cdh/5/sqoop2-1.99.3-cdh5.1.0.tar.gz

++ awk 'BEGIN {FS=":"} { if ($1 == "Last-Modified") { print substr($2,2) } }'

+ MTIME='Tue, 15 Jul 2014 20'

+ echo 'Tue, 15 Jul 2014 20'

+ diff /home/romain/.hue_cache/.sqoop_mtime -

+ return 0

+ '[' '!' -f /home/romain/.hue_cache/sqoop2-1.99.3-cdh5.1.0.tar.gz ']'

+ SQOOP_DIR=/home/romain/projects/hue-master/ext/sqoop

+ export SQOOP_HOME=/home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0

+ SQOOP_HOME=/home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0

+ mkdir -p /home/romain/projects/hue-master/ext/sqoop

+ rm -rf /home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0

+ echo 'Unpacking /home/romain/.hue_cache/sqoop2-1.99.3-cdh5.1.0.tar.gz to /home/romain/projects/hue-master/ext/sqoop'

Unpacking /home/romain/.hue_cache/sqoop2-1.99.3-cdh5.1.0.tar.gz to /home/romain/projects/hue-master/ext/sqoop

+ tar -C /home/romain/projects/hue-master/ext/sqoop -xzf /home/romain/.hue_cache/sqoop2-1.99.3-cdh5.1.0.tar.gz

+ export SQOOP_CONF_DIR=/home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0/server/conf

+ SQOOP_CONF_DIR=/home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0/server/conf

+ rm -rf /home/romain/projects/hue-master/ext/sqoop/sqoop

+ ln -sf /home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0 /home/romain/projects/hue-master/ext/sqoop/sqoop

+ sed -i.bk 's/12000/${test.port.http}/g' /home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0/server/conf/server.xml

+ sed -i.bk 's/12001/${test.port.shutdown}/g' /home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0/server/conf/server.xml

+ sed -i.bk 's/localhost/${test.host.local}/g' /home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0/server/conf/server.xml

+ sed -i.bk 's|\(common.loader.\*$\)|\1,/home/romain/projects/hue-master/ext/hadoop/hadoop/share/hadoop/common/lib/\*.jar,/home/romain/projects/hue-master/ext/hadoop/hadoop/share/hadoop/mapreduce1/\*.jar,/home/romain/projects/hue-master/ext/hadoop/hadoop/share/hadoop/mapreduce1/lib/\*.jar|g' /home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0/server/conf/catalina.properties

+ sed -i.bk 's|${catalina\.base}/logs|${test.log.dir}|g' /home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0/server/conf/logging.properties

+ sed -i.bk 's|\@LOGDIR\@|${test.log.dir}|g' /home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0/server/conf/sqoop.properties

+ sed -i.bk 's|\@BASEDIR\@|${test.base.dir}|g' /home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0/server/conf/sqoop.properties

+ sed -i.bk 's|/etc/hadoop/conf|${test.hadoop.conf.dir}|g' /home/romain/projects/hue-master/ext/sqoop/sqoop2-1.99.3-cdh5.1.0/server/conf/sqoop.properties

+ make apps

cd /home/romain/projects/hue-master/maven && mvn install

[INFO] Scanning for projects...

[INFO]

[INFO] ------------------------

[INFO] Building Hue Maven Parent POM 3.6.0-SNAPSHOT

[INFO] ------------------------

[INFO]

[INFO] - maven-enforcer-plugin:1.0:enforce (default) @ hue-parent -

[INFO]

[INFO] - maven-install-plugin:2.3:install (default-install) @ hue-parent -

[INFO] Installing /home/romain/projects/hue-master/maven/pom.xml to /home/romain/.m2/repository/com/cloudera/hue/hue-parent/3.6.0-SNAPSHOT/hue-parent-3.6.0-SNAPSHOT.pom

[INFO] ------------------------

[INFO] BUILD SUCCESS

[INFO] ------------------------

[INFO] Total time: 1.488s

[INFO] Finished at: Thu Jul 24 10:46:56 PDT 2014

[INFO] Final Memory: 7M/150M

[INFO] ------------------------

And that's it! You can run all the tests or some parts with this syntax:

build/env/bin/hue test specific filebrowser.views_test:test_listdir_sort_and_filter &> a; vim a

 

Note:

In some cases you might need to clear up your your cache with something like:

rm /home/romain/.hue_cache/.*

then in

ext/

delete everything but

thirdparty/

 

Sum-up

We hope that is now easier to help you contribute to Hue 😉

As usual feel free to send feedback on the hue-user list or @gethue!


comments powered by Disqus

More recent stories

15 November 2021
Create SQL tables from excel files
Read More
21 September 2021
Access your data in ABFS without any credential keys!
Read More
14 September 2021
SSO for REST APIs with your custom JWT authentication
Read More