Using Hadoop MR2 and YARN with an alternative Job Browser interface

Published on 13 January 2014 in Browsing - 1 minute read - Last modified on 04 February 2020

Hue now defaults to using Yarn since version 3.

 

First, it is a bit simpler to configure Hue with MR2 than in MR1 as Hue does not need to use the Job Tracker plugin since Yarn provides a REST API. Yarn is also going to provide an equivalent of Job Tracker HA with YARN-149.

Here is how to configure the clusters in hue.ini. Mainly, if you are using a pseudo distributed cluster it will work by default. If not, you will just need to update all the localhost to the hostnames of the Resource Manager and History Server:

[hadoop]

...

\# Configuration for YARN (MR2)

\# ------------------------

[[yarn_clusters]]

[[[default]]]

\# Enter the host on which you are running the ResourceManager

resourcemanager_host=localhost

\# The port where the ResourceManager IPC listens on

resourcemanager_port=8032

\# Whether to submit jobs to this cluster

submit_to=True

\# URL of the ResourceManager API

resourcemanager_api_url=http://localhost:8088

\# URL of the ProxyServer API

proxy_api_url=http://localhost:8088

\# URL of the HistoryServer API

history_server_api_url=http://localhost:19888

\# Configuration for MapReduce (MR1)

\# ------------------------

[[mapred_clusters]]

[[[default]]]

\# Whether to submit jobs to this cluster

submit_to=False

And that’s it! You can now look at jobs in Job Browser, get logs and submit jobs to Yarn!

 

As usual feel free to comment on thehue-user list or@gethue!


comments powered by Disqus

More recent stories

10 January 2021
SQL Querying a log stream and outputting Calculations to another stream
Read More
31 December 2020
A Spark SQL Editor via Hue and the Spark SQL Server
Read More
17 November 2020
Easy Querying of live Kafka data in a Big Table like HBase with SQL
Read More