JobTracker High Availability (HA) in MR1

30 December 2013 in Browsing - 2 minutes read

When the Job Tracker goes down, Hue cannot display the Jobs in File Browser or submit to the correct cluster.

 

In MR1, Hadoop can support two Job Trackers, a master Job Tracker that can fail over to a standby Job Tracker and hence provide Job Tracker HA. Let’s see how Hue 3.5 and CDH5beta1 (and probably CDH4.6) can take advantage of this.

Note: in MR1 Hue is using a plugin to communicate with the Job Tracker. This can be configured in CDH or Hadoop 0.23 / 1.2.0 (MAPREDUCE-461).

 

We configure two Job Trackers in the hue.ini:

[hadoop]

...

[[mapred_clusters]]

[[[default]]]

\# Enter the host on which you are running the Hadoop JobTracker

jobtracker_host=host-1

\# Whether to submit jobs to this cluster

submit_to=True

[[[ha-standby]]]

\# Enter the host on which you are running the Hadoop JobTracker

jobtracker_host=host-2

\# Whether to submit jobs to this cluster

submit_to=True

 

And that’s it! Hue will communicate with the available Job Tracker automatically!

 

Notice that in the case of Oozie jobs, Oozie will try to re-submit the job but will need a logical name (HUE-1631). To enable this in Hue, specify it in each MapReduce cluster, e.g.:

[hadoop]

[[mapred_clusters]]

[[[default]]]

\# JobTracker logical name.

\## logical_name=MY_NAME

 

As usual feel free to comment on the hue-user list or@gethue!


comments powered by Disqus

More recent stories

13 November 2019
Visually surfacing SQL information like Primary Keys, Foreign Keys, Views and Complex Types
Read More
31 October 2019
Missing some color? How to improve or add your own SQL syntax Highlighter
Read More
24 October 2019
How to create a HBase table on Kerberized Hadoop clusters
Read More