JobTracker High Availability (HA) in MR1

Published on 30 December 2013 in - 1 minute read - Last modified on 06 March 2021

When the Job Tracker goes down, Hue cannot display the Jobs in File Browser or submit to the correct cluster.

 

In MR1, Hadoop can support two Job Trackers, a master Job Tracker that can fail over to a standby Job Tracker and hence provide Job Tracker HA. Let’s see how Hue 3.5 and CDH5beta1 (and probably CDH4.6) can take advantage of this.

Note: in MR1 Hue is using a plugin to communicate with the Job Tracker. This can be configured in CDH or Hadoop 0.23 / 1.2.0 (MAPREDUCE-461).

 

We configure two Job Trackers in the hue.ini:

[hadoop]

...

[[mapred_clusters]]

[[[default]]]

\# Enter the host on which you are running the Hadoop JobTracker

jobtracker_host=host-1

\# Whether to submit jobs to this cluster

submit_to=True

[[[ha-standby]]]

\# Enter the host on which you are running the Hadoop JobTracker

jobtracker_host=host-2

\# Whether to submit jobs to this cluster

submit_to=True

 

And that’s it! Hue will communicate with the available Job Tracker automatically!

 

Notice that in the case of Oozie jobs, Oozie will try to re-submit the job but will need a logical name (HUE-1631). To enable this in Hue, specify it in each MapReduce cluster, e.g.:

[hadoop]

[[mapred_clusters]]

[[[default]]]

\# JobTracker logical name.

\## logical_name=MY_NAME

 

As usual feel free to comment on the hue-user list or@gethue!


comments powered by Disqus

More recent stories

26 June 2024
Integrating Trino Editor in Hue: Supporting Data Mesh and SQL Federation
Read More
03 May 2023
Discover the power of Apache Ozone using the Hue File Browser
Read More