JobTracker High Availability (HA) in MR1

JobTracker High Availability (HA) in MR1

When the Job Tracker goes down, Hue cannot display the Jobs in File Browser or submit to the correct cluster.

 

In MR1, Hadoop can support two Job Trackers, a master Job Tracker that can fail over to a standby Job Tracker and hence provide Job Tracker HA. Let’s see how Hue 3.5 and CDH5beta1 (and probably CDH4.6) can take advantage of this.

Note: in MR1 Hue is using a plugin to communicate with the Job Tracker. This can be configured in CDH or Hadoop 0.23 / 1.2.0 (MAPREDUCE-461).

 

We configure two Job Trackers in the hue.ini:

[hadoop]
  ...

  [[mapred_clusters]]

    [[[default]]]
      # Enter the host on which you are running the Hadoop JobTracker
      jobtracker_host=host-1

      # Whether to submit jobs to this cluster
      submit_to=True

    [[[ha-standby]]]
      # Enter the host on which you are running the Hadoop JobTracker
      jobtracker_host=host-2

      # Whether to submit jobs to this cluster
      submit_to=True

 

And that’s it! Hue will communicate with the available Job Tracker automatically!

 

Notice that in the case of Oozie jobs, Oozie will try to re-submit the job but will need a logical name (HUE-1631). To enable this in Hue, specify it in each MapReduce cluster, e.g.:

[hadoop]
  [[mapred_clusters]]
    [[[default]]]
      # JobTracker logical name.
      ## logical_name=MY_NAME

 

As usual feel free to comment on the hue-user list or@gethue!

4 Comments

  1. John Ross 6 months ago

    What is the equivalent of JobTracker on HDP/ Ambari?

Leave a reply

Your email address will not be published. Required fields are marked *

*