YARN Resource Manager High Availability (HA) in MR2

18 June 2014 in Administration / Browsing - 1 minute read

Similarly to the JobTracker High Availability configuration for MR1, Hue supports (since today’s date in master or Hue 3.7 or CDH5.1) more than one Resource Manager in case the Resource Manager goes down.

Hue will automatically pick up the active Resource Manager even if it failed over. This is possible because:

  • When submitting Oozie jobs, the logical name of the Resource Manager is used instead of the hostname of the current Resource Manager
  • Job Browser will automatically look for the active Resource Manager API if needed

Here is an example of configuration for the [[yarn_clusters]] section in hue.ini:

[hadoop]

\# Configuration for YARN (MR2)

\# ------------------------

[[yarn_clusters]]

[[[default]]]

\# Whether to submit jobs to this cluster

submit_to=True

\# Name used when submitting jobs

logical_name=ha-rm

\# URL of the ResourceManager API

resourcemanager_api_url=http://gethue-1.com:8088

\# URL of the ProxyServer API

proxy_api_url=http://gethue-1.com:8088

\# URL of the HistoryServer API

history_server_api_url=http://gethue-1.com:19888

[[[ha]]]

\# Enter the host on which you are running the failover Resource Manager

resourcemanager_api_url=http://gethue-2.com:8088

logical_name=ha-rm

submit_to=True

We hope that the multi Resource Manager support will make your life with Hadoop easier!

As usual feel free to send feedback on the hue-user list or @gethue!


comments powered by Disqus

More recent stories

13 November 2019
Visually surfacing SQL information like Primary Keys, Foreign Keys, Views and Complex Types
Read More
31 October 2019
Missing some color? How to improve or add your own SQL syntax Highlighter
Read More
24 October 2019
How to create a HBase table on Kerberized Hadoop clusters
Read More