Using Hadoop MR2 and YARN with an alternative Job Browser interface

Using Hadoop MR2 and YARN with an alternative Job Browser interface

Hue now defaults to using Yarn since version 3.

 

First, it is a bit simpler to configure Hue with MR2 than in MR1 as Hue does not need to use the Job Tracker plugin since Yarn provides a REST API. Yarn is also going to provide an equivalent of Job Tracker HA with YARN-149.

Here is how to configure the clusters in hue.ini. Mainly, if you are using a pseudo distributed cluster it will work by default. If not, you will just need to update all the localhost to the hostnames of the Resource Manager and History Server:

[hadoop]
  ...

  # Configuration for YARN (MR2)
  # ------------------------------------------------------------------------
  [[yarn_clusters]]

    [[[default]]]
      # Enter the host on which you are running the ResourceManager
      resourcemanager_host=localhost

      # The port where the ResourceManager IPC listens on
      resourcemanager_port=8032

      # Whether to submit jobs to this cluster
      submit_to=True

      # URL of the ResourceManager API
      resourcemanager_api_url=http://localhost:8088

      # URL of the ProxyServer API
      proxy_api_url=http://localhost:8088

      # URL of the HistoryServer API
      history_server_api_url=http://localhost:19888

  # Configuration for MapReduce (MR1)
  # ------------------------------------------------------------------------
  [[mapred_clusters]]

    [[[default]]]

      # Whether to submit jobs to this cluster
      submit_to=False

And that’s it! You can now look at jobs in Job Browser, get logs and submit jobs to Yarn!

 

As usual feel free to comment on thehue-user list or@gethue!

3 Comments

  1. wuchang 2 months ago

    the job browser will query out all the jobs ? My system has 10000+ jobs , so the waiting time is always extremely long.

    • Author
      Hue Team 2 months ago

      Hue queries for 1000 jobs max. For finer grain, the user can use the Username filters, Time filter of the Job Status filters…

Leave a reply

Your email address will not be published. Required fields are marked *

*