HBase Browsing with doAs impersonation and Kerberos

HBase Browsing with doAs impersonation and Kerberos

Hue comes with an HBase App that lets you create table, search for rows, read cell content… in just a few clicks. We are now glad to release the last missing piece of security (available in the upcoming Hue 3.8) for making the app production ready!

The HBase app talks to HBase through a proxy server (called Thrift Server V1) which forwards the commands to HBase. Because Hue stands between the proxy server and the actual user, the proxy server thinks that all the operations (e.g. create a table, scan some data…) are coming from the ‘hue’ user and not the actual Web user. This is obviously not very secure!

In order to secure the HBase app for real we need to:

  • make sure that the actual logged in user in Hue performs the operations with his privileges. This is the job of Impersonation.
  • make sure that the Hue server only sends these calls. This is the job of Kerberos strong authentication.

 

Note

We assume that you have installed an HBase Thrift Server in your cluster. If using Cloudera Manager, go to the list of instances of the HBase service and click on ‘Add Role Instances’ and select ‘HBase Thrift Server’.

 

Impersonation

HBase can now be configured to offer impersonation (with or without Kerberos). In our case this means that users can send commands to HBase through Hue without losing the fact that they will be ran under their own credentials (instead of the ‘hue’ user).

First, make sure you have this in your hbase-site.xml:

<property>
  <name>hbase.thrift.support.proxyuser</name>
  <value>true</value>
</property>
 
<property>
  <name>hbase.regionserver.thrift.http</name>
  <value>true</value>
</property>

 

Note

If using Cloudera Manager, this is done by typing ‘thrift’ in the configuration search of the HBase service and checking the first two results.

 

Then check in core-site.xml that HBase is authorized to impersonates someone:

<property>
  <name>hadoop.proxyuser.hbase.hosts</name>
  <value>*</value>
</property>
 
<property>
  <name>hadoop.proxyuser.hbase.groups</name>
  <value>*</value>
</property>

 

And finally check that Hue point to a local config directory of HBase specified in its hue.ini:

[hbase]
hbase_conf_dir=/etc/hbase/conf

 

Note

If you are using Cloudera Manager, you might want to select the HBase Thrift server in the Hue configuration and enter something like this in the Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini.

[hbase]
hbase_conf_dir={{HBASE_CONF_DIR}}

 

And that’s it, start the HBase Thrift Server and Hue and you are ready to go!

 

Security with Kerberos

Now that the Hue can send commands to the HBase Thrift Server and tell him to execute them as a certain user, we need to make sure that only Hue is allowed to do this. We are using Kerberos in order to strongly authenticate the users to the HBase service. In our case, the HBase Thrift server will accept commands only if they come from the Hue user only.

Make sure that HBase is configured with Kerberos and that you have this in the hbase-site.xml pointed by Hue:

<property>
  <name>hbase.security.authentication</name>
  <value>KERBEROS</value>
</property>
 
<property>
  <name>hbase.thrift.kerberos.principal</name>
  <value>hbase/_HOST@ENT.CLOUDERA.COM</value>
</property>

 

 

Note

If using Cloudera Manager or regular Thrift without impersonation, make sure to set the “HBase Thrift Authentication” hbase.thrift.security.qop must be set to one of the following:

  • auth-conf: authentication, integrity and confidentiality checking
  • auth-int: authentication and integrity checking
  • auth: authentication only

If using Cloudera Manager, go to “Hbase service > Configuration > Service-Wide / Security : HBase Thrift Authentication ” and select one of the following three options.

And similarly to above, make sure that the hue.ini points to a valid directory with hbase-site.xml:

[hbase]
hbase_conf_dir=/etc/hbase/conf

or

[hbase]
hbase_conf_dir={{HBASE_CONF_DIR}}

 

Note

If using Impersonation, make sure the HTTP/_HOST principal is in the keytab of for their HBase Thrift Server.

 

Restart HBase and Hue, and they should be all secured now!

 

Conclusion

You can now be sure that Hue users can only see or modify what they are allowed to at the HBase level. Hue guarantees that if a user cannot perform a certain operation in the HBase shell, it will exactly the same through Hue (Hue acts like a ‘view’ on top of HBase).

Note that HBase chose to support impersonation through HTTP Thrift, so regular Thrift won’t work when using impersonation. The previous Kerberos support also now makes sense since all the operations are not seeing as coming from the Hue user anymore! More work is on the way to make all these configuration steps only one click.

 

Now it is time to play with the table examples and open-up HBase to all your users!

 

hbase

 

As usual feel free to comment on the hue-user list or @gethue!

 

Note

This error means that the above ‘hadoop.proxyuser.hbase.hosts’ / ‘hadoop.proxyuser.hbase.groups’ properties are not correct:

Api Error: Error 500 User: hbase is not allowed to impersonate romain HTTP ERROR 500 Problem accessing /. 
Reason: User: hbase is not allowed to impersonate bob Caused by:javax.servlet.ServletException: 
User: hbase is not allowed to impersonate bob at org.apache.hadoop.hbase.thrift.ThriftHttpServlet.doPost(ThriftHttpServlet.java:117) at

Note

You might now see permission errors like below.

Api Error: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions (user=admin, scope=default, action=CREATE)...

This is because either:

  • you are using impersonation and your user ‘bob’ does not have enough HBase privileges
  • you are not using impersonation and the ‘hue’ user does not have enough HBase privileges

 

A quick way to fix this is to just give all the permissions. Obviously this is not recommended for a real setup, instead read more about HBase Access Control!

sudo -u hbase hbase shell 
 
hbase(main):004:0> grant 'bob', 'RWC'

 

Note

If you are getting a “Api Error: TSocket read 0 bytes”, this is because Hue does not know that the Thrift Server is expecting Thrift HTTP. Double check that Hue points to an hbase-site.xml that contains the hbase.regionserver.thrift.http property set to true.

A temporary hack would be to insert this in the hue.ini:

[hbase]
use_doas=true

 

Note

“Api Error: maximum recursion depth exceeded” means that the HBase Thrift server is not running as an HTTP Kerberos service.

In the latest Hue 3.8 you should now just get a 401 error instead.

 

Note

buffered transport mode was not tested when using impersonation but might work.

 

Note

If you are getting this error:

Caused by: org.apache.hadoop.hbase.thrift.HttpAuthenticationException: Authorization header received from the client is empty.

You are very probably hitting https://issues.apache.org/jira/browse/HBASE-13069. Also make sure the HTTP/_HOST principal is in the keytab of for their HBase Thrift Server. Beware that as a follow-up you might get https://issues.apache.org/jira/browse/HBASE-14471.

There is also an issue with framed transport which is not supported yet. We recommend to use the buffered transport instead.

 

38 Comments

  1. Dave 3 years ago

    Thanks for the great work and video. Just to clarify, this new feature allows Hue to maintain HBase’s ACLs even down the cell permissions? In other words, I can use my Hadoop group mapper to provide access controls to my Hue user that are enforced by HBase? My team is very interested in maintaining granular security controls on data (cell level) and if Hue can now provide that along with all it’s other features, that’s HU(E)GE! Thanks again!

    • Hue Team 3 years ago

      Yes, all the operations (scans, getRow, mutateRow…) are sent to HBase as if they were coming from the real logged-in user. This now makes the app usable in practice as only the ‘hue’ user could do operations in the past.
      The app is exactly like a view on top of HBase, like the HBase shell but in a browser. HBase is the one doing the permission checking, not Hue.

      We are doing the same in all the other apps, for example like the HDFS File Browser. HBase was a bit long to get impersonation support 😉

  2. Thelmo 3 years ago

    Great article, but I’ve hit an issue and can’t get out of it. The platform: HDP 2.3 with Hue 3.8.1 (also tested with 3.9.0) with IPA server as Kerberos provider.

    Starting the hbase thrift server manually, but then Hue fails to access the thrift server due to an authentication failure. The main reason on the stack trace is:
    org.apache.hadoop.hbase.thrift.HttpAuthenticationException: Authorization header received from the client is empty.
    at org.apache.hadoop.hbase.thrift.ThriftHttpServlet$HttpKerberosServerAction.getAuthHeader(ThriftHttpServlet.java:212)

    So hue is sending an empty authorization header?

    Other Hue apps, like Hive, the file browser and job browser work fine with Kerberos enabled… Any ideas?

    • Hue Team 3 years ago

      “Authorization header received from the client is empty.”

      Might mean that Hue does not point to a valid hbase-site.xml in its ini
      hbase_conf_dir=/etc/hbase/conf

      (and so does not set the impersonation header)

  3. Thelmo 3 years ago

    Hi, Thanks for the answer. Sorry for using the blog comment box for support 🙁

    Anyway, I’ve checked and double checked all the configuration (Belive me! I’m pulling my hair with this one), and the hbase-site.xml file is accessible to Hue. I’ve used the directory with and without trailing slash, and also, in HDP the directory is a symbolic link, changed it to the real directory and no luck.
    Doing a tcpdump/wireshark dump of the request shows that no Authorization header is sent…. Changed the keytabs/principals, and so on. Only with the HBase app that I have this problem on a Kerberized HDP cluster. FileBrowser, Hive, Job browser, work ok.

  4. Thelmo 3 years ago

    Ok. Solved. After some fiddling around, the correct principal for the thrift kerberos user is HTTP/[email protected] and the correct keytab.

    • Hue Team 3 years ago

      Glad to hear!

    • John 2 years ago

      What about your “hbase.thrift.keytab.file” property?

    • Dhaval Patel 2 years ago

      can you please provide detail on how you solve it. i am struggling big time after upgrading from 2.3.6 to 2.5.3

  5. Tom 3 years ago

    Hi,
    I have HDP 2.3 an Hue 3.9 and similar issue. After reading this comments I changed my user to HTTP/[email protected] and now Hbase in Hue hungs instead of “Api Error: Unable to authenticate” information. There is still error: “Authorization header received from the client is empty” in hbase thrift server log. Thelmo could you send your config parameters, please.

    • Daniel 11 months ago

      Have you solved this?

  6. Murali Meesala 3 years ago

    I followed the above steps to setup HUE 3.9.0 over HDP 2.2.6. I see the below error with Data Browsers –> HBASE

    Api Error: Error 405 Method Not Allowed HTTP ERROR 405 Problem accessing /. Reason: Method Not AllowedPowered by Jetty://

    Can someone help me to resolve it.

    Appreciate your help.

    • Hue Team 3 years ago

      Looks like your HBase might be different. For information this was tested on 1.0 (CDH5.4) and a few more patches.

  7. Murali Meesala 3 years ago

    We are using HBase _0.98.4.2.2 on HDP-2.2.6.0-2800 stack.

    • Hue Team 3 years ago

      We recommend 1.0 to be sure to have the necessary patches.

  8. Anand 2 years ago

    I followed the steps but getting API Error :

    Api Error: Could not start SASL: Error in sasl_client_start (-1) SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available)

    Please help me.

    • Hue Team 2 years ago

      Your kerberos configuration is incorrect, you could first check if the Hue ticket is valid

  9. Anand 2 years ago

    How do i specify hue ticket configurations for kerberos.

  10. stronger 2 years ago

    Hi, I use CDH 5.6.0 、HBase 1.0.0-cdh5.6.0,hue3.9.0.
    Everything is OK includes hive、pig、HDFS File Browser,except hbase browser,
    I configure hbase and hue as above in impersonation mode,but when click hbase browser,i got:

    User: hbase is not allowed to impersonate hueCaused by:javax.servlet.ServletException: User: hbase is not allowed to impersonate hue at org.apache.hadoop.hbase.thrift.ThriftHttpServlet.doPost(ThriftHttpServlet.java:117) at

    • Author
      Hue Team 2 years ago

      The hbase user is probably not in the list of Hadoop proxy users in the core-site.xml

      • stronger 2 years ago

        I config the key in CDH’s HDFS config:
        hadoop.proxyuser.hue.groups *
        hadoop.proxyuser.hue.hosts *

  11. Jay Purkayastha 2 years ago

    Hi, I have a HBase Thrift server started successfully, which I can verify through http://host:9095/thrift.jsp. I’ve configured to execute HBase queries. But from the Hue web console when I got to Data Browsers->HBase, I get Api Error: unpack requires a string argument of length 4. The section under the ‘Table Name’ column keeps spinning. I’m not able to figure out the problem. I’d really appreciate any help with this.

    • Author
      Hue Team 2 years ago

      Which HBase version are you using?

  12. marcello 2 years ago

    i am getting error too
    now my thrift server return
    “user root is not allowed to impersonate admin”
    how can i change this ? so i can run hbase thrift and configure it to hue..
    thank you very mush

    • Author
      Hue Team 2 years ago

      You need to add ‘root’ as a proxyuser, like explained for the hue user in the post.

  13. Debra Montague 1 year ago

    Hi Hue Team,
    We kerberized our HDP cluster and now we are not able to access hbase tables or the use the file browser in HUE. HIVE & Job Browser work successfully. As far as HBASE, I was originally getting this error: The kerberos principal name is missing from the hbase-site.xml configuration file.
    I then followed your instructions above and set up these properties in core-site and hbase-site and restarted all services including HUE:
    hbase.thrift.support.proxyuser=true, hbase.thrift.support.proxyuser=true, hbase.regionserver.thrift.http=true, hbase.thrift.kerberos.principal=HTTP/[email protected], hadoop.proxyuser.hbase.hosts=*,
    hadoop.proxyuser.hbase.groups=*.

    Now I don’t receive any errors but nothing happens it doesn’t show the HBASE tables. The processing icon just keeps processing but it doesn’t show the data. What else should I do. Oh and I have the kerberos settings in the hue.ini file.

    Thanks,

    • Author
      Hue Team 1 year ago

      What logs do you see in the /logs page of Hue after trying to open HBase?
      Anything in the HBase Thrift Server logs too?

      • Debra Montague 1 year ago

        Hello Hue Team,
        I see these errors in runcpserver.logs

        File “/usr/local/hadoop_apps/hue/apps/hbase/src/hbase/views.py”, line 76, in api_router
        return api_dump(HbaseApi(request.user).query(*url_params))
        File “/usr/local/hadoop_apps/hue/apps/hbase/src/hbase/api.py”, line 54, in query
        raise PopupException(_(“Api Error: %s”) % e.message)
        PopupException: Api Error: (‘Connection aborted.’, BadStatusLine(‘\x04\x00\x00\x00\x11Invalid status 80’,))

        [17/May/2017 11:33:44 -0700] middleware INFO Processing exception: Api Error: (‘Connection aborted.’, BadStatusLine(‘\x04\x00\x00\x00\x11Invalid status 80’,)): Traceback (most recent call last):
        File “/usr/local/hadoop_apps/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/core/handlers/base.py”, line 112, in get_response
        response = wrapped_callback(request, *callback_args, **callback_kwargs)
        File “/usr/local/hadoop_apps/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/db/transaction.py”, line 371, in inner
        return func(*args, **kwargs)
        File “/usr/local/hadoop_apps/hue/apps/hbase/src/hbase/views.py”, line 76, in api_router
        return api_dump(HbaseApi(request.user).query(*url_params))
        File “/usr/local/hadoop_apps/hue/apps/hbase/src/hbase/api.py”, line 54, in query
        raise PopupException(_(“Api Error: %s”) % e.message)
        PopupException: Api Error: (‘Connection aborted.’, BadStatusLine(‘\x04\x00\x00\x00\x11Invalid status 80’,))

        I see this error in the thrift server logs:

        2017-05-17 13:22:12,103 ERROR [thrift-worker-9] thrift.TBoundedThreadPoolServer: Error occurred during processing of message.
        java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: Invalid status 80
        at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
        at org.apache.hadoop.hbase.thrift.TBoundedThreadPoolServer$ClientConnnection.run(TBoundedThreadPoolServer.java:283)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
        Caused by: org.apache.thrift.transport.TTransportException: Invalid status 80
        at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232)
        at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:184)
        at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
        at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
        at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
        at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
        … 4 more

      • Debra Montague 1 year ago

        Hello Hue Team,

        Any reply to the logs I uploaded last week?

        Thanks,

        • Author
          Hue Team 1 year ago

          I don’t remember seeing this error while implementing it 🙁

  14. Eda 1 year ago

    Hi guys,
    I have two clusters with the same config. On the first cluster HBase in HUE is working good (an error “Authorization header received from the client is empty” appears but I can list, create etc. HBase tables). On my second cluster I am also obtaining “Authorization header received from the client is empty” error, but also in DEBUG mode I see “token.AuthenticationTokenSelector: No matching token found […] security.HBaseSaslRpcClient: SASL client context established. Negotiated QoP: auth […] Call exception” and HBase in HUE is not working. After 34 tries I get MasterNotRunningException. The thing is that I had Kerberized cluster and it was working, then I turned off Kerberos, and then I turned it on again and it is not working now.
    I am using HUE-3.11 and HDP2.5. I am pretty sure that my thrift server is working good. Also HBase is working correctly using shell.
    Any ideas?
    Thank you

    • Kevin LEFEVRE 10 months ago

      I have the same problem.

      As HUE team pointed out:
      You are very probably hitting https://issues.apache.org/jira/browse/HBASE-13069. Also make sure the HTTP/_HOST principal is in the keytab of for their HBase Thrift Server. Beware that as a follow-up you might get https://issues.apache.org/jira/browse/HBASE-14471.

      I don’t know what to do, my HDP 2.6 still hase the HBase version with the bug, and i can’t upgrade hbase 1.1.2 to hbase 1.1.3 at least where the bug has been fixed….

  15. joy 6 months ago

    After following steps using Cloudera Manager, getting below error while trying to open HBase Browser from Hue, please help.

    ” Api Error: The kerberos principal name is missing from the hbase-site.xml configuration file.”

Leave a reply

Your email address will not be published. Required fields are marked *

*

This site uses Akismet to reduce spam. Learn how your comment data is processed.