Browsing Hive tables data and metadata is getting faster and prettier

Browsing Hive tables data and metadata is getting faster and prettier

Hue continues to boost the experience of SQL users with its improved autocompletes, smarter assist panel, SQL notebooks… and now a revamped Hive Metastore Browser.

The app is now single page and offers speed and more accessible statistics and data preview. See more of the improvements below or take a look at the following video demo.

New UI

Fresh restart

The front-end has been rewritten to be slicker and more user-friendly. More information is displayed and navigating across tabs is seamless as it no longer entails a page refresh.

blog-57-metastore-table

Table specific page after

blog-55-metastore-table.png

Table specific page before

 

blog-57-metastore-tables

List of tables page after

blog-55-metastore-tables

List of tables page before

 

Easy edition

Table and column comments can be edited in a single click.

blog-edit-comments

Getting pretty

Scrolling bars are not flaky and ugly like in the previous version. They are invisible by default, and subtle when displayed

New

Old default scrollbar

Old

 

Creation wizards

We also refreshed the wizard for creating tables and databases quickly, whether designing the schema manually or automatically from a sample file.

blog-57-create table

 

Speed

 

Single page app: first, now the initial page loads very quickly and fetches asynchronously the list of tables, table statistics, data sample, partition list. We are not blocking anymore until everything is queried in Hive. Subsequent navigation clicks will trigger only 1 or 2 calls to the server, instead of reloading all the page resources again. As an added bonus, the browser history now works on all the pages.

meta-slow meta-quick

Caching: The new assist caches all the Hive metadata. The pages listing tables and database also point to the same cache, as well as the editor autocomplete. This means that now the fetching of thousand of Hive tables and databases will only happen once. On the Hive side, these calls have even been optimized for taking seconds instead of previously minutes (HIVE-7575).

New Metastore blog post caching and hue each(1)

Don’t freeze my browser: on top of the caching, Hue is now much smarter thanks to its custom foreachVisible Ko Js [link] binding that displays only the elements visible on the screen. For example if the user has a list of 5000 tables, only tens of them will actually be rendered (the rendering is a costly part). The binding operates a sliding window of top of all this data, supports scrolling and won’t block your browser tab or even crash it anymore when displaying large list of objects.

New Metastore blog post caching and hue each

Data rich

Statistics

Stats are easier to access than before. Most of the database and table metadata are displayed on top. Top values, min, max can also be retrieved and refreshed directly from the UI. Links to HDFS locations are automatically added.blog-metastore-stats

blog-col-stats

 

Preview data

The first few rows of table data are displayed on the first page to provide users with a convenient quick preview.

blog-metastore-preview-data

Preview partitions

Partitions keys and a sample of partition values can also be found on the main page. Advanced users can filter and query all the partitions in the partition browser.

Metastore Partition View

 

The Hue team hopes that these new features will make your SQL data search and discovery easier! The metastore app will keep getting better by making data partitions and indexes easier to access and by adding more table, column, usage, top queries statistics. More is on a way, with a brand new SQL Editor!

Feel free to send feedback on the hue-user list or @gethue!

10 Comments

  1. Antonio B. 1 year ago

    Hello.
    The paragraph on caching mentions some Hive change improving the performance of some actions: “On the Hive side, these calls have even been optimized for taking seconds instead of previously minutes (HIVE-….)”.
    What Hive ticket or commit would that be?

    Cheers

  2. Tomasz U 6 months ago

    “-1 rows” in blog-metastore-stats.png looks odd, is it a mistake?

    • Author
      Hue Team 6 months ago

      It means the stats have not been calculated yet or are not accurate/fresh.

  3. Daniel 5 months ago

    Hi, I created a table in the SQL editor without column comments. So is showing: “from deserializer”, I change the comment but it seems that is not updating, because when I come back is showing “from deserializer” again. Any thoughts?

    Thanks!

    • Daniel 5 months ago

      By the way, I’m using Hue 3.12

      • Daniel 5 months ago

        Apparently the problem is because i’m using:
        ROW FORMAT SERDE ‘org.apache.hadoop.hive.serde2.OpenCSVSerde’ in the table creation.

        Anyway, I created a view to point to that table, but when I try to update the column comments I have an error:

        Failed to alter column `default`.`fc_full_view` `id`: Bad status for request TExecuteStatementReq(confOverlay={}, sessionHandle=TSessionHandle(sessionId=THandleIdentifier(secret=’z2\x8b\xa6\xe2\xb8Gi\xaf\x90\x02\x1e\x021\xaeK’, guid=’\xaei>e\xa7\xe9OJ\xb8\xc1 +\x1c\xc2~\x1a’)), runAsync=True, statement=”ALTER TABLE `default`.`fc_full_view` CHANGE COLUMN `id` `id` INT COMMENT ‘ID ‘”): TExecuteStatementResp(status=TStatus(errorCode=10131, errorMessage=’Error while compiling statement: FAILED: SemanticException [Error 10131]: To alter a view you need to use the ALTER VIEW command.’, sqlState=’42000′, infoMessages=[‘*org.apache.hive.service.cli.HiveSQLException:Error while compiling statement: FAILED: SemanticException [Error 10131]: To alter a view you need to use the ALTER VIEW command.:47:46’, ‘org.apache.hive.service.cli.operation.Operation:toSQLException:Operation.java:335’, ‘org.apache.hive.service.cli.operation.SQLOperation:prepare:SQLOperation.java:148’, ‘org.apache.hive.service.cli.operation.SQLOperation:runInternal:SQLOperation.java:226’, ‘org.apache.hive.service.cli.operation.Operation:run:Operation.java:276’, ‘org.apache.hive.service.cli.session.HiveSessionImpl:executeStatementInternal:HiveSessionImpl.java:468’, ‘org.apache.hive.service.cli.session.HiveSessionImpl:executeStatementAsync:HiveSessionImpl.java:456’, ‘sun.reflect.GeneratedMethodAccessor101:invoke::-1’, ‘sun.reflect.DelegatingMethodAccessorImpl:invoke:DelegatingMethodAccessorImpl.java:43’, ‘java.lang.reflect.Method:invoke:Method.java:498’, ‘org.apache.hive.service.cli.session.HiveSessionProxy:invoke:HiveSessionProxy.java:78’, ‘org.apache.hive.service.cli.session.HiveSessionProxy:access$000:HiveSessionProxy.java:36’, ‘org.apache.hive.service.cli.session.HiveSessionProxy$1:run:HiveSessionProxy.java:63’, ‘java.security.AccessController:doPrivileged:AccessController.java:-2’, ‘javax.security.auth.Subject:doAs:Subject.java:422’, ‘org.apache.hadoop.security.UserGroupInformation:doAs:UserGroupInformation.java:1724’, ‘org.apache.hive.service.cli.session.HiveSessionProxy:invoke:HiveSessionProxy.java:59’, ‘com.sun.proxy.$Proxy23:executeStatementAsync::-1’, ‘org.apache.hive.service.cli.CLIService:executeStatementAsync:CLIService.java:298’, ‘org.apache.hive.service.cli.thrift.ThriftCLIService:ExecuteStatement:ThriftCLIService.java:506’, ‘org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement:getResult:TCLIService.java:1317’, ‘org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement:getResult:TCLIService.java:1302’, ‘org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39’, ‘org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39’, ‘org.apache.thrift.server.TServlet:doPost:TServlet.java:83’, ‘org.apache.hive.service.cli.thrift.ThriftHttpServlet:doPost:ThriftHttpServlet.java:206’, ‘javax.servlet.http.HttpServlet:service:HttpServlet.java:727’, ‘javax.servlet.http.HttpServlet:service:HttpServlet.java:820’, ‘org.eclipse.jetty.servlet.ServletHolder:handle:ServletHolder.java:565’, ‘org.eclipse.jetty.servlet.ServletHandler:doHandle:ServletHandler.java:479’, ‘org.eclipse.jetty.server.session.SessionHandler:doHandle:SessionHandler.java:225’, ‘org.eclipse.jetty.server.handler.ContextHandler:doHandle:ContextHandler.java:1031’, ‘org.eclipse.jetty.servlet.ServletHandler:doScope:ServletHandler.java:406’, ‘org.eclipse.jetty.server.session.SessionHandler:doScope:SessionHandler.java:186’, ‘org.eclipse.jetty.server.handler.ContextHandler:doScope:ContextHandler.java:965’, ‘org.eclipse.jetty.server.handler.ScopedHandler:handle:ScopedHandler.java:117’, ‘org.eclipse.jetty.server.handler.HandlerWrapper:handle:HandlerWrapper.java:111’, ‘org.eclipse.jetty.server.Server:handle:Server.java:349’, ‘org.eclipse.jetty.server.AbstractHttpConnection:handleRequest:AbstractHttpConnection.java:449’, ‘org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler:content:AbstractHttpConnection.java:925’, ‘org.eclipse.jetty.http.HttpParser:parseNext:HttpParser.java:857’, ‘org.eclipse.jetty.http.HttpParser:parseAvailable:HttpParser.java:235’, ‘org.eclipse.jetty.server.AsyncHttpConnection:handle:AsyncHttpConnection.java:76’, ‘org.eclipse.jetty.io.nio.SelectChannelEndPoint:handle:SelectChannelEndPoint.java:609’, ‘org.eclipse.jetty.io.nio.SelectChannelEndPoint$1:run:SelectChannelEndPoint.java:45’, ‘java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1142’, ‘java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:617’, ‘java.lang.Thread:run:Thread.java:745’, ‘*org.apache.hadoop.hive.ql.parse.SemanticException:To alter a view you need to use the ALTER VIEW command.:56:10’, ‘org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer:validateAlterTableType:DDLSemanticAnalyzer.java:1296’, ‘org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer:addInputsOutputsAlterTable:DDLSemanticAnalyzer.java:1492’, ‘org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer:addInputsOutputsAlterTable:DDLSemanticAnalyzer.java:1438’, ‘org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer:analyzeAlterTableRenameCol:DDLSemanticAnalyzer.java:2649’, ‘org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer:analyzeInternal:DDLSemanticAnalyzer.java:280’, ‘org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer:analyze:BaseSemanticAnalyzer.java:230’, ‘org.apache.hadoop.hive.ql.Driver:compile:Driver.java:464’, ‘org.apache.hadoop.hive.ql.Driver:compile:Driver.java:320’, ‘org.apache.hadoop.hive.ql.Driver:compileInternal:Driver.java:1219’, ‘org.apache.hadoop.hive.ql.Driver:compileAndRespond:Driver.java:1213’, ‘org.apache.hive.service.cli.operation.SQLOperation:prepare:SQLOperation.java:146’], statusCode=3), operationHandle=None)

        It seems that is trying to do an ALTER TABLE instead of an ALTER VIEW?

  4. sravas 4 weeks ago

    Hi,

    while uploading my csv file into table that created in metastore management database i found below error. but i could successfully uploaded text file where csv file with “;” delimeter.

    Error while compiling statement: FAILED: RuntimeException java.lang.RuntimeException:

Leave a reply

Your email address will not be published. Required fields are marked *

*