Dynamic Search Dashboard improvements!

Dynamic Search Dashboard improvements!

Apache Solr is getting easier to use! This new version of the Search app provides a general improved dashboard experience by refreshing only the widgets that changed and integrating better with live indexing. A series of additional new functionalities are detailed below.

See it all in action in the video demo and the Bike Data analysis with Analytics Facets widgets and Real time indexing of Tweet with Spark Streaming posts!

 

 

Live filtering when moving on the map

Points close to each other are grouped together and will expand when zooming-in. A Yelp-like search filtering experience can also be created by checking the box.

search-marker-map

 

 

Edit indexed records and link to original documents

Indexed records can be directly edited in the Grid or HTML widgets by admins.

Links to the original documents can also be inserted. Add to the record a field named ‘link-meta’ that contains some json describing the URL or address of a table or file that can be open in the HBase Browser, Metastore App or File Browser:

Any link

{'type': 'link', 'link': 'gethue.com'}

HBase Browser

{'type': 'hbase', 'table': 'document_demo', 'row_key': '20150527'}
{'type': 'hbase', 'table': 'document_demo', 'row_key': '20150527', 'fam': 'f1'}
{'type': 'hbase', 'table': 'document_demo', 'row_key': '20150527', 'fam': 'f1', 'col': 'c1'}

File Browser

{'type': 'hdfs', 'path': '/data/hue/file.txt'}

Metastore

{'type': 'hive', 'database': 'default', 'table': 'sample_07'}

search-link

 

 

Export/import/Share saved dashboards

Dashboard can be selected and exported as a Hue document (a ‘json’ file’, like in the Oozie Editor) directly from the interface. This allow to back-up them more easily until we introduce git support. It is also convenient for transferring dashboards between clusters.

search-export

search-import

 

Save and reload the full search query definition

Current selected facets and filters, query strings can be saved with a name within the dashboard. These are useful for defining “cohorts” or pre-selection of records and quickly reloading them.

search-query-def

‘Fixed’ or ‘rolling’ time window filtering

Real time indexing can now shine with the rolling window filter and the automatic refresh of the dashboard every N seconds. See it in action in the real time Twitter indexing with Spark streaming post.

search-rolling-timesearch-fixed-time

 

Full mode Player display

The dashboard experience is even more real with this new browser full screen mode (and even better with full screen with the F11 key).

search-full-mode

 

Preview of nested Analytics facets

Solr 5.1 is seeing new Analytics Facets. A beta support for them has been added and can be enabled in the hue.ini with:

[search]
latest=true

A more comprehensive demo is available on the BikeShare data visualization post.

search-nested-facet search-hit-widget

 

 

So it is time to create more dashboards! Feel free to send feedback on the hue-user list or @gethue!

60 Comments

  1. Ruslan 1 year ago

    Wow, that’s really cool. A couple of questions:
    1. Which CDH version do you plan to include Solr 5.x in?
    2. Does this feature also support nested collections? E.g. support for nested data types is coming in Impala – http://conferences.oreilly.com/strata/big-data-conference-ny-2015/public/schedule/detail/43568 .. it would be interesting to have the same in Solr. Or is it already available in Solr 4.10?
    Thanks!

  2. Fred 1 year ago

    Hi All,
    Completly new in Haddop’s world. I try to run a search samply but I get lot of error. Could you provide a step by step sample, starting from the index creation ?
    Fred

  3. Fred 1 year ago

    Sorry, here is my configuration :
    Hadoop : 2.6
    Hue : 3.9
    Solr : 5.2.1

  4. Fred 1 year ago

    Thanks for your quick answer. Do you confirm this sample will work with my configuration ? because it seems I cannot run “solrctl”. I sounds like this command is deprecated. Maybe I’ve to use “solr” instead. Do you confirm ?
    I’ve installed samples in Hue. I can see it. but when I choose “Web Logs” sample, Dashboards begin to charge and I get the following error :
    Error 404 Not Found HTTP ERROR 404 Problem accessing /solr/log_analytics_demo/select. Reason: Not FoundPowered by Jetty:// (error 404)

    • Hue Team 1 year ago

      Ha, if you are using Upstream Solr I don’t think the indexer works yet with it.

      If you are looking at getting some indexes up, the README of Solr explains how to use their builtin examples. http://yonik.com/ has some good post with mini datasets.

  5. Fred 1 year ago

    Surprised about your answer. I don’t use Upstream, I just want to use these products to prepare a POC. Reading this post, I thought it was possible with the Tools I’ve already installed (Hue, Hbase, Hive, Solr, …), I’ve never read anything about Upstream. Any help possible about the error I receive ? And what about “solr” command instead of “solrctl” ? Anyway, I’m going to try again.

    • Hue Team 1 year ago

      Upstream means the latest from any repository, like nightly binaries or github repos. What about downloading Cloudera’s Quickstart VM (http://go.cloudera.com/vm-download) and play with all the tools there? No need for configuration, and Hue is included! 🙂

  6. Fred 1 year ago

    Ok I’m going to think about this solution even if I did not want to install Cloudera … Does It means that it definitively won’t works by this way ?It seems I’m closed to make it work. Didn’t you already solved this issue :
    Error 404 Not Found HTTP ERROR 404 Problem accessing /solr/log_analytics_demo/select. Reason: Not FoundPowered by Jetty:// (error 404)

  7. Fred 1 year ago

    Sorry, additional question about it. Do I have to upload (and/or modify) some files (schema.xml, solrconfig.xml, …) into HDFS (for example into solr directory) to run this sample ? I’ve located the directory in /home/hue/apps/search/examples/collections/solr_configs_log_analytics_demo/conf

  8. Fred 1 year ago

    Thanks for your recommandations. I’m now able to create Dashboard by this way. Now, I’m looking for by which way dashboard could be automaticaly & dynamically refreshed when some data are added or removed for example ?

  9. Fred 1 year ago

    Ok … I’ve spent few hours to try to use Spark. I’ve tried to run it … HUE configuration (I’ve configured the Hue.ini) page tells me : The app won’t work without a running Livy Spark Server
    So I’ve tried to run Livy. Impossible.
    I found a “spark” directory into : /home/hue/apps, so I’ve tried to run the command line (as mentionned in your tutorial : ./bin/livy-server :
    Always the same error : you need to build Livy before running this program. Following the same tutorial, the command : mvn -DskipTests clean package doesn’t work too.
    Could you help me ?

    • Hue Team 1 year ago

      How did you install Hue? CDH/Cloudera Manager or from master?
      In any case, you can compile Livy like this
      cd /home/hue/apps/spark/java && mvn clean install -DskipTests
      Of course you will need some development requirement installed (like JDK, Maven, …)

  10. Fred 1 year ago

    Sorry for my late answer but I could not access to your web site. Anyway, I’ve installed spark and it seems to be right now. I thought I could use spark provided by HUE 3.9 installation. I’m going to continue now and will let you know if I still have some problem. Thanks again for your help.

  11. Fred 1 year ago

    Hi,
    Following your tutorial … I’m able to run “apache_log.py sample” but I cannot create the collection in HUE.
    Here is the error msg I receive (I hope you will be able to read it) :

    { “message”: “{\”responseHeader\”:{\”status\”:400,\”QTime\”:126},\”Operation create caused exception:\”:\”org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Can not find the specified config set: apache_logs\”,\”exception\”:{\”msg\”:\”Can not find the specified config set: apache_logs\”,\”rspCode\”:400},\”error\”:{\”msg\”:\”Can not find the specified config set: apache_logs\”,\”code\”:400}}\n (error 400)”, “traceback”: [ [ “/home/hue/desktop/libs/libsolr/src/libsolr/api.py”, 470, “create_collection”, “response = self._root.post(‘admin/collections’, params=params, contenttype=’application/json’)” ], [ “/home/hue/desktop/core/src/desktop/lib/rest/resource.py”, 122, “post”, “return self.invoke(\”POST\”, relpath, params, data, self._make_headers(contenttype, headers))” ], [ “/home/hue/desktop/core/src/desktop/lib/rest/resource.py”, 78, “invoke”, “urlencode=self._urlencode)” ], [ “/home/hue/desktop/core/src/desktop/lib/rest/http_client.py”, 161, “execute”, “raise self._exc_class(ex)” ] ], “detail”: “None”, “title”: “Erreur lors de l’acc\u00e8s \u00e0 Solr” }

    • Hue Team 1 year ago

      Yes, Hue indexer is in beta and does not support non Cloudera Solr for now. You need to create the collection outside of Hue, then use the Hue Search Dashboard

  12. Fred 1 year ago

    Hi,
    Once again, thanks for your help. I’ve followed the tutorial ( http://gethue.com/build-a-real-time-analytic-dashboard-with-solr-search-and-spark-streaming/) and installed spark-solr. Could you provide to me help on how to use it to ingest dynamically log-apache logs. For example how to use spark-defaults.conf, …

  13. Baris 9 months ago

    Hi,

    Is there any option for keeping filters permanently? query definition is just keeping the query for current session, so when I login or refresh the page I lose the saved queries.

    Thanks.

    • Hue Team 9 months ago

      Yes, look at the “Save and reload the full search query definition” section

      • Baris 9 months ago

        Yes I know but query definition section just keeps the query for current section. When I refresh the page the saved queries will be disappeared. thanks

  14. Aniruddh Sharma 8 months ago

    i am using CDH5.7 which has Solr 4.10.3. Can you please guide me how to install solr version 5.2.1 on Cloudera in order to try this out.

    • Hue Team 8 months ago

      Unfortunately Solr 5 is not supported in CDH 5.7, so you will need to install in on the side.

  15. Jochen 7 months ago

    Is there a workaround to provide multi shard search in one dasboard?

  16. Yuri 6 months ago

    I’ve upgraded to CDH 5.8.0 but Solr is still 4.10.3. I can see secondary dimensions on charts in the examples
    – Yelp Reviews http://hue.site:8888/search/?collection=3 (Stars and States)
    – Web Logs http://hue.site:8888/search/?collection=1 (Country and Browser)

    I can’t find the way to group my chart same way. Shall I migrate to Solr5 or there is some hidden features?

  17. PRASANNA S KRISHNAN 5 months ago

    hello –

    I have indexed a csv file containing (filename, vendor which is extracted from filename, full unix destination path of the file)
    The dashboards look good and the filters work great. But, I seem to have some issues.

    1. Text facet (filtering) does not have scroll bar. It is a pain to keep hitting show more to find desired value if the list is big.
    2. When I do a search for a vendor, I expect all fields/records containing that vendor name only to appear on the result grid. But, it is not happening. There is some highlight if there are matches, but it is not filtering based on search criteria
    3. Does Hue dashboards/search support partial text matching?

  18. Chris 3 months ago

    Hey I have a hue dashboard with predefined query definitions. Filter Date=1/1/2016. I get a URL to the search which i can distribute to users. Can i change the query definition dynamically? Say i want to change from Date from 1/1/2016 to 1/2/2016 and have the dashboard displayed. Can i do this by passing parameters in the URL?

    https://hello.com/search/?collection=200#q=83114978-cbf9-28ec-a047-c5e348320bce

  19. Chris 3 months ago

    Tried the following
    https://hello.com/search/?collection=200&System:TEST
    https://hello.com/search/?collection=200#q=System:TEST
    https://hello.com/search/?collection=200&q=System:TEST
    None of the above seem to work

    I actually have multiple filter parameters. Is there a work around? Also can you give me an example of multiple filter parameters?

  20. Chris 3 months ago

    Can you give me an example of sending multiple query parameters to Hue search dashboard?

    • Author
      Hue Team 3 months ago

      At the end of the search input, there is a ‘+’ icon that will add one more query box.

      • Chris 3 months ago

        Are you talking about the + Icon in the GUI? I want to pass multiple parameters in the Hue Dashboard URL. Can you please help me with an example of Hue Dashboard URL with multiple search parameters? I cant seem to get it to work.

  21. Prasanna 2 months ago

    Hello

    I am using Hue 3.9.0 and CDH 5.7.3 with parcels.

    A couple of questions –
    1. I have a dashboard with grid based results and a bunch of pie/bar charts, text facets sourced from a solr index. Now, is there a way display static text (say legend or some helpful hints for users) anywhere in the dashboard? The text facets currently displays the actual field name from the index. I have changed it to a readable format. But, would like to display the actual fieldnames so that users can search using them, if needed.

    2. Under current configuration, is there a way to add custom widgets to the dashboard? If so, can you please point me to a good starting point? Thanks.

    • Author
      Hue Team 2 months ago

      #1 You can edit the Title of each widgets
      #2 Not out of the box, you would need to modify the code youself

      • Prasanna 2 months ago

        Is there a way display static text anywhere in the dashboard?

        • Author
          Hue Team 2 months ago

          No, except in the title of the widgets. Are you thinking in the sense of a Markdown widget or something like that?

          • Prasanna 2 months ago

            Yes. Something like that – to display additional comments, texts, links etc. A simple text display widget.

          • Author
            Hue Team 2 months ago

            Created https://issues.cloudera.org/browse/HUE-5550 to track the feature

  22. Varun 2 months ago

    i’m using HDP 2.5. I’m trying to access hbase in hue 3.11. Hbase configuartions

    hbase-site.xml:

    hbase.thrift.support.proxyuser true hbase.regionserver.thrift.http true

    core-site.xml

    hadoop.proxyuser.hbase.hosts * hadoop.proxyuser.hbase.groups *

    kerberos setup and principal has been successfully created and configured in hbase-site.xml.

    Thrift sever started as below command.

    hbase thrift start –infoport 9095 -p 9090. Corresponding configured in hue.ini.

    Still now i can’t able to resolve this hbase authentication issue.

    i got log in hue supervisor (‘Connection aborted.’, BadStatusLine(”,))

  23. Barış 2 months ago

    Hi,

    I am using HUE 3.9.0 with CDH 5.7.1.

    I applied hbase lily indexer so every record in hbase is indexed to solr. I can not see the Edit option or show original document option in my dashboard. How can I see these options? Is there any patch etc?

    Thanks

    • Author
      Hue Team 2 months ago

      Are you logged in as a Hue admin? Currently those are the only one that can edit indexes.

      • Barış 2 months ago

        No I did not log in with admin user. Must I log in with admin user to see the edit and show orginal doc. buttons ? Thanks

        • Author
          Hue Team 2 months ago

          Yes, Hue admins are currently the only ones that can edit indexes.

          • Barış 1 month ago

            Thanks a lot.

  24. Barış 1 month ago

    Hi,

    I am using HUE 3.9.0 with CDH 5.7.1.

    Can I read the query definitions that saved on dashboards from external code? I thought that if I can find the path where the query definitions keeped on server then I can read and use them in my project.

    How can I read the saved searches ? If there is any way to do that, could you share with me?

    Thanks a lot..

    • Author
      Hue Team 1 month ago

      Query definitions can’t be read by external code, just imported/exporter in the dashboard itself: http://gethue.com/export-and-import-your-search-dashboards/

      To read saved searches, just click on the white icon bookmark in the dashboard.

      • Barış 1 month ago

        thanks, so where are the query definations keeps on hue server?

        • Author
          Hue Team 1 month ago

          Query definition are saved in the Hue databases in the documen2 table.

          • Barış 4 weeks ago

            thanks.

  25. Barış 4 weeks ago

    Hi,

    I can just add one html layout for the resultset in hue search dashboard. Is there any way to add one more html layout or html widget in same dashboard. Dou you have future plan to add new widget for html layout? For example I want to add html widget for adding new graph etc. not for just resultset.

    Thanks.

    • Author
      Hue Team 3 weeks ago

      Current plan is to add grid/graph to any widget (in Hue 4), then add a new custom visualization option that lets you build your own widget.

Leave a reply

Your email address will not be published. Required fields are marked *

*