Build a Real Time Analytic dashboard with Solr Search and Spark Streaming

Build a Real Time Analytic dashboard with Solr Search and Spark Streaming

Search is a great way to interactively explore your data. The Search App is continuously improving and now comes with a better support for real time!

In this video, we are collecting tweets with Spark Streaming and directly indexing them into Solr with the Spark Solr app. Note that we are using a slightly modified version that adds more tweet information.

 

You can see the tweets rolling in! Compared to the previous version:

  • the dashboard updates its widgets only when the data changes without any page jumping
  • the dashboard can refresh itself automatically every N seconds
  • a main date filter lets you quickly select a rolling date range for all the dashboard

 

live-search

Tweets coming in

 

Instructions
Download a nightly Solr 5.x, uncompress it and start it:

bin/solr start -cloud
bin/solr create -c tweets

Then compile the Spark Solr app.

Enable the analytic widgets in hue.ini:

[search]
latest=true

Sum-up

They are other ways to index data in near real time but we took this approach as the scenario was working out of the box with just Spark Streaming and the Solr app. Next time, we will preview the new Analytics Features of Solr 5.2 and show how we can use Python Spark to index some data!

As usual feel free to comment on the hue-user list or @gethue!

23 Comments

  1. Blair Krotenko 2 years ago

    Hello Hue Team,

    I’m trying to work through this tutorial but I’m running into a few issues.

    When I click the drop-down to select a search index, I don’t see tweets_shard1_replica1. However, if I check the box for Show cores, I do see one called twitter_demo_shard1_replica1. But, that dataset only has 16 fields compared to the 42 in the demo, and the field ‘hashtags’ is not included. Also, the global time filter does not appear when I select this dataset. Do I need to install this dataset? I did install all application examples from step 2 of the Hue Quick Start Wizard.

    I’m using Hue 3.7.0 from the quickstart v. 5.4.0 VMware vm.

    Thanks,
    Blair

    • Hue Team 2 years ago

      The global time filter is in Hue 3.9 which is not released yet, but you should not need it.

      If you are seeing only 16 fields, this is probably because not all the dynamic fields were fetch when you opened the dashboard. Could you just reload the page?

      • Blair Krotenko 2 years ago

        I tried a few things to get the dynamic fields to appear, but was not successful. I tried page reload, create a new dashboard, re-install the application examples, and delete and re-import the VM from the download zip file. Also, the counter button does not exist in my version.

        Thanks,
        Blair

        • Hue Team 2 years ago

          Yes, counters are coming in Hue 3.9 which is not released but are in master.

          When you click on a result row, do you see some dynamic fields value there?

  2. Victor 8 months ago

    Hello there,

    We’re trying to create a dynamic dashboard using the Solr Search using a file to load some information and make hue read and create the graphics. But the panel that we are using is not refreshing even checking the checkbox of AutoRefreshing. The values from index still static…

    Is there something that we didn’t noticed?

    Thanks

    • Author
      Hue Team 8 months ago

      I just checked on http://demo.gethue.com/search/?collection=14 and it works if you check and uncheck the box after picking the time.

      • Victor 8 months ago

        Sorry. When I wrote that message I forgot to mention that we are trying to add some values into an index created. In that demo has something like that?

        • Author
          Hue Team 8 months ago

          Yes, don’t you see in the video the new tweets coming in?

          • Victor 8 months ago

            Yes but when we run our application in spark updating a file that was being used by the index, the workbook didn’t updated the charts and the tables. Could you release the spark code to us? Thanks

          • Author
            Hue Team 8 months ago

            The spark code is listed already: https://github.com/romainr/spark-solr

            What you might miss is what we do at 3:50 in the video to trigger the automatic refresh of the dashboard.

  3. Khaled Idriss 5 months ago

    Hello, How to use Metrics in HUE Dashboards, I want to create bar chart with Y-Axis using Sum not Count as usual

  4. Poshita Singh 1 month ago

    Will Ubuntu 16.04 support Hue?

    • Author
      Hue Team 1 month ago

      I just tried and a freshly installed 16.04 supports Hue. The steps I did:

      sudo add-apt-repository ppa:webupd8team/java
      sudo apt-get update
      sudo apt-get install oracle-java8-installer

      sudo apt-get install ant gcc g++ libffi-dev libkrb5-dev libmysqlclient-dev libsasl2-dev libsasl2-modules-gssapi-mit libsqlite3-dev libssl-dev libxml2-dev libxslt-dev make maven libldap2-dev python-dev python-setuptools libgmp3-dev libz-dev

      git clone https://github.com/cloudera/hue.git
      cd hue
      make apps
      build/env/bin/hue runserver

Leave a reply

Your email address will not be published. Required fields are marked *

*