Drag & Drop saved Hive queries into your workflows

Drag & Drop saved Hive queries into your workflows

The Oozie Workflow editor is getting a new generic action that let’s you drag & drop any of your Hive query of the SQL Editor. Here it is in action:

 

  • Always in sync with the saved query
  • Autocomplete Hive parameter
  • Behave like any other actions, can fork, see the logs…
  • Generates for you the query files in the HDFS workspace

 

Directly adding Hive queries from the SQL Editor

Directly adding Hive queries from the SQL Editor

 

We hope that you like the ease of use of this new drag & drop. Expect more type of documents like Pig, Spark, PySpark in the next Hue version!

And feel free to send feedback on the hue-user list or @gethue!

7 Comments

  1. bibhas 8 months ago

    Awesome…..it looks Good…Perfect.

  2. Savitri 5 months ago

    WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher – Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately on failure

    Unable to perform shell action from oozie
    shell script contains pig and hive script.

    Please let me know if there is any solution for this problem.

    • Author
      Hue Team 5 months ago

      Could you just at the logs of the MR jobs in the Job Browsers? They will have more details.

      • Savitri 5 months ago

        log4j:WARN Please initialize the log4j system properly.
        log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
        2016-08-10 05:09:32,555 [main] INFO org.apache.pig.Main – Apache Pig version 0.12.0-cdh5.8.0 (rUnversioned directory) compiled Jul 12 2016, 16:01:20
        2016-08-10 05:09:32,555 [main] INFO org.apache.pig.Main – Logging error messages to: /yarn/nm/usercache/root/appcache/application_1470662351891_0202/container_1470662351891_0202_01_000002/pig_1470823772545.log
        2016-08-10 05:09:33,072 [main] INFO org.apache.pig.impl.util.Utils – Default bootup file /var/lib/hadoop-yarn/.pigbootup not found
        2016-08-10 05:09:33,154 [main] INFO org.apache.hadoop.conf.Configuration.deprecation – mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
        2016-08-10 05:09:33,154 [main] INFO org.apache.hadoop.conf.Configuration.deprecation – fs.default.name is deprecated. Instead, use fs.defaultFS
        2016-08-10 05:09:33,154 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine – Connecting to hadoop file system at: hdfs://namednode.apporchid.com:8020
        2016-08-10 05:09:33,506 [main] INFO org.apache.hadoop.conf.Configuration.deprecation – fs.default.name is deprecated. Instead, use fs.defaultFS
        2016-08-10 05:09:33,533 [main] INFO org.apache.hadoop.conf.Configuration.deprecation – fs.default.name is deprecated. Instead, use fs.defaultFS
        2016-08-10 05:09:33,554 [main] INFO org.apache.hadoop.conf.Configuration.deprecation – fs.default.name is deprecated. Instead, use fs.defaultFS
        2016-08-10 05:09:33,583 [main] INFO org.apache.hadoop.conf.Configuration.deprecation – fs.default.name is deprecated. Instead, use fs.defaultFS
        2016-08-10 05:09:33,612 [main] INFO org.apache.hadoop.conf.Configuration.deprecation – fs.default.name is deprecated. Instead, use fs.defaultFS
        2016-08-10 05:09:33,640 [main] INFO org.apache.hadoop.conf.Configuration.deprecation – fs.default.name is deprecated. Instead, use fs.defaultFS
        2016-08-10 05:09:33,664 [main] INFO org.apache.hadoop.conf.Configuration.deprecation – fs.default.name is deprecated. Instead, use fs.defaultFS
        2016-08-10 05:09:33,679 [main] INFO org.apache.hadoop.conf.Configuration.deprecation – fs.default.name is deprecated. Instead, use fs.defaultFS
        2016-08-10 05:09:33,711 [main] INFO org.apache.hadoop.conf.Configuration.deprecation – fs.default.name is deprecated. Instead, use fs.defaultFS
        2016-08-10 05:09:34,116 [main] INFO org.apache.hadoop.conf.Configuration.deprecation – fs.default.name is deprecated. Instead, use fs.defaultFS
        2016-08-10 05:09:34,187 [main] INFO org.apache.pig.tools.pigstats.ScriptState – Pig features used in the script: UNKNOWN
        2016-08-10 05:09:34,211 [main] INFO org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer – {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, DuplicateForEachColumnRewrite, GroupByConstParallelSetter, ImplicitSplitInserter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, MergeForEach, NewPartitionFilterOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter], RULES_DISABLED=[FilterLogicExpressionSimplifier, PartitionFilterOptimizer]}
        2016-08-10 05:09:34,229 [main] INFO org.apache.hadoop.conf.Configuration.deprecation – mapred.textoutputformat.separator is deprecated. Instead, use mapreduce.output.textoutputformat.separator
        2016-08-10 05:09:34,279 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler – File concatenation threshold: 100 optimistic? false
        2016-08-10 05:09:34,294 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer – MR plan size before optimization: 1
        2016-08-10 05:09:34,294 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer – MR plan size after optimization: 1
        2016-08-10 05:09:34,341 [main] INFO org.apache.hadoop.yarn.client.RMProxy – Connecting to ResourceManager at namednode.apporchid.com/10.52.122.48:8032
        2016-08-10 05:09:34,461 [main] INFO org.apache.pig.tools.pigstats.ScriptState – Pig script settings are added to the job
        2016-08-10 05:09:34,516 [main] INFO org.apache.hadoop.conf.Configuration.deprecation – mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use mapreduce.reduce.markreset.buffer.percent
        2016-08-10 05:09:34,516 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler – mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
        2016-08-10 05:09:34,516 [main] INFO org.apache.hadoop.conf.Configuration.deprecation – mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
        2016-08-10 05:09:34,877 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler – creating jar file Job678262684864998704.jar
        2016-08-10 05:09:37,922 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler – jar file Job678262684864998704.jar created
        2016-08-10 05:09:37,922 [main] INFO org.apache.hadoop.conf.Configuration.deprecation – mapred.jar is deprecated. Instead, use mapreduce.job.jar
        2016-08-10 05:09:37,938 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler – Setting up single store job
        2016-08-10 05:09:37,943 [main] INFO org.apache.pig.data.SchemaTupleFrontend – Key [pig.schematuple] is false, will not generate code.
        2016-08-10 05:09:37,943 [main] INFO org.apache.pig.data.SchemaTupleFrontend – Starting process to move generated code to distributed cache
        2016-08-10 05:09:37,943 [main] INFO org.apache.pig.data.SchemaTupleFrontend – Setting key [pig.schematuple.classes] with classes to deserialize []
        2016-08-10 05:09:37,981 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher – 1 map-reduce job(s) waiting for submission.
        2016-08-10 05:09:37,982 [main] INFO org.apache.hadoop.conf.Configuration.deprecation – mapred.job.tracker.http.address is deprecated. Instead, use mapreduce.jobtracker.http.address
        2016-08-10 05:09:37,988 [JobControl] INFO org.apache.hadoop.yarn.client.RMProxy – Connecting to ResourceManager at namednode.apporchid.com/10.52.122.48:8032
        2016-08-10 05:09:37,998 [JobControl] INFO org.apache.hadoop.conf.Configuration.deprecation – fs.default.name is deprecated. Instead, use fs.defaultFS
        2016-08-10 05:09:38,201 [JobControl] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat – Total input paths to process : 1
        2016-08-10 05:09:38,201 [JobControl] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil – Total input paths to process : 1
        2016-08-10 05:09:38,211 [JobControl] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil – Total input paths (combined) to process : 1
        2016-08-10 05:09:38,285 [JobControl] INFO org.apache.hadoop.mapreduce.JobSubmitter – number of splits:1
        2016-08-10 05:09:38,410 [JobControl] INFO org.apache.hadoop.mapreduce.JobSubmitter – Submitting tokens for job: job_1470662351891_0203
        2016-08-10 05:09:38,410 [JobControl] INFO org.apache.hadoop.mapreduce.JobSubmitter – Kind: mapreduce.job, Service: job_1470662351891_0202, Ident: ([email protected]de1c3)
        2016-08-10 05:09:38,419 [JobControl] INFO org.apache.hadoop.mapreduce.JobSubmitter – Kind: RM_DELEGATION_TOKEN, Service: 10.52.122.48:8032, Ident: (owner=root, renewer=oozie mr token, realUser=oozie, issueDate=1470823762522, maxDate=1471428562522, sequenceNumber=262, masterKeyId=3)
        2016-08-10 05:09:38,560 [JobControl] INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl – Submitted application application_1470662351891_0203
        2016-08-10 05:09:38,582 [JobControl] INFO org.apache.hadoop.mapreduce.Job – The url to track the job: http://namednode.apporchid.com:8088/proxy/application_1470662351891_0203/
        2016-08-10 05:09:38,582 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher – HadoopJobId: job_1470662351891_0203
        2016-08-10 05:09:38,582 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher – Processing aliases A,B
        2016-08-10 05:09:38,582 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher – detailed locations: M: A[5,4],A[-1,-1],B[6,4] C: R:
        2016-08-10 05:09:38,620 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher – 0% complete
        2016-08-10 05:09:44,009 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher – Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately on failure.
        2016-08-10 05:09:44,009 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher – job job_1470662351891_0203 has failed! Stop running all dependent jobs
        2016-08-10 05:09:44,009 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher – 100% complete
        2016-08-10 05:09:44,050 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil – 1 map reduce job(s) failed!
        2016-08-10 05:09:44,051 [main] INFO org.apache.pig.tools.pigstats.SimplePigStats – Script Statistics:

        HadoopVersion PigVersion UserId StartedAt FinishedAt Features
        2.6.0-cdh5.8.0 0.12.0-cdh5.8.0 yarn 2016-08-10 05:09:34 2016-08-10 05:09:44 UNKNOWN

        Failed!
        This is complete log details while running pig script in hue 3.10.

        unable to perform even the example script of pig.

        Please help

        • Author
          Hue Team 5 months ago

          You will have more info if you look at the failed tasks of job_1470662351891_0203 in Job Browser

  3. hxx 2 months ago

    The hql file in the workflow using hiveserver2 use comment ‘–‘ prefix occur error,cannot recoginze the comments prefix,what is the problem?thank you!

Leave a reply

Your email address will not be published. Required fields are marked *

*