Running an Oozie workflow without having installed the sharelib

Running an Oozie workflow without having installed the sharelib

If after installing your cluster and submitting some Oozie jobs you are seeing this type of error:

2015-03-11 09:11:19,821  WARN ActionStartXCommand:544 - SERVER[local] USER[hue] GROUP[-] TOKEN[] APP[pig-app-hue-script] JOB[0000000-150311091052117-oozie-hue-W] ACTION[[email protected]] Error starting action [pig]. ErrorType [FAILED], ErrorCode [It should never happen], Message [File /user/oozie/share/lib does not exist]
org.apache.oozie.action.ActionExecutorException: File /user/oozie/share/lib does not exist
	at org.apache.oozie.action.hadoop.JavaActionExecutor.addShareLib(JavaActionExecutor.java:601)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.addActionShareLib(JavaActionExecutor.java:725)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.addAllShareLibs(JavaActionExecutor.java:707)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.setLibFilesArchives(JavaActionExecutor.java:700)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:895)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1145)
	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:228)
	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
	at org.apache.oozie.command.XCommand.call(XCommand.java:281)
	at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:323)
	at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:252)
	at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

or

  Error: java.io.IOException: Split class org.apache.oozie.action.hadoop.OozieLauncherInputFormat$EmptySplit not found
  at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:363)
  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:423)
  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
  at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:396)
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
  Caused by: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.OozieLauncherInputFormat$EmptySplit not found
  at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1953)
  at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:361)
  ... 7 more

This is because the Oozie Share Lib is not installed. Here is one command to install the YARN one:

sudo -u oozie /usr/lib/oozie/bin/oozie-setup.sh sharelib create -fs hdfs://localhost:8020 -locallib /usr/lib/oozie/oozie-sharelib-yarn.tar.gz

  setting JAVA_LIBRARY_PATH="$JAVA_LIBRARY_PATH:/usr/lib/hadoop/lib/native"
  setting OOZIE_DATA=/var/lib/oozie
  setting OOZIE_CATALINA_HOME=/usr/lib/bigtop-tomcat
  setting CATALINA_TMPDIR=/var/lib/oozie
  setting CATALINA_PID=/var/run/oozie/oozie.pid
  setting CATALINA_BASE=/var/lib/oozie/tomcat-deployment
  setting OOZIE_HTTPS_PORT=11443
  setting OOZIE_HTTPS_KEYSTORE_PASS=password
  setting CATALINA_OPTS="$CATALINA_OPTS -Doozie.https.port=${OOZIE_HTTPS_PORT}"
  setting CATALINA_OPTS="$CATALINA_OPTS -Doozie.https.keystore.pass=${OOZIE_HTTPS_KEYSTORE_PASS}"
  setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
  setting OOZIE_CONFIG=/etc/oozie/conf
  setting OOZIE_LOG=/var/log/oozie
  setting JAVA_LIBRARY_PATH="$JAVA_LIBRARY_PATH:/usr/lib/hadoop/lib/native"
  setting OOZIE_DATA=/var/lib/oozie
  setting OOZIE_CATALINA_HOME=/usr/lib/bigtop-tomcat
  setting CATALINA_TMPDIR=/var/lib/oozie
  setting CATALINA_PID=/var/run/oozie/oozie.pid
  setting CATALINA_BASE=/var/lib/oozie/tomcat-deployment
  setting OOZIE_HTTPS_PORT=11443
  setting OOZIE_HTTPS_KEYSTORE_PASS=password
  setting CATALINA_OPTS="$CATALINA_OPTS -Doozie.https.port=${OOZIE_HTTPS_PORT}"
  setting CATALINA_OPTS="$CATALINA_OPTS -Doozie.https.keystore.pass=${OOZIE_HTTPS_KEYSTORE_PASS}"
  setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
  setting OOZIE_CONFIG=/etc/oozie/conf
  setting OOZIE_LOG=/var/log/oozie
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/oozie/libserver/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/oozie/libserver/slf4j-simple-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
the destination path for sharelib is: /user/oozie/share/lib/lib_20141003111250

On latest version of Oozie, just point to a folder instead:

sudo -u oozie /usr/lib/oozie/bin/oozie-setup.sh sharelib create -fs hdfs://localhost:8020 -locallib /usr/lib/oozie/oozie-sharelib-yarn

And how to check it:

sudo -u oozie oozie  admin -shareliblist -oozie http://localhost:11000/oozie

[Available ShareLib]
oozie
hive
distcp
hcatalog
sqoop
mapreduce-streaming
hive2
pig

 

Note
If you have upgraded your cluster, use ‘upgrade’ instead of ‘create’:

sudo -u oozie /usr/lib/oozie/bin/oozie-setup.sh sharelib upgrade -fs hdfs://localhost:8020 -locallib /usr/lib/oozie/oozie-sharelib-yarn.tar.gz

Note

If you are seeing:

sharelib.system.libpath (unavailable)

You need something like that in your oozie-site.xml

<property>
<name>oozie.service.HadoopAccessorService.hadoop.configurations</name>
<value>*=/etc/hadoop/conf</value>
</property>

And now restart Oozie:

sudo service oozie restart

That’s it, you are now ready to submit workflows!

 

As usual feel free to comment and send feedback on the hue-user list or @gethue!

 

0 Comments

Leave a reply

Your email address will not be published. Required fields are marked *

*