Oozie shared libs in MS Azure Blob Storage

repka_mar11 — Mon, 25 Jan 2016 23:28:25 GMT

When creating structure in HDFS (Azure Storage Blob) using standard HDFS commands we were able to created directory structure.

But using standard inicialization procedure using we weren't able to install oozie shared libs.

Any suggestions? Thanks in advance.

[oozie@XXXXX oozie-server]$ ./bin/oozie-setup.sh sharelib create -fs "wasb://blob-hdfs@XXXXX.blob.core.windows.net"
  setting OOZIE_CLIENT_OPTS="${OOZIE_CLIENT_OPTS} -Doozie.connection.retry.count=5"
  setting CATALINA_BASE=${CATALINA_BASE: -/usr/hdp/2.2.0.0-2041/oozie}
  setting CATALINA_OPTS="${CATALINA_OPTS} -Xmx2048m -XX:MaxPermSize=256m"
  setting CATALINA_TMPDIR=${CATALINA_TMPDIR:-/var/tmp/oozie}
  setting OOZIE_CATALINA_HOME=/usr/lib/bigtop-tomcat
  setting OOZIE_LOG=/var/log/oozie
  setting CATALINA_PID=/var/run/oozie/oozie.pid
  setting OOZIE_DATA=/var/db/oozie
  setting JAVA_LIBRARY_PATH=/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
/usr/hdp/current/oozie-server/conf/oozie-env.sh: line 29: CATALINA_BASE: -/usr/hdp/2.2.0.0-2041/oozie: syntax error: operand expected (error token is "/usr/hdp/2.2.0.0-2041/oozie")
  setting OOZIE_CLIENT_OPTS="${OOZIE_CLIENT_OPTS} -Doozie.connection.retry.count=5"
  setting CATALINA_BASE=${CATALINA_BASE: -/usr/hdp/2.2.0.0-2041/oozie}
  setting CATALINA_OPTS="${CATALINA_OPTS} -Xmx2048m -XX:MaxPermSize=256m"
  setting CATALINA_TMPDIR=${CATALINA_TMPDIR:-/var/tmp/oozie}
  setting OOZIE_CATALINA_HOME=/usr/lib/bigtop-tomcat
  setting OOZIE_LOG=/var/log/oozie
  setting CATALINA_PID=/var/run/oozie/oozie.pid
  setting OOZIE_DATA=/var/db/oozie
  setting JAVA_LIBRARY_PATH=/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.2.0.0-2041/oozie/libserver/slf4j-simple-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.2.0.0-2041/oozie/libserver/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.SimpleLoggerFactory]
the destination path for sharelib is: /user/oozie/share/lib/lib_20160121102113

Error: E0904: Scheme [wasb] not supported in uri [wasb://blob-hdfs@XXXXX.blob.core.windows.net]

Stack trace for the error was (for debug purposes):
--------------------------------------
org.apache.oozie.service.HadoopAccessorException: E0904: Scheme [wasb] not supported in uri [wasb://blob-hdfs@XXXXX.blob.core.windows.net]
        at org.apache.oozie.service.HadoopAccessorService.checkSupportedFilesystem(HadoopAccessorService.java:601)
        at org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:467)
        at org.apache.oozie.tools.OozieSharelibCLI.run(OozieSharelibCLI.java:158)
        at org.apache.oozie.tools.OozieSharelibCLI.main(OozieSharelibCLI.java:56)
--------------------------------------

Re: Oozie shared libs in MS Azure Blob Storage

jluniya — Tue, 26 Jan 2016 02:32:21 GMT

I dont have a cluster handy to validate this, but I think this should work.

If you have Namenode HA enabled try with -fs hdfs://{dfs.nameservices} (i.e. oozie-setup.sh sharelib create -fs hdfs://mycluster).

Re: Oozie shared libs in MS Azure Blob Storage

jluniya — Tue, 26 Jan 2016 04:45:37 GMT

Actually, I tried with wasb URI in a cluster and it worked. Can you check if wasb in the list of supported filesystems in oozie-site.xml?

<key>oozie.service.HadoopAccessorService.supported.filesystems</key>

<value>hdfs,hftp,webhdfs,asv,wasb,asvs,wasbs,swebhdfs,adl</value>

Re: Oozie shared libs in MS Azure Blob Storage

repka_mar11 — Tue, 26 Jan 2016 18:39:56 GMT

Thank you!

question Oozie shared libs in MS Azure Blob Storage in Archives of Support Questions (Read Only)

Oozie shared libs in MS Azure Blob Storage

Re: Oozie shared libs in MS Azure Blob Storage

Re: Oozie shared libs in MS Azure Blob Storage

Re: Oozie shared libs in MS Azure Blob Storage