Oozie shared libs in MS Azure Blob Storage

[#1]

When creating structure in HDFS (Azure Storage Blob) using standard HDFS commands we were able to created directory structure.

But using standard inicialization procedure using we weren't able to install oozie shared libs.

Any suggestions? Thanks in advance.

[oozie@XXXXX oozie-server]$ ./bin/oozie-setup.sh sharelib create -fs "wasb://blob-hdfs@XXXXX.blob.core.windows.net"
  setting OOZIE_CLIENT_OPTS="${OOZIE_CLIENT_OPTS} -Doozie.connection.retry.count=5"
  setting CATALINA_BASE=${CATALINA_BASE: -/usr/hdp/2.2.0.0-2041/oozie}
  setting CATALINA_OPTS="${CATALINA_OPTS} -Xmx2048m -XX:MaxPermSize=256m"
  setting CATALINA_TMPDIR=${CATALINA_TMPDIR:-/var/tmp/oozie}
  setting OOZIE_CATALINA_HOME=/usr/lib/bigtop-tomcat
  setting OOZIE_LOG=/var/log/oozie
  setting CATALINA_PID=/var/run/oozie/oozie.pid
  setting OOZIE_DATA=/var/db/oozie
  setting JAVA_LIBRARY_PATH=/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
/usr/hdp/current/oozie-server/conf/oozie-env.sh: line 29: CATALINA_BASE: -/usr/hdp/2.2.0.0-2041/oozie: syntax error: operand expected (error token is "/usr/hdp/2.2.0.0-2041/oozie")
  setting OOZIE_CLIENT_OPTS="${OOZIE_CLIENT_OPTS} -Doozie.connection.retry.count=5"
  setting CATALINA_BASE=${CATALINA_BASE: -/usr/hdp/2.2.0.0-2041/oozie}
  setting CATALINA_OPTS="${CATALINA_OPTS} -Xmx2048m -XX:MaxPermSize=256m"
  setting CATALINA_TMPDIR=${CATALINA_TMPDIR:-/var/tmp/oozie}
  setting OOZIE_CATALINA_HOME=/usr/lib/bigtop-tomcat
  setting OOZIE_LOG=/var/log/oozie
  setting CATALINA_PID=/var/run/oozie/oozie.pid
  setting OOZIE_DATA=/var/db/oozie
  setting JAVA_LIBRARY_PATH=/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.2.0.0-2041/oozie/libserver/slf4j-simple-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.2.0.0-2041/oozie/libserver/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.SimpleLoggerFactory]
the destination path for sharelib is: /user/oozie/share/lib/lib_20160121102113

Error: E0904: Scheme [wasb] not supported in uri [wasb://blob-hdfs@XXXXX.blob.core.windows.net]

Stack trace for the error was (for debug purposes):
--------------------------------------
org.apache.oozie.service.HadoopAccessorException: E0904: Scheme [wasb] not supported in uri [wasb://blob-hdfs@XXXXX.blob.core.windows.net]
        at org.apache.oozie.service.HadoopAccessorService.checkSupportedFilesystem(HadoopAccessorService.java:601)
        at org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:467)
        at org.apache.oozie.tools.OozieSharelibCLI.run(OozieSharelibCLI.java:158)
        at org.apache.oozie.tools.OozieSharelibCLI.main(OozieSharelibCLI.java:56)
--------------------------------------

[#2 · Accepted solution]

I dont have a cluster handy to validate this, but I think this should work.

If you have Namenode HA enabled try with -fs hdfs://{dfs.nameservices} (i.e. oozie-setup.sh sharelib create -fs hdfs://mycluster).

[#3]

Actually, I tried with wasb URI in a cluster and it worked. Can you check if wasb in the list of supported filesystems in oozie-site.xml?

<key>oozie.service.HadoopAccessorService.supported.filesystems</key>

<value>hdfs,hftp,webhdfs,asv,wasb,asvs,wasbs,swebhdfs,adl</value>

[#4]

Thank you!