question Re: Need instructions to setup WASB as storage for HDP on Azure IaaS in Archives of Support Questions (Read Only)

Need instructions to setup WASB as storage for HDP on Azure IaaS

vnair — Fri, 16 Sep 2022 09:44:22 GMT

Some customers opt to set up HDP on Azure IaaS instead of HDInsight. They may still need to persist data in the Azure Blob Store. If you have setup HDP on Azure IaaS with WASB for storage please share your setup instructions and related best practices so all can benefit.

Re: Need instructions to setup WASB as storage for HDP on Azure IaaS

nsabharwal — Sun, 18 Oct 2015 03:30:50 GMT

This blog talks about installing HDP in Azure (IaaS) + Configure WASB ( with screen shots)

Re: Need instructions to setup WASB as storage for HDP on Azure IaaS

cnauroth — Sun, 18 Oct 2015 03:33:04 GMT

The main thing that needs to be done to enable access to WASB is to configure the credentials. This involves editing core-site.xml, and the instructions are documented in Apache here:

http://hadoop.apache.org/docs/r2.7.1/hadoop-azure/index.html

After configuring the credentials, you'll be able to access WASB files by specifying URLs that use "wasb" as the scheme. HDInsight clusters also make WASB the default file system by setting configuration property fs.defaultFS to a wasb URL. The same Apache documentation shows usage examples for both cases.

Note that setting fs.defaultFS is only done if your intention is to replace HDFS fully as the default file system. To my knowledge, this has only ever been done in HDInsight.

Re: Need instructions to setup WASB as storage for HDP on Azure IaaS

ldaluz — Mon, 19 Oct 2015 20:40:56 GMT

Here are some details on setting up WASB as the default filesystem (fs.defaultFS) using HDP 2.3.x on Azure IaaS:

Architecture Considerations

Before using WASB as the defaultFS, it is important to understand the architectural impact this will have on the cluster.

What is WASB? + Pros

Windows Azure Storage Blob (WASB) is an extension built on top of the HDFS APIs that interfaces with data stored within an Azure Blob Storage account. One of the key advantages of using WASB is that it creates a layer of abstraction that enables separation of storage from compute. You can also add/remove/modify files in the Azure blob store without regard to the Hadoop cluster, as directory and file references are checked at runtime for each job. This allows you to better utilise the flexibility of a cloud deployment, as data can persist without the need for a cluster / compute nodes. You have the following deployment options when using WASB:

HDFS as defaultFS - Deploy a HDFS cluster and configure an external WASB filesystem to interface with a separate Blob Storage account. In this scenario your default Hadoop data would still be stored on HDFS, and you would have some data stored in a Blob Storage account that can be accessed when needed.
WASB as defaultFS - Deploy the cluster with WASB as the default filesystem to completely replace HDFS with WASB. In this scenario HDFS still technically exists but will be empty as all files / folders deployed with the cluster are stored in WASB (blob storage). Note, the steps below describe how to configure a cluster for this approach.

In many ways you can interface with the WASB filesystem as you would with HDFS, by specifying the WASB URLs:

wasb://<containername>@<accountname>.blob.core.windows.net/<path>

For example, the following hadoop FileSystem Shell command to WASB would behave similar to the same call made to HDFS:

hadoop fs -ls wasb://<containername>@<accountname>.blob.core.windows.net/

However, it is important to note that WASB and HDFS are separate filesystems and WASB currently has some limitations that should be considered before implementing.

Current Limitations - As at October 2015

The following limitations currently exist within the WASB filesystem:

WebHDFS is not compatible with WASB
- This limits the functionality within both Hue and Ambari Views (and any application that interfaces with the WebHDFS APIs)
Security permissions are persisted in WASB, but are not enforced.
- File owner and group are persisted for all directories and files, but the permission model is not enforced at the filesystem level and all authorisation occurs at the level of the entire Azure Blob Storage account.

When deciding to use WASB as the defaultFS, it is recommended that you first review and assess your security and data access requirements and ensure you can meet them against the above limitations.

Installation + Configuration Steps

Important: There is a known bug in HDP 2.3.0 that prevents non HDFS filesystems (S3, WASB etc) being set as the default filesystem [HADOOP-11618, HADOOP-12304]. For the following steps to work, you will need to use either a patched version of HDP 2.3 or use HDP 2.3.2 or later.

To configure WASB as the default filesystem in-place of HDFS, the following Azure information is required:

Azure Storage Account URL
Container Name
Storage Access Key

This information is used along with the Hadoop configurations within core-site.xml to configure WASB.

For a fresh install, follow the standard documentation for installing HDP via Ambari found here until you get to Customize Services, where you will need to configure your WASB properties. Also note that when using WASB as the defaultFS, you do not need to mount any additional data drives to the servers (as you would for a HDFS cluster).

Customize Services

The following is a list of configurations that should be modified to configure WASB:

fs.defaultFS

wasb://<containername>@<accountname>.blob.core.windows.net

fs.AbstractFileSystem.wasb.impl

org.apache.hadoop.fs.azure.Wasb

fs.azure.account.key.<accountname>.blob.core.windows.net

<storage_access_key>

Even though WASB will be set as the fs.defaultFS, you still need to define DataNode directories for HDFS. As the intent here is to use WASB as the primary FS, you can set the HDFS datanode directories to the temporary /mnt/resource mount point that is provided with Azure compute servers if you only plan to use HDFS for temporary job files. DataNode Directories

/mnt/resource/Hadoop/hdfs/data

Outside of these core-site.xml configurations, hive has the following requirements when working with blob storage on Azure:

Point archive and jar files to 'wasb://' instead of 'hdfs://' templeton.hive.archive templeton.pig.archive templeton.sqoop.archive templeton.streaming.jar

wasb:///hdp/apps/${hdp.version}/hive/hive.tar.gz
wasb:///hdp/apps/${hdp.version}/pig/pig.tar.gz
wasb:///hdp/apps/${hdp.version}/sqoop/sqoop.tar.gz
wasb:///hdp/apps/${hdp.version}/mapreduce/hadoop-streaming.jar

Skip azure metrics in custom webhcat-site When WASB is used, metrics collection is enabled by default. For webhcat server, this causes an unnecessary overhead that we can disable. fs.azure.skip.metrics

true

Further Reading

Re: Need instructions to setup WASB as storage for HDP on Azure IaaS

aervits — Sat, 24 Oct 2015 00:33:53 GMT

One thing to keep in mind is that SmartSense will collect core-site.xml so if you have WASB access keys, they will be collected, so to opt-out you need to disable the properties you don't want collected.

Re: Need instructions to setup WASB as storage for HDP on Azure IaaS

aervits — Tue, 27 Oct 2015 01:28:22 GMT

just went through it yesterday, steps here.

Re: Need instructions to setup WASB as storage for HDP on Azure IaaS

dorio — Thu, 10 Dec 2015 06:01:13 GMT

If you also want to look at the performance remember to put Storage Account in the same region than IaaS deployment. You need to remenber Azure limits for the IaaS: do not put more than 2 DataNodes on the same Storage Account (for max IOPS limits).

Look also at Azure Data Lake Store if you plan to use a storage for both IaaS and PaaS (HDInsight).

Re: Need instructions to setup WASB as storage for HDP on Azure IaaS

sdutta — Thu, 04 Feb 2016 01:38:50 GMT

How about use of DASH? Cloudbreak suggests DASH with WASB.

Re: Need instructions to setup WASB as storage for HDP on Azure IaaS

pawan_mathur — Wed, 11 May 2016 16:33:31 GMT

Hi, does this support Transparent Data Encryption? How does the Disaster Recovery/Copy to another region Azure Blob storage work?

Re: Need instructions to setup WASB as storage for HDP on Azure IaaS

subhankar_dey — Tue, 30 Jan 2018 20:56:46 GMT

Hello,

I have set up the my cluster with the above configurations and everything is working fine except spark. I am receiving a lot of errors while starting spark-shell, after changing the storage from HDFS to WASB. The errors look like the following:

java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState': at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$reflect(SparkSession.scala:983) at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:110) at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:109) at org.apache.spark.sql.SparkSession$Builder$anonfun$getOrCreate$5.apply(SparkSession.scala:878) at org.apache.spark.sql.SparkSession$Builder$anonfun$getOrCreate$5.apply(SparkSession.scala:878) at scala.collection.mutable.HashMap$anonfun$foreach$1.apply(HashMap.scala:99) at scala.collection.mutable.HashMap$anonfun$foreach$1.apply(HashMap.scala:99) at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40) at scala.collection.mutable.HashMap.foreach(HashMap.scala:99) at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:878) at org.apache.spark.repl.Main$.createSparkSession(Main.scala:96) ... 47 elided Caused by: java.lang.reflect.InvocationTargetException: java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveExternalCatalog': at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$reflect(SparkSession.scala:980) ... 58 more Caused by: java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveExternalCatalog': at org.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$reflect(SharedState.scala:176) at org.apache.spark.sql.internal.SharedState.<init>(SharedState.scala:86) at org.apache.spark.sql.SparkSession$anonfun$sharedState$1.apply(SparkSession.scala:101) at org.apache.spark.sql.SparkSession$anonfun$sharedState$1.apply(SparkSession.scala:101) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.SparkSession.sharedState$lzycompute(SparkSession.scala:101) at org.apache.spark.sql.SparkSession.sharedState(SparkSession.scala:100) at org.apache.spark.sql.internal.SessionState.<init>(SessionState.scala:157) at org.apache.spark.sql.hive.HiveSessionState.<init>(HiveSessionState.scala:32) ... 63 more Caused by: java.lang.reflect.InvocationTargetException: java.lang.reflect.InvocationTargetException: java.lang.RuntimeException: org.apache.hadoop.fs.azure.AzureException: java.util.NoSuchElementException: An error occurred while enumerating the result, check the original exception for details. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$reflect(SharedState.scala:173) ... 71 more Caused by: java.lang.reflect.InvocationTargetException: java.lang.RuntimeException: org.apache.hadoop.fs.azure.AzureException: java.util.NoSuchElementException: An error occurred while enumerating the result, check the original exception for details. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:264) at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:358) at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:262) at org.apache.spark.sql.hive.HiveExternalCatalog.<init>(HiveExternalCatalog.scala:65) ... 76 more Caused by: java.lang.RuntimeException: org.apache.hadoop.fs.azure.AzureException: java.util.NoSuchElementException: An error occurred while enumerating the result, check the original exception for details. at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522) at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:188) ... 84 more Caused by: org.apache.hadoop.fs.azure.AzureException: java.util.NoSuchElementException: An error occurred while enumerating the result, check the original exception for details. at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:2027) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:2081) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1447) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.conditionalRedoFolderRename(NativeAzureFileSystem.java:2137) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:2104) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1447) at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:596) at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:554) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508) ... 85 more Caused by: java.util.NoSuchElementException: An error occurred while enumerating the result, check the original exception for details. at com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:113) at org.apache.hadoop.fs.azure.StorageInterfaceImpl$WrappingIterator.hasNext(StorageInterfaceImpl.java:130) at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:2006) ... 93 more Caused by: com.microsoft.azure.storage.StorageException: The server encountered an unknown failure: OK at com.microsoft.azure.storage.StorageException.translateException(StorageException.java:101) at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:199) at com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:109) ... 95 more Caused by: java.lang.ClassCastException: org.apache.xerces.parsers.XIncludeAwareParserConfiguration cannot be cast to org.apache.xerces.xni.parser.XMLParserConfiguration at org.apache.xerces.parsers.SAXParser.<init>(Unknown Source) at org.apache.xerces.parsers.SAXParser.<init>(Unknown Source) at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.<init>(Unknown Source) at org.apache.xerces.jaxp.SAXParserImpl.<init>(Unknown Source) at org.apache.xerces.jaxp.SAXParserFactoryImpl.newSAXParser(Unknown Source) at com.microsoft.azure.storage.core.Utility.getSAXParser(Utility.java:668) at com.microsoft.azure.storage.blob.BlobListHandler.getBlobList(BlobListHandler.java:72) at com.microsoft.azure.storage.blob.CloudBlobContainer$6.postProcessResponse(CloudBlobContainer.java:1284) at com.microsoft.azure.storage.blob.CloudBlobContainer$6.postProcessResponse(CloudBlobContainer.java:1248) at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:146) ... 96 more <console>:14: error: not found: value spark import spark.implicits._ ^ <console>:14: error: not found: value spark import spark.sql

Please help in resolving the issue and let me know if it is possible at all, for spark to function in the WASB storage environment.

Regards,

Subhankar

Re: Need instructions to setup WASB as storage for HDP on Azure IaaS

subhankar_dey — Tue, 30 Jan 2018 21:03:05 GMT

Hello,

Could you please assist me in answering the following query of mine:

https://community.hortonworks.com/questions/167906/we-are-unable-to-access-sparkspark2-when-we-change.html

Regards,

Subhankar