The steps below show how to configure authentication between your HDP cluster and WASB using the wasb connector.
Note for Cloudbreak users: When you create a cluster with Cloudbreak, you can configure authentication with WASB on the "Add File System" page of the create cluster wizard. If you do this, you do not need to perform the steps below. If you have already created a cluster with Cloudbreak but did not perform WASB configuration on the"Add File System" page of the create cluster wizard, follow the steps below:
Prerequisites
1. The Azure prerequisite to configuring authentication is having a storage subscription and creating a storage account. For example:
2. On the HDP side, you must use a version that supports the WASB integration. I am using HDP 2.6.1, which supports connecting to WASB using the WASB connector.
Steps
In order to access data stored in your Azure blob storage account, you must configure your storage account access key in core-site.xml. The configuration property that you must use is fs.azure.account.key.<account name>.blob.core.windows.net and the value is the access key.
For example the following property should be used for a storage account called "testaccount":
<property> <name>fs.azure.account.key.testaccount.blob.core.windows.net</name> <value>TESTACCOUNT-ACCESS-KEY</value> </property>
You can obtain your access key from the Access keys in your storage account settings:
It's recommended that you protect your credentials with credential providers. For steps, see Protecting Azure Credentials with Credential Providers.
Working with Blob Storage
To make sure that the authentication works, try accessing data.
When accessing blob storage via the WASB connector, the URL structure is:
wasb://<container_name>@<storage_account_name>.blob.core.windows.net/dir/file
For example, to access a file called "testfile" located in a directory called "testdir", stored in the container called "testcontainer" on the account called "hortonworks", the URL is:
wasb://testcontainer@hortonworks.blob.core.windows.net/testdir/testfile
You can also use wasbs prefix to utilize SSL-encrypted HTTPS access:
wasbs://<container_name>@<storage_account_name>.blob.core.windows.net/dir/file
For example, the following Hadoop FileSystem shell commands demonstrate access to a storage account named myaccount and a container named mycontainer:
hadoop fs -ls wasb://mycontainer@myaccount.blob.core.windows.net/ hadoop fs -mkdir wasb://mycontainer@myaccount.blob.core.windows.net/testDir hadoop fs -put testFile wasb://mycontainer@myaccount.blob.core.windows.net/testDir/testFile hadoop fs -cat wasb://mycontainer@myaccount.blob.core.windows.net/testDir/testFile test file content
Learn More
For more information about working with WASB using the WASB connector, refer to Getting Started with WASB in Hortonworks documentation.
