Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

The steps below show how to configure authentication between your HDP cluster and WASB using the wasb connector.

Note for Cloudbreak users: When you create a cluster with Cloudbreak, you can configure authentication with WASB on the "Add File System" page of the create cluster wizard. If you do this, you do not need to perform the steps below. If you have already created a cluster with Cloudbreak but did not perform WASB configuration on the"Add File System" page of the create cluster wizard, follow the steps below:

Prerequisites

1. The Azure prerequisite to configuring authentication is having a storage subscription and creating a storage account. For example:

15965-wasb1.png

2. On the HDP side, you must use a version that supports the WASB integration. I am using HDP 2.6.1, which supports connecting to WASB using the WASB connector.

Steps

In order to access data stored in your Azure blob storage account, you must configure your storage account access key in core-site.xml. The configuration property that you must use is fs.azure.account.key.<account name>.blob.core.windows.net and the value is the access key.

For example the following property should be used for a storage account called "testaccount":

<property>
  <name>fs.azure.account.key.testaccount.blob.core.windows.net</name>
  <value>TESTACCOUNT-ACCESS-KEY</value>
</property>

You can obtain your access key from the Access keys in your storage account settings:

15964-wasb2.png

It's recommended that you protect your credentials with credential providers. For steps, see Protecting Azure Credentials with Credential Providers.

Working with Blob Storage

To make sure that the authentication works, try accessing data.

When accessing blob storage via the WASB connector, the URL structure is:

wasb://<container_name>@<storage_account_name>.blob.core.windows.net/dir/file

For example, to access a file called "testfile" located in a directory called "testdir", stored in the container called "testcontainer" on the account called "hortonworks", the URL is:

wasb://testcontainer@hortonworks.blob.core.windows.net/testdir/testfile

You can also use wasbs prefix to utilize SSL-encrypted HTTPS access:

wasbs://<container_name>@<storage_account_name>.blob.core.windows.net/dir/file

For example, the following Hadoop FileSystem shell commands demonstrate access to a storage account named myaccount and a container named mycontainer:

hadoop fs -ls wasb://mycontainer@myaccount.blob.core.windows.net/

hadoop fs -mkdir wasb://mycontainer@myaccount.blob.core.windows.net/testDir

hadoop fs -put testFile wasb://mycontainer@myaccount.blob.core.windows.net/testDir/testFile

hadoop fs -cat wasb://mycontainer@myaccount.blob.core.windows.net/testDir/testFile
test file content

Learn More

For more information about working with WASB using the WASB connector, refer to Getting Started with WASB in Hortonworks documentation.

3,159 Views
Comments
Guru

Pretty informative and useful. Thanks @Dominika Bialek for writing this. Keep it up !!

Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
2 of 2
Last update:
‎08-17-2019 12:41 PM
Updated by:
 
Contributors
Top Kudoed Authors