Community Articles

Find and share helpful community-sourced technical articles.
Announcements
Celebrating as our community reaches 100,000 members! Thank you!
avatar

The steps below show how to configure authentication between your HDP cluster and WASB using the wasb connector.

Note for Cloudbreak users: When you create a cluster with Cloudbreak, you can configure authentication with WASB on the "Add File System" page of the create cluster wizard. If you do this, you do not need to perform the steps below. If you have already created a cluster with Cloudbreak but did not perform WASB configuration on the"Add File System" page of the create cluster wizard, follow the steps below:

Prerequisites

1. The Azure prerequisite to configuring authentication is having a storage subscription and creating a storage account. For example:

15965-wasb1.png

2. On the HDP side, you must use a version that supports the WASB integration. I am using HDP 2.6.1, which supports connecting to WASB using the WASB connector.

Steps

In order to access data stored in your Azure blob storage account, you must configure your storage account access key in core-site.xml. The configuration property that you must use is fs.azure.account.key.<account name>.blob.core.windows.net and the value is the access key.

For example the following property should be used for a storage account called "testaccount":

<property>
  <name>fs.azure.account.key.testaccount.blob.core.windows.net</name>
  <value>TESTACCOUNT-ACCESS-KEY</value>
</property>

You can obtain your access key from the Access keys in your storage account settings:

15964-wasb2.png

It's recommended that you protect your credentials with credential providers. For steps, see Protecting Azure Credentials with Credential Providers.

Working with Blob Storage

To make sure that the authentication works, try accessing data.

When accessing blob storage via the WASB connector, the URL structure is:

wasb://<container_name>@<storage_account_name>.blob.core.windows.net/dir/file

For example, to access a file called "testfile" located in a directory called "testdir", stored in the container called "testcontainer" on the account called "hortonworks", the URL is:

wasb://testcontainer@hortonworks.blob.core.windows.net/testdir/testfile

You can also use wasbs prefix to utilize SSL-encrypted HTTPS access:

wasbs://<container_name>@<storage_account_name>.blob.core.windows.net/dir/file

For example, the following Hadoop FileSystem shell commands demonstrate access to a storage account named myaccount and a container named mycontainer:

hadoop fs -ls wasb://mycontainer@myaccount.blob.core.windows.net/

hadoop fs -mkdir wasb://mycontainer@myaccount.blob.core.windows.net/testDir

hadoop fs -put testFile wasb://mycontainer@myaccount.blob.core.windows.net/testDir/testFile

hadoop fs -cat wasb://mycontainer@myaccount.blob.core.windows.net/testDir/testFile
test file content

Learn More

For more information about working with WASB using the WASB connector, refer to Getting Started with WASB in Hortonworks documentation.

6,458 Views
Comments
avatar
Guru

Pretty informative and useful. Thanks @Dominika Bialek for writing this. Keep it up !!

Version history
Last update:
‎09-16-2022 01:40 AM
Updated by:
Contributors