Community Articles

Find and share helpful community-sourced technical articles.
Announcements
Celebrating as our community reaches 100,000 members! Thank you!
avatar
Master Mentor

HDP deployment in Azure and configuring wasb in ambari

Lessons learned: All VMs needs to be under virtual private network and hostnames needs to be changed.

1) Create virtual network for your cluster

Note: screen shot has name hdptest01 but it's test

2) Create VMs and choose network created in the above step

This is very important step and we need to pay attention to the hostname.

Added endpoint for 8080.

There is a step to add disks to vm. You can follow this blog

At this point all the hosts are created and we will make changes to hostnames.

for example: the following needs to be modified. The command hostname should show internal hostname or same output as hostname -f so modify /etc/sysconfig/network and reboot all the nodes or change hostname using hostname command.

[root@hdpmaster01 ~]# cat /etc/sysconfig/network

HOSTNAME=hdpmaster01.hdpmaster01.j3.internal.cloudapp.net

NETWORKING=yes

#hostname

hdpmaster01.hdpmaster01.j3.internal.cloudapp.net

Follow HDP Docs to install cluster using Ambari

Use hostname entries for install , No public DNS

Once cluster is installed then add WASB config using Ambari

HDFS --> Configs --> under Custom hdfs-site

add property

fs.azure.account.key.hdptest01.blob.core.windows.net

WASB: Get secret key from Azure portal.

Restart the services and then test if you can use the wasb

[root@hdpmaster01 ~]# cat > test.txt

abc

[root@hdpmaster01 ~]# hdfs dfs -put test.txt wasb://hdpmastercontainer@hdptest01.blob.core.windows.net/

15/03/11 23:49:21 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties

15/03/11 23:49:21 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 60 second(s).

15/03/11 23:49:21 INFO impl.MetricsSystemImpl: azure-file-system metrics system started

15/03/11 23:49:23 INFO impl.MetricsSystemImpl: Stopping azure-file-system metrics system...

15/03/11 23:49:23 INFO impl.MetricsSystemImpl: azure-file-system metrics system stopped.

15/03/11 23:49:23 INFO impl.MetricsSystemImpl: azure-file-system metrics system shutdown complete.

[root@hdpmaster01 ~]# hdfs dfs -ls -R wasb://hdpmastercontainer@hdptest01.blob.core.windows.net/

15/03/11 23:49:35 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties

15/03/11 23:49:35 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 60 second(s).

15/03/11 23:49:35 INFO impl.MetricsSystemImpl: azure-file-system metrics system started

-rw-r--r-- 1 root supergroup 4 2015-03-11 23:49 wasb://hdpmastercontainer@hdptest01.blob.core.windows.net/test.txt

15/03/11 23:49:35 INFO impl.MetricsSystemImpl: Stopping azure-file-system metrics system...

15/03/11 23:49:35 INFO impl.MetricsSystemImpl: azure-file-system metrics system stopped.

15/03/11 23:49:35 INFO impl.MetricsSystemImpl: azure-file-system metrics system shutdown complete.

Useful links:

HDP docs

WASB Configs - Helpful link

Attaching disk

1,565 Views
Comments
avatar

@Neeraj what would be the steps to use WASB in a cluster deployed using cloudbreak? Would that work if I just add key to hdfs-site?

avatar
Master Mentor

@Guilherme Braccialli

Yes

example: This is based on google compute

avatar

@Neeraj I needed to add credential to hive-site to wasb to work inside hive. Did it work for you only with hdfs-site?

Version history
Last update:
‎09-16-2022 01:32 AM
Updated by:
Contributors