Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HDP deployment in Azure and configuring wasb in ambari

Lessons learned: All VMs needs to be under virtual private network and hostnames needs to be changed.

1) Create virtual network for your cluster

Note: screen shot has name hdptest01 but it's test

2) Create VMs and choose network created in the above step

This is very important step and we need to pay attention to the hostname.

Added endpoint for 8080.

There is a step to add disks to vm. You can follow this blog

At this point all the hosts are created and we will make changes to hostnames.

for example: the following needs to be modified. The command hostname should show internal hostname or same output as hostname -f so modify /etc/sysconfig/network and reboot all the nodes or change hostname using hostname command.

[root@hdpmaster01 ~]# cat /etc/sysconfig/network

HOSTNAME=hdpmaster01.hdpmaster01.j3.internal.cloudapp.net

NETWORKING=yes

#hostname

hdpmaster01.hdpmaster01.j3.internal.cloudapp.net

Follow HDP Docs to install cluster using Ambari

Use hostname entries for install , No public DNS

Once cluster is installed then add WASB config using Ambari

HDFS --> Configs --> under Custom hdfs-site

add property

fs.azure.account.key.hdptest01.blob.core.windows.net

WASB: Get secret key from Azure portal.

Restart the services and then test if you can use the wasb

[root@hdpmaster01 ~]# cat > test.txt

abc

[root@hdpmaster01 ~]# hdfs dfs -put test.txt wasb://hdpmastercontainer@hdptest01.blob.core.windows.net/

15/03/11 23:49:21 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties

15/03/11 23:49:21 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 60 second(s).

15/03/11 23:49:21 INFO impl.MetricsSystemImpl: azure-file-system metrics system started

15/03/11 23:49:23 INFO impl.MetricsSystemImpl: Stopping azure-file-system metrics system...

15/03/11 23:49:23 INFO impl.MetricsSystemImpl: azure-file-system metrics system stopped.

15/03/11 23:49:23 INFO impl.MetricsSystemImpl: azure-file-system metrics system shutdown complete.

[root@hdpmaster01 ~]# hdfs dfs -ls -R wasb://hdpmastercontainer@hdptest01.blob.core.windows.net/

15/03/11 23:49:35 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties

15/03/11 23:49:35 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 60 second(s).

15/03/11 23:49:35 INFO impl.MetricsSystemImpl: azure-file-system metrics system started

-rw-r--r-- 1 root supergroup 4 2015-03-11 23:49 wasb://hdpmastercontainer@hdptest01.blob.core.windows.net/test.txt

15/03/11 23:49:35 INFO impl.MetricsSystemImpl: Stopping azure-file-system metrics system...

15/03/11 23:49:35 INFO impl.MetricsSystemImpl: azure-file-system metrics system stopped.

15/03/11 23:49:35 INFO impl.MetricsSystemImpl: azure-file-system metrics system shutdown complete.

Useful links:

HDP docs

WASB Configs - Helpful link

Attaching disk

407 Views
Comments

@Neeraj what would be the steps to use WASB in a cluster deployed using cloudbreak? Would that work if I just add key to hdfs-site?

@Guilherme Braccialli

Yes

example: This is based on google compute

@Neeraj I needed to add credential to hive-site to wasb to work inside hive. Did it work for you only with hdfs-site?

Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
1 of 1
Last update:
‎11-01-2015 01:24 AM
Updated by:
 
Contributors
Top Kudoed Authors