HDP deployment in Azure and configuring wasb in ambari
Lessons learned: All VMs needs to be under virtual private network and hostnames needs to be changed.
1) Create virtual network for your cluster
Note: screen shot has name hdptest01 but it's test
2) Create VMs and choose network created in the above step
This is very important step and we need to pay attention to the hostname.
Added endpoint for 8080.
There is a step to add disks to vm. You can follow this blog
At this point all the hosts are created and we will make changes to hostnames.
for example: the following needs to be modified. The command hostname should show internal hostname or same output as hostname -f so modify /etc/sysconfig/network and reboot all the nodes or change hostname using hostname command.
[root@hdpmaster01 ~]# cat /etc/sysconfig/network
HOSTNAME=hdpmaster01.hdpmaster01.j3.internal.cloudapp.net
NETWORKING=yes
#hostname
hdpmaster01.hdpmaster01.j3.internal.cloudapp.net
Follow HDP Docs to install cluster using Ambari
Use hostname entries for install , No public DNS
Once cluster is installed then add WASB config using Ambari
HDFS --> Configs --> under Custom hdfs-site
add property
fs.azure.account.key.hdptest01.blob.core.windows.net
WASB: Get secret key from Azure portal.
Restart the services and then test if you can use the wasb
[root@hdpmaster01 ~]# cat > test.txt
abc
[root@hdpmaster01 ~]# hdfs dfs -put test.txt wasb://hdpmastercontainer@hdptest01.blob.core.windows.net/
15/03/11 23:49:21 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
15/03/11 23:49:21 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 60 second(s).
15/03/11 23:49:21 INFO impl.MetricsSystemImpl: azure-file-system metrics system started
15/03/11 23:49:23 INFO impl.MetricsSystemImpl: Stopping azure-file-system metrics system...
15/03/11 23:49:23 INFO impl.MetricsSystemImpl: azure-file-system metrics system stopped.
15/03/11 23:49:23 INFO impl.MetricsSystemImpl: azure-file-system metrics system shutdown complete.
[root@hdpmaster01 ~]# hdfs dfs -ls -R wasb://hdpmastercontainer@hdptest01.blob.core.windows.net/
15/03/11 23:49:35 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
15/03/11 23:49:35 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 60 second(s).
15/03/11 23:49:35 INFO impl.MetricsSystemImpl: azure-file-system metrics system started
-rw-r--r-- 1 root supergroup 4 2015-03-11 23:49 wasb://hdpmastercontainer@hdptest01.blob.core.windows.net/test.txt
15/03/11 23:49:35 INFO impl.MetricsSystemImpl: Stopping azure-file-system metrics system...
15/03/11 23:49:35 INFO impl.MetricsSystemImpl: azure-file-system metrics system stopped.
15/03/11 23:49:35 INFO impl.MetricsSystemImpl: azure-file-system metrics system shutdown complete.
Useful links:
WASB Configs - Helpful link
