HDP deployment in Azure and configuring wasb in ambari Lessons learned: All VMs needs to be under virtual private network and hostnames needs to be changed. 1) Create virtual network for your cluster
Note: screen shot has name hdptest01 but it's test 2) Create VMs and choose network created in the above step This is very important step and we need to pay attention to the hostname.
Added endpoint for 8080. There is a step to add disks to vm. You can follow this blog At this point all the hosts are created and we will make changes to hostnames. for example: the following needs to be modified. The command hostname should show internal hostname or same output as hostname -f so modify /etc/sysconfig/network and reboot all the nodes or change hostname using hostname command. [root@hdpmaster01 ~]# cat /etc/sysconfig/network HOSTNAME=hdpmaster01.hdpmaster01.j3.internal.cloudapp.net NETWORKING=yes #hostname hdpmaster01.hdpmaster01.j3.internal.cloudapp.net Follow HDP Docs to install cluster using Ambari Use hostname entries for install , No public DNS Once cluster is installed then add WASB config using Ambari HDFS --> Configs --> under Custom hdfs-site add property fs.azure.account.key.hdptest01.blob.core.windows.net WASB: Get secret key from Azure portal. Restart the services and then test if you can use the wasb [root@hdpmaster01 ~]# cat > test.txt abc [root@hdpmaster01 ~]# hdfs dfs -put test.txt wasb://hdpmastercontainer@hdptest01.blob.core.windows.net/ 15/03/11 23:49:21 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 15/03/11 23:49:21 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 60 second(s). 15/03/11 23:49:21 INFO impl.MetricsSystemImpl: azure-file-system metrics system started 15/03/11 23:49:23 INFO impl.MetricsSystemImpl: Stopping azure-file-system metrics system... 15/03/11 23:49:23 INFO impl.MetricsSystemImpl: azure-file-system metrics system stopped. 15/03/11 23:49:23 INFO impl.MetricsSystemImpl: azure-file-system metrics system shutdown complete. [root@hdpmaster01 ~]# hdfs dfs -ls -R wasb://hdpmastercontainer@hdptest01.blob.core.windows.net/ 15/03/11 23:49:35 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 15/03/11 23:49:35 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 60 second(s). 15/03/11 23:49:35 INFO impl.MetricsSystemImpl: azure-file-system metrics system started -rw-r--r-- 1 root supergroup 4 2015-03-11 23:49 wasb://hdpmastercontainer@hdptest01.blob.core.windows.net/test.txt 15/03/11 23:49:35 INFO impl.MetricsSystemImpl: Stopping azure-file-system metrics system... 15/03/11 23:49:35 INFO impl.MetricsSystemImpl: azure-file-system metrics system stopped. 15/03/11 23:49:35 INFO impl.MetricsSystemImpl: azure-file-system metrics system shutdown complete. Useful links: HDP docs WASB Configs - Helpful link Attaching disk
... View more
