HDP deployment in Azure and configuring wasb in ambari Lessons learned: All VMs needs to be under virtual private network and hostnames needs to be changed. 1) Create virtual network for your cluster Note: screen shot has name hdptest01 but it's test 2) Create VMs and choose network created in the above step This is very important step and we need to pay attention to the hostname. Added endpoint for 8080. There is a step to add disks to vm. You can follow this blog At this point all the hosts are created and we will make changes to hostnames. for example: the following needs to be modified. The command hostname should show internal hostname or same output as hostname -f so modify /etc/sysconfig/network and reboot all the nodes or change hostname using hostname command. [root@hdpmaster01 ~]# cat /etc/sysconfig/network HOSTNAME=hdpmaster01.hdpmaster01.j3.internal.cloudapp.net NETWORKING=yes #hostname hdpmaster01.hdpmaster01.j3.internal.cloudapp.net Follow HDP Docs to install cluster using Ambari Use hostname entries for install , No public DNS Once cluster is installed then add WASB config using Ambari HDFS --> Configs --> under Custom hdfs-site add property fs.azure.account.key.hdptest01.blob.core.windows.net WASB: Get secret key from Azure portal. Restart the services and then test if you can use the wasb [root@hdpmaster01 ~]# cat > test.txt abc [root@hdpmaster01 ~]# hdfs dfs -put test.txt wasb://hdpmastercontainer@hdptest01.blob.core.windows.net/ 15/03/11 23:49:21 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 15/03/11 23:49:21 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 60 second(s). 15/03/11 23:49:21 INFO impl.MetricsSystemImpl: azure-file-system metrics system started 15/03/11 23:49:23 INFO impl.MetricsSystemImpl: Stopping azure-file-system metrics system... 15/03/11 23:49:23 INFO impl.MetricsSystemImpl: azure-file-system metrics system stopped. 15/03/11 23:49:23 INFO impl.MetricsSystemImpl: azure-file-system metrics system shutdown complete. [root@hdpmaster01 ~]# hdfs dfs -ls -R wasb://hdpmastercontainer@hdptest01.blob.core.windows.net/ 15/03/11 23:49:35 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 15/03/11 23:49:35 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 60 second(s). 15/03/11 23:49:35 INFO impl.MetricsSystemImpl: azure-file-system metrics system started -rw-r--r-- 1 root supergroup 4 2015-03-11 23:49 wasb://hdpmastercontainer@hdptest01.blob.core.windows.net/test.txt 15/03/11 23:49:35 INFO impl.MetricsSystemImpl: Stopping azure-file-system metrics system... 15/03/11 23:49:35 INFO impl.MetricsSystemImpl: azure-file-system metrics system stopped. 15/03/11 23:49:35 INFO impl.MetricsSystemImpl: azure-file-system metrics system shutdown complete. Useful links: HDP docs WASB Configs - Helpful link Attaching disk ... View more

Post based on KB article: As soons as you realize that you have accidentally deleted the the files/directories in HDFS, immediately force the name node to go in SafeMode (hadoop namenode -safemode enter) to avoid further damage to file system. If possible, shutdown the NN and DataNodes Check out the core-site.xml, to verify if Trash is enabled (look for configuration parameter ‘fs.trash.interval’. The value greater than zero value indicates number or minutes that data will be kept in Trash) If the number is more than the duration between delete command execution and namenode shutdown/safe mode, then you can recover the files from user’s Trash folder. If Trash is not enabled, contact Hortonworks support immediately. DO NOT restart the namenode and/or bring it out of safe mode. Doing so may delete your data permanently. ... View more

XML: [hdfs@sandbox ~]$ hdfs dfs -mkdir -p /test/sequences [hdfs@sandbox ~]$ [hdfs@sandbox ~]$ vi sample.xml <sequences period=”5″> <moment game-clock=”300.00″ time=”1433732215737″ game-event-id=”” shot-clock=”24.00″ locations=”-1,-1,96.95182,1.98648,5.75987;9,173004,45.54661,17.35545,0;9,338365,24.04722,25.67399,0;9,457611,46.95292,27.93478,0;9,468895,48.59834,33.96586,0;9,552336,33.73381,24.05929,0;5,214152,59.26872,24.12006,0;5,253997,45.71551,17.41071,0;5,457186,48.59834,33.96586,0;5,531447,78.09629,34.24688,0;5,552806,47.79678,22.8155,0″/> <moment game-clock=”300.00″ time=”1433732215794″ game-event-id=”” shot-clock=”24.00″ locations=”-1,-1,97.79683,.89407,3.67626;9,173004,45.62283,17.34854,0;9,338365,24.04248,25.6784,0;9,457611,46.84978,27.8463,0;9,468895,48.52017,33.89189,0;9,552336,33.74064,24.03493,0;5,214152,59.27282,24.07895,0;5,253997,45.87101,17.38532,0;5,457186,48.52017,33.89189,0;5,531447,78.06394,34.2155,0;5,552806,47.8269,22.81393,0″/> <moment game-clock=”300.00″ time=”1433732215829″ game-event-id=”” shot-clock=”24.00″ locations=”-1,-1,97.78946,.91006,3.68332;9,173004,45.61995,17.35703,0;9,338365,24.03815,25.68402,0;9,457611,46.71077,27.71191,0;9,468895,48.37095,33.77756,0;9,552336,33.74769,24.00829,0;5,214152,59.27627,24.06055,0;5,253997,46.00077,17.36555,0;5,457186,48.37095,33.77756,0;5,531447,78.0439,34.20521,0;5,552806,47.84297,22.83292,0″/> <moment game-clock=”300.00″ time=”1433732215856″ game-event-id=”” shot-clock=”24.00″ locations=”-1,-1,97.73786,1.02206,3.73271;9,173004,45.57851,17.34979,0;9,338365,24.04207,25.61049,0;9,457611,46.63871,27.56226,0;9,468895,48.2033,33.7142,0;9,552336,33.75497,23.97935,0;5,214152,59.27906,24.06485,0;5,253997,46.10481,17.35141,0;5,457186,48.29748,33.63262,0;5,531447,78.03618,34.216,0;5,552806,47.84498,22.87247,0″/> <moment game-clock=”300.00″ time=”1433732215905″ game-event-id=”” shot-clock=”24.00″ locations=”-1,-1,97.59781,1.32606,3.8668;9,173004,45.57865,17.34643,0;9,338365,24.04224,25.61058,0;9,457611,46.56615,27.44014,0;9,468895,48.01722,33.7018,0;9,552336,33.76247,23.94813,0;5,214152,59.27976,24.07223,0;5,253997,46.26668,17.38672,0;5,457186,48.29974,33.45708,0;5,531447,78.02931,34.2208,0;5,552806,47.86752,22.85019,0″/> </sequences> [hdfs@sandbox ~]$ hdfs dfs -put sample.xml /test/sequences/ [hdfs@sandbox ~]$ [hdfs@sandbox ~]$ wget http://search.maven.org/remotecontent?filepath=com/ibm/spss/hive/serde2/xml/hivexmlserde/1.0.5.3/hivexmlserde-1.0.5.3.jar [hdfs@sandbox ~]$ mv remotecontent?filepath=com%2Fibm%2Fspss%2Fhive%2Fserde2%2Fxml%2Fhivexmlserde%2F1.0.5.3%2Fhivexmlserde-1.0.5.3.jar hivexmlserde-1.0.5.3.jar [ hdfs@sandbox ~]$ pwd /home/hdfs [hdfs@sandbox ~]$ mv hivexmlserde-1.0.5.3.jar /tmp hive hive> add jar /tmp/hivexmlserde-1.0.5.3.jar; Added [/tmp/hivexmlserde-1.0.5.3.jar] to class path Added resources: [/tmp/hivexmlserde-1.0.5.3.jar] hive> drop table sequences; OK Time taken: 3.717 seconds CREATE EXTERNAL TABLE sequences( gameclock double, time bigint, gameeventid string, shotclock double, locations string ) ROW FORMAT SERDE ‘com.ibm.spss.hive.serde2.xml.XmlSerDe’ WITH SERDEPROPERTIES ( “column.xpath.gameclock”=”/moment/@game-clock”, “column.xpath.time”=”/moment/@time”, “column.xpath.gameeventid”=”/moment/@game-event-id”, “column.xpath.shotclock”=”/moment/@shot-clock”, “column.xpath.locations”=”/moment/@locations” ) STORED AS INPUTFORMAT ‘com.ibm.spss.hive.serde2.xml.XmlInputFormat’ OUTPUTFORMAT ‘org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat’ LOCATION ‘/test/sequences’ TBLPROPERTIES ( “xmlinput.start”=”<moment “, “xmlinput.end”=”/” ) ; OK Time taken: 2.871 seconds hive> hive> select * from sequences; OK 300.01433732215737 24.0 -1,-1,96.95182,1.98648,5.75987;9,173004,45.54661,17.35545,0;9,338365,24.04722,25.67399,0;9,457611,46.95292,27.93478,0;9,468895,48.59834,33.96586,0;9,552336,33.73381,24.05929,0;5,214152,59.26872,24.12006,0;5,253997,45.71551,17.41071,0;5,457186,48.59834,33.96586,0;5,531447,78.09629,34.24688,0;5,552806,47.79678,22.8155,0 300.01433732215794 24.0 -1,-1,97.79683,.89407,3.67626;9,173004,45.62283,17.34854,0;9,338365,24.04248,25.6784,0;9,457611,46.84978,27.8463,0;9,468895,48.52017,33.89189,0;9,552336,33.74064,24.03493,0;5,214152,59.27282,24.07895,0;5,253997,45.87101,17.38532,0;5,457186,48.52017,33.89189,0;5,531447,78.06394,34.2155,0;5,552806,47.8269,22.81393,0 300.01433732215829 24.0 -1,-1,97.78946,.91006,3.68332;9,173004,45.61995,17.35703,0;9,338365,24.03815,25.68402,0;9,457611,46.71077,27.71191,0;9,468895,48.37095,33.77756,0;9,552336,33.74769,24.00829,0;5,214152,59.27627,24.06055,0;5,253997,46.00077,17.36555,0;5,457186,48.37095,33.77756,0;5,531447,78.0439,34.20521,0;5,552806,47.84297,22.83292,0 300.01433732215856 24.0 -1,-1,97.73786,1.02206,3.73271;9,173004,45.57851,17.34979,0;9,338365,24.04207,25.61049,0;9,457611,46.63871,27.56226,0;9,468895,48.2033,33.7142,0;9,552336,33.75497,23.97935,0;5,214152,59.27906,24.06485,0;5,253997,46.10481,17.35141,0;5,457186,48.29748,33.63262,0;5,531447,78.03618,34.216,0;5,552806,47.84498,22.87247,0 300.01433732215905 24.0 -1,-1,97.59781,1.32606,3.8668;9,173004,45.57865,17.34643,0;9,338365,24.04224,25.61058,0;9,457611,46.56615,27.44014,0;9,468895,48.01722,33.7018,0;9,552336,33.76247,23.94813,0;5,214152,59.27976,24.07223,0;5,253997,46.26668,17.38672,0;5,457186,48.29974,33.45708,0;5,531447,78.02931,34.2208,0;5,552806,47.86752,22.85019,0 Time taken: 2.8 seconds, Fetched: 5 row(s) hive> > select gameclock, time,gameeventid, shotclock, concat(substr(locations,1,10), ‘ …’) as sequence_truncated_for_email_to_anoop from sequences; OK 300.01433732215737 24.0 -1,-1,96.9 … 300.01433732215794 24.0 -1,-1,97.7 … 300.01433732215829 24.0 -1,-1,97.7 … 300.01433732215856 24.0 -1,-1,97.7 … 300.01433732215905 24.0 -1,-1,97.5 … Time taken: 0.898 seconds, Fetched: 5 row(s) hive> ... View more