@shyamshaw    This is already highlighted in community here by @LesterMartin  - https://community.cloudera.com/t5/Support-Questions/Reason-for-Hive-dependency-on-PIg-during-installation-of/td-p/239407 >>>> Probably for using HCatalog with can be extremely useful for Pig programmers even if they don't want to use Hive and just leverage this for schema management instead of defining AS clauses in their LOAD commands? Just as likely this is something hard-coded into Ambari? If you really don't want Hive, I bet you can just delete it after installation. For giggles, I stood up an HDFS-only HDP 3.1.0 cluster for https://community.hortonworks.com/questions/245432/is-it-possible-to-install-only-hdfs-on-linux-mach... and just added Pig (required YARN, MR, Tez & ZK, but that makes sense!) and did NOT require Hive to be added as seen below.   Please check the link for full details.   Also same you can remove PIG after installation which will not impact your HIVE. ... View more

Hi@Manoj690 ,   Seems your AMS hbase master is not able to start -     Please try below steps -   In the Ambari Dashboard, go to the 'Ambari Metrics' section and under the 'Service Actions' dropdown click 'Stop'.  Check and confirm from backend that ams process is stopped. If the process is still running use below command to stop - # ambari-metrics-collector stop Delete all AMS Hbase data - In the Ambari Dashboard, under the Ambari Metrics section do a search for the following configuration values  "hbase.rootdir" Remove entire files from “hbase.rootdir” Eg.  #hdfs dfs -cp  /user/ams/hbase/* /tmp/         #hdfs dfs -rm -r -skipTrash  /user/ams/hbase/* In the Ambari Dashboard, under the Ambari Metrics section do a search for the following configuration values  “hbase.tmp.dir”. Backup the directory and remove the data. Eg. #cp /var/lib/ambari-metrics-collector/hbase-tmp/*  /tmp/        #rm –fr /var/lib/ambari-metrics-collector/hbase-tmp/* Remove the znode for hbase in zookeeper cli Login to ambari UI -> Ambari Metrics -> Configs -> Advance ams-hbase-site and search for property  “zookeeper.znode.parent” #/usr/hdp/current/zookeeper-client/bin/zkCli.sh #rmr /ams-hbase-secure   Start AMS   Let know if you still have issue   ... View more

Hi@Manoj690    1.  First check which is the directory configured for dfs data storage. Login to Ambari UI -> Services-> HDFS -> Configs -> [search for dfs.datanode.data.dir] Capture this list of directories defined here. EG. i have list as below - /data01/hadoop/hdfs/data,/data02/hadoop/hdfs/data 2. Login to the datanodes and go to the mount. In my case- $cd /data01/hadoop/hdfs/ 3. Check if there is any directory/data inside "/data01/hadoop/hdfs/" or "/data01/" 4. Other data than "**data" is consider as nondfs and which will show you in dfsadmin -report. 5. You need to get rid of those data which will lesser your NON DFS used.   You can share your output if you have any confusion/need help.   ... View more

@mike_bronson7  you just need to backup  /hadoop/hdfs/namenode/current from active namenode   Also if you backup one week earlier the activity and lets say your first cluster is going serve more request to clients then you will loose that data which was written after backup. So best is to do savenamespace and backup when you are going to do activity and freeze clients not accessing the cluster.   ... View more

@TheBroMeister    I will try to comment my views inline -   1.) How different would the Setup and configuration be for Physical Servers as to VMs. Yes, Setting up the VMs would be faster as compared to the physical ones but are there any additional configurations or settings that we would need to look into? -- If we talk regarding general configuration they below points will be taken in account which counting on performance - a. Disks b Network c. Memory/CPU d. SLA   2.) We've read that one possible issue with setting the cluster on VMs is with Data Locality and redundancy. On how no 2 replicas should not be in the same physical node but since one physical node may house several VMs, would there be a way around this issue? -- VM with external storage[like SAN] will be impacting data locality. You can go with dedicated disk for the VM's which will be a good hybrid approach. 'YES' , also for data locality addon components from virtual vendors[like vmware] are provided - such as BDE [Big Data Extensions] also for Network compromises of NSX technology which will help to speed up systems to avoid performance impacts. But you need to take licensing cost into account.   3.) Since the specs of the VMs would be restricted to the specs of the physical node and its resources be split depending on how many VMs it is housing, wouldn't it be better to have separate servers to house 1 node of a cluster to get better performance? and would having several VMs in one physical node affect the parallelism of the jobs that will run on the cluster? -- Its difficult to put decision at first moment based upon actual experiences. This decision purely depends upon your sla. At start while running hadoop applications, you might not be aware of how much time it takes for your application to process or meet the SLA. This can be purely POC base approach you need to test and also run benchmarking before you go for actual dev/uat/prod implementations. benchmarking results will give you fair idea about performance and computational stats. That can be easy then to take the decision.     Pls do check below links which might be useful - https://community.cloudera.com/t5/Support-Questions/Virtual-Machines-in-Hadoop-cluster/td-p/119675 https://www.kdnuggets.com/2015/12/myths-virtualizing-hadoop-vsphere-explained.html https://pubs.vmware.com/bde-2/index.jsp   ... View more

@VamshiDevraj  If you are still facing issue can you share details about the error or screenshot for the same? ... View more