@ Rajkumar Singh TThanks you so much .. In my case i don't want to display some data so that i have drooped some records in table using partitioned drooped.. so again if i want to display that partitioned records .is there way to display that records? Note: i could see records in warehouse directory , i have droped only partitioned , i have't touch the data still i could browse the data.. ... View more

Please select the best answer to close out the thread ... View more

HI @jss I could find some configuration difference here In PROD cluster ->managed ambari->views>hive->hive views->cluster configuration is set to "Local Ambari Managed Cluster" and its working fine IN Dev cluster ->managed ambari->views>hive->hive views->cluster configuration is set to " custom" here it is not working So should i change to "Local Ambari Managed Cluster" if you want to me do this one , after save this should i need to restart the ambari server? please suggest me ... View more

This is a standard approach. You could add a data count check on source and target, just to give you peace of mind. ... View more

@Ayub Pathan Both are using same version HDP 2.1.2 I for got mansion port but it has 8020 both cluster export table db_c720_dcm.network_matchtables_act_ad to 'apps/hive/warehouse/sankar7_dir'; and i could see sankar7_dir in /user/hdfs/apps/hive/warehouse/sankar7_dir in source cluster... hadoop distcp hdfs://xx.xx.xx.xx:8020/apps/hive/warehouse/sankar7_dir hdfs://yy.yy.yy.yy:8020/apps/hive/warehous e/sankar7_dir 16/10/13 01:01:05 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=fa lse, maxMaps=20, sslConfigurationFile='null', copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[hdfs:///xx.xx.xx.xx:8020/apps/h ive/warehouse/sankar7_dir], targetPath=hdfs://yy.yy.yy.yy4:8020/apps/hive/warehouse/sankar7_dir} 16/10/13 01:01:05 INFO client.RMProxy: Connecting to ResourceManager at stlts8711/39.0.8.13:8050 16/10/13 01:01:06 ERROR tools.DistCp: Invalid input: org.apache.hadoop.tools.CopyListing$InvalidInputException: hdfs:///xx.xx.xx.xx:8020/apps/hive/warehouse/sankar7_dir doesn't exist at org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:84) at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:80) at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:327) at org.apache.hadoop.tools.DistCp.execute(DistCp.java:151) at org.apache.hadoop.tools.DistCp.run(DistCp.java:118) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.tools.DistCp.main(DistCp.java:375) hdfs:///xx.xx.xx.xx:8020/apps/hive/warehouse/sankar7_dir doesn't existIif see my error while doing Distcp with out creating sankar_7. But i export table to directory :. export table db_c720_dcm.network_matchtables_act_ad to 'apps/hive/warehouse/sankar7_dir'; ... View more

@rama These directories are used by yarn for job logs. There are similar directories used for localization, the yarn-local dirs. They are not distributed so much as used when containers are allocated on that node. They should get cleaned up when jobs complete but can leave orphaned files in the event of a Resource Manager restart or a Node Manager restart. The directories are configured via yarn and it is a comma separated list of locations so you can add additional mounts/directories but they will apply to all node managers managed by Yarn. Hope this helps. ... View more

I am using HDP 2.1.2 I can't see this property i have screen short hiveheap.png ... View more

@rama Yes. That's the idea. One of the files is somehow corrupted. ... View more

@sankar rao, Actually I don't have the yarn cluster ready to confirm you what log lines needs to be searched in the container logs for the file names. Probably , it will be better if you can raise a support case so that a dedicated team can look into the issue specifically. As it depends, How is the data loaded in the table/hdfs , how they are zipped, which input format you are using etc.? ... View more

Thank you @Constantin Stanca I have learn lot of things from your answer ..I have a few concerns over this comment, Can you please make to overcome these.. #How can i set the alerts to preempt YARN 100% # How can i achieve this one " could do a lot of things with Tez parameters" ... View more