For example your physical memory size is 10GB you need to allocated for every container request at the RM, is around 9GB (90% of your physical memory) then you can mention in below parameter yarn.scheduler.maximum-allocation-mb 9216 Similarly amount of physical memory allocated for containers in node manager, that can be mention using below parameter. yarn.nodemanager.resource.memory-mb 9216 These all settings are manage by single configuration file I.e., yarn-site.xml which you can modify via ambari under Yarn —> Configs —> Settings —>Advanced —> Advanced yarn-site I hope that the above answers your questions. Please accept the answer you found most useful. ... View more

@Anjali Shevadkar Below article help you to configure spark fine grain security. https://community.hortonworks.com/content/kbentry/72454/apache-spark-fine-grain-security-with-llap-test-dr.html Please accept the answer you found most useful. ... View more

If you set 100% ram only for yarn then other services will get crash / GC issue. It's not recommendable to set all physical memory to one component. ... View more

For changing memory value through ambari configuration manager, login into ambari UI, Yarn —> Configs —> Settings —> memory —> Memory allocated for all YARN containers on a node and Minimum / Maximum Container Size memory. and for advanced turning of yarn modify Yarn —> Configs —> Settings —>Advanced —> Advanced yarn-site ... View more

@Jack From your above output I can see it's MANAGED_TABLE. If the table has TBLPROPERTIES ("auto.purge"="true") the previous data of the table is not moved to Trash when INSERT OVERWRITE query is run against the table. This functionality is applicable only for managed tables and is turned off when "auto.purge" property is unset or set to false. For more detail HIVE-15880 ... View more

@Jack Please can you share output of below command. It is hive 3.x or hive 2.x ? DESCRIBE FORMATTED <table_name> ... View more

@Ilia K You can manually configure your memory allocation settings based on the hardware specifications. Please refer below links for more tuning up RAM in yarn configuration. https://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/ https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_command-line-installation/content/determine-hdp-memory-config.htm ... View more

@Jack Seems your tables are not partitions. When you try to “INSERT OVERWRITE” to a partition of an external table under existing HDFS directory, depending on whether the partition definition already exists in the HIVE metastore or not, Hive will behave differently: Suppose if partition definition does not exist, it will not try to guess where the target partition directories are either static or dynamic partitions, so it will not be able to delete existing files/data under those partitions. So that will retained as historical data of HDFS like your case. But if partition definition does exist, it will attempt to remove all files/data under the target partition directory before writing new data into those directories. Reference here Please accept the answer you found most useful. ... View more

Please can you attached detailed log ? ... View more