Hi @saikat As I can understand you are running a merge query and it is failing with java.lang.OutOfMemoryError error. Step 1: Could you please run major compaction on all the tables involves in the merge query(If it is an ACID table or else ignore step1). Once the major compaction is triggered make sure it got completed by running "show compactions;" command in the beeline. This will bring down some stats collection burden for the hive. How to run minor and major compaction? Alter table <table name> compact 'MAJOR'; Step 2: Once step1 is done. Please set the following propery in beeline session level and re-run the merge query set hive.tez.container.size=16384; set hive.tez.java.opts=-Xmx13107m; set tez.runtime.io.sort.mb=4096; set tez.task.resource.memory.mb=16384; set tez.am.resource.memory.mb=16384; set tez.am.launch.cmd-opts=-Xmx13107m; set hive.auto.convert.join=false; The TEZ container and AM size is set as 16GB, if the query got failed you can increase the value to 20GB(then hive.tez.java.opts and tez.am.launch.cmd-opts need to be configured 80% of container and AM size that is 16384). If the query got succeeded with 16GB of TEZ container and AM size then you can try to decrease it too 14/12/10 and figure out a benchmark where it is failing and getting succeeded. In this way, you can save resources. If you are happy with the comment, Mark it "Accept as Solution".
... View more
Hi @Nil_kharat Still not resolved the issue. You may need to check the HS2 logs and application logs to figure the slowness. And one more thing how can we track the job that are running by user's. 1. Go to RM UI > Running/finished/killed > check the User column 2. CM > YARN > Applications > Based upon the user you can search over here. If you are happy with the response mark it as Accepts as Solution
... View more
Hi @Nil_kharat Generally, in Hive you may see Query slowness, Query failure, Configuration issue, Alerts, Services down, Vulnerability issue, some bugs this kind of issues you may see.
... View more
Hi @Nil_kharat Yes, you can also use the particular lsof command as well to figure out the number of established connections to the particular port.
... View more
Hi @amitshanker Thanks for the update. I can see than you had set hive.server2.webui.use.spnego as true, that means kerberos is enabled in your cluster. If spnego and kerberos are enabled then few settings needs to be changed in browser. Could you please follow the below link. https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cdh_sg_browser_access_kerberos_protected_url.html
... View more
Hi @Shifu , So supplying these config properties at runtime didn't work but changing the service configuration to modify the below parameters did the job for me: set tez.runtime.io.sort.mb=3072; set tez.task.resource.memory.mb=8192; set tez.am.resource.memory.mb=8192; set tez.am.launch.cmd-opts=-Xmx6553m; Not sure why that might be the case, but the issue seems to have been fixed. Thanks, Megh
... View more
Hi @PURUSHOTHAMAN_S I can see there are a lot of alerts(28) in Ambari, if I were you I will start checking with HDFS service at first like namenode are up and running because it is vital for other services to come up. Then you may need to check YARN and then you can concentrate on others. Check out the Ambari startup logs to see why and where it is getting failed. Hope it helps.
... View more
@Bryan_zh I believe HDP 3.1.5 supports Spark 2.X only. Please check the below link https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.5/spark-overview/content/analyzing_data_with_apache_spark.html How to integrate Hive and Spark? https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.0.1/integrating-hive/content/hive_hivewarehouseconnector_for_handling_apache_spark_data.html
... View more
