Hi Guys, We are using HDP 3 , we met a problem that when user shoot a big query , it will make our HS2 ( or HS2-interactive) OOM , even i already increase the heap size to 20G. The query is simple but it will return 50,000,000 records. We told user that they should limit the query, but not sure if there is a way to auto detect the size of resultset and auto kill it to prevent it make our HS2 crash.  So my question is , how to protect our hs2 ?  ... View more

Hi Guys , I am finding a solution to monitoring Spark SQL from backend . For hive , we implement a hook on hiveserver2 so that all hive sql via Cli/JDBC/Beeline can be captured, include SQL , run time , user ,etc. Is there a way to do same thing in Spark ? In our scenario , we use spark submit/spark shell / spark-sql to run spark program . ... View more