@sanjeevan mahajan ... and to add to what Predrag stated based on the documentation, the same is true for all other databases, including Oracle, PostgreSQL, etc. The query needs to be rewritten to achieve the expected result, first find the min(s.from_date) per s.emp_no and at the second step join with e.emp_no to retrieve the needed other fields, as lookups. Try this: SELECT e.emp_no, e.birth_date,e.first_name, e.last_name, e.gender, s.min_from_date FROM employees e, (SELECT emp_no, min(from_date) as min_from_date FROM new2_salaries GROUP BY s.emp_no) s WHERE s.emp_no = e.emp_no; If any of the responses to your question addressed the problem don't forget to vote and accept the answer. If you fix the issue on your own, don't forget to post the answer to your own question. A moderator will review it and accept it. ... View more

@Thomas Larsson Resource Manager UI. You can get to it in two ways: http:/hostname:8088, where hostname is the host name of the server where Resource Manager service runs. Otherwise, from Ambari UI click on YARN (left bar) then click on Quick Links at top middle, then select Resource Manager. You will see the memory and CPU used for each container. One container is allocated to a task. Good tutorial here: http://hadooptutorial.info/yarn-web-ui/ This is the visual. You could also build your own Grafana dashboard making calls to the REST API: https://hadoop.apache.org/docs/r2.7.0/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html Please don't forget to vote/accept best answer for your question. ... View more

@sankar rao An archive has been corrupted. You probably store compressed files (e.g. gzip or lzo) in Hive table directory and at least one of those files is corrupted. I would start moving files out of that folder (HDFS) in reverse chronological order and repeat the query until successful. That way you can find the corrupted archive. There are other ways to test your archives. You could do that too. Try and let me know. If this response or any response in this thread, please don't forget to vote and accept best answer. ... View more

@chandramouli muthukumaran Just to clarify, SparlSQL does not access or use Hive engine. It just consumes the metadata of Hive data structures. Assuming that both can execute the query functionally (SparkSQL is quite limited functionally compared with Hive), but the query will need to churn through 40 TB of data, then I would say likely Hive on Tez is your optimal choice. That is also driven by the cost associated with your Spark cluster RAM additional to Hive's requirements because I assume that you will still have some cases where running Hive is needed. I noticed that if the amount of data is less than 1 TB, SparkSQL outperforms Hive on Tez. Anyhow, be aware, that with HDP 2.5 LLAP is in Tech Preview and soon will be GA. If you were asking Hive on LLAP vs. SparkSQL, I would say without hesitation for most of the queries, Hive on LLAP. Again, for some sofisticated queries with limited amount of data, and limited function, SparkSQL may be a winner, but in the big picture is too expensive to maintain both approaches and I would still consider Hive on Tez and LLAP over SparkSQL for most of the cases that deal with BIG DATA. Otherwise, 1 TB does not need Hadoop for fast queries. Read more about Hive on LLAP here: http://hortonworks.com/blog/llap-enables-sub-second-sql-hadoop/ Give LLAP a shot before deciding to use SparkSQL, especially, if you already have the queries written in HiveQL. If this response or any response in this thread was helpful, please don't forget to vote/accept it as the best answer. ... View more

It seems that the data does not go to thrash. A simple restart of Kafka service addressed the issue. Kerberos was enabled recently and probably this service was not restarted. The symlink suggestion from deepak is an interesting approach which while not applicable here, is worth it to remember for other situations. ... View more