Member since
11-08-2016
32
Posts
7
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1081 | 11-09-2016 08:58 AM |
03-23-2017
04:44 PM
awesome @Ken Jiiii hive-site.xml should be available across the cluster in /etc/spark/conf ( where /usr/hdp/current/spark-client/conf will be symlink to) and spark client need to be installed across the cluster worker nodes for your yarn-cluster mode to run as your spark driver can run on any worker node and should be having client installed with spark/conf. If you are using Ambari it will taking care of hive-site.xml available in /spark-client/conf/
... View more
02-27-2017
10:15 AM
Hi Sunile, Thanks for your answer. We think we will store our initial model and then all alter scripts. But all alter scripts will be included in the initial model in cas a complete re-deployment is wanted. To view a logical model a tool will be used which can reverse engineer the ddl. We try to establish a workflow in that fashion and hope that works.
... View more
12-29-2016
12:27 AM
3 Kudos
@Ken Jiiii You could create another file and call it from h_script.hql, or you just add lines to h_script.hql like the following: This set statements will override the global settings for that specific job session. Example of line: set mapred.reduce.tasks=32;
Almost anything that is Hive global environments XML is game for override at session level. Look at this post: https://community.hortonworks.com/articles/22419/hive-on-tez-performance-tuning-determining-reducer.html This shows a lot of these set statements which you can include in your set_h_script.hql invoked from h_script.hql or in the block added the beginning of your h_script.hql. Personally, I prefer a separate file invoked from my job script.
... View more
11-09-2016
08:58 AM
It seems that I managed to solve the problem. What was missing: oozie admin -oozie http://localhost:11000/oozie -sharelibupdate One can check the share lib by oozie admin -oozie http://localhost:11000/oozie -shareliblist sqoop I hope this helps also others and this should be the same for different database systems.
... View more