Member since
05-25-2018
77
Posts
2
Kudos Received
0
Solutions
02-07-2017
04:29 AM
sunile , after i set hive.tez.container.size to -1 , this issue got resolved. But still i get below error when in run insert statements from beeline 17/02/07 04:06:36 [HiveServer2-Background-Pool: Thread-137]: WARN tez.DagUtils: hive.tez.java.opts will be ignored because hive.tez.container.size is not set!
... View more
02-07-2017
03:54 AM
Hi Sunile, I cannot increase hive.tez.container.size more than yarn.scheduler.maximum-allocation-mb . Increase to 4000mb , and still getting the same error. below are my setting now. yarn.scheduler.minimum-allocation-mb : 2048mb yarn.scheduler.maximum-allocation-mb : 5120mb hive.tez.container.size 2048 ----> 4000mb [root@node01 ~]# vmstat -s
8 009212 K total memory 3091192 K used memory 3869668 K active memory 2283236 K inactive memory 723736 K free memory
... View more
02-06-2017
11:43 PM
Hi, When implementing Ranger-hive policy , i made below changes 1) hive.conf.restricted.list old : hive.security.authenticator.manager,hive.security.authorization.manager,hive.users.in.admin.role new : hive.security.authenticator.manager,hive.security.authorization.manager,hive.users.in.admin.role,hive.security.authorization.enabled 2) hive.tez.container.size old : -1 New : 2048 mb ( equal to yarn.scheduler.minimum-allocation-mb) 3) hive.auto.convert.join.noconditionaltask.size old : 52428800 new : 357913941 4) hive.security.authorization.enabled old : false new : true 5) hive.security.authorization.manager : old : org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactory new : org.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizerFactory There after started getting below error from both beeline and hive , when i execute commands Status: Failed Vertex failed, vertexName=Map 1, vertexId=vertex_1486082581831_0016_1_00, diagnostics=[Task failed, taskId=task_1486082581831_0016_1_00_000000, diagnostics=[TaskAttempt 0 failed, info=[Container container_1486082581831_0016_01_000002 finished with diagnostics set to [Container failed, exitCode=1. Exception from container-launch. Container id: container_1486082581831_0016_01_000002 Exit code: 1 Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:576) at org.apache.hadoop.util.Shell.run(Shell.java:487) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:303) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 1 ]], TaskAttempt 1 failed, info=[Container container_1486082581831_0016_01_000003 finished with diagnostics set to [Container failed, exitCode=1. Exception from container-launch. Container id: container_1486082581831_0016_01_000003 Exit code: 1 Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:576) at org.apache.hadoop.util.Shell.run(Shell.java:487) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:303) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) *************** yarn version is 2.7.1.2.4 tez version is 0.7.0.2.4
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Ranger
02-06-2017
03:50 AM
Can you please give me the site url to download the needed rmp of zlib?
... View more
02-06-2017
02:42 AM
Hi , Can you please give the download link for zlib compression file / rpm for centos and installation steps Thanks JJ
... View more
01-30-2017
10:49 PM
Hi , Thank you very much for your suggestion. Now the issue got resolved. This was related to port open issue. Thanks JJ
... View more
01-30-2017
09:55 PM
Hi , it says : node1.finity.com refused to connect. when i checked the process and netstat and lsof it show below results : [ranger@node1 bin]$ ps -ef|grep -i ranger root 5974 14338 0 21:04 pts/2 00:00:00 su - ranger ranger 5975 5974 0 21:04 pts/2 00:00:00 -bash ranger 12223 1 0 03:31 ? 00:01:50 java -Dproc_rangeradmin -XX:MaxPermSize=256m -Xmx1024m -Xms1024m -Dlogdir=/usr/hdp/2.4.2.0-258/ranger-admin/ews/logs/ -Dcatalina.base=/usr/hdp/2.4.2.0-258/ranger-admin/ews -cp /usr/hdp/2.4.2.0-258/ranger-admin/ews/webapp/WEB-INF/classes/conf:/usr/hdp/2.4.2.0-258/ranger-admin/ews/lib/*:/usr/hdp/2.4.2.0-258/ranger-admin/ews/ranger_jaas/*:/usr/hdp/2.4.2.0-258/ranger-admin/ews/webapp/WEB-INF/classes/conf/ranger_jaas:/usr/lib/jvm/jre-1.8.0/lib/*: org.apache.ranger.server.tomcat.EmbeddedServer ranger 12457 1 0 03:31 ? 00:00:42 java -Dproc_rangerusersync -Dlog4j.configuration=file:/etc/ranger/usersync/conf/log4j.xml -Dlogdir=/var/log/ranger/usersync -cp /usr/hdp/2.4.2.0-258/ranger-usersync/dist/*:/usr/hdp/2.4.2.0-258/ranger-usersync/lib/*:/usr/hdp/2.4.2.0-258/ranger-usersync/conf org.apache.ranger.authentication.UnixAuthenticationService -enableUnixAuth ranger 15216 5975 0 21:17 pts/2 00:00:00 ps -ef ranger 15217 5975 0 21:17 pts/2 00:00:00 grep --color=auto -i ranger [ranger@node1 bin]$ netstat -a | grep 6080 tcp6 0 0 [::]:6080 [::]:* LISTEN tcp6 0 0 node1.:6080 node1:38894 TIME_WAIT tcp6 0 0 snode1.:6080 node1:38898 TIME_WAIT [ranger@node1 bin]$ lsof -i :6080 -S COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 12223 ranger 18u IPv6 199416 0t0 TCP *:6080 (LISTEN) **************** your help is needed further Thanks JJ
... View more
01-30-2017
07:29 PM
Hi , I am not able to access the Ranger Admin UI from Ranger--> Quick links Can you please suggest what log file i should check and work around for it . Ranger version we are using is Ranger 0.5.0.2.4 Thanks JJ
... View more
Labels:
- Labels:
-
Apache Ranger
01-27-2017
04:04 AM
1 Kudo
Hi , Can you give me the steps for running the .hql scripts from beeline prompt? i get below error. 0: jdbc:hive2://> source /home/local/hive/p.hql; NoViableAltException(26@[]) at org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1079) at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202) at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166) **************** i am comfortable with below command and it works 1) hive> source /home/local/hive/p.hql; OK table_1 table_2 Time taken: 0.231 seconds, Fetched: 2 row(s) hive> 2) beeline -u jdbc:hive2://xxxxxxxxxx:10000/default -n hive -p hive -f /home/local/hive/st.hql >> out.txt 3) $HIVE_HOME/bin/hive -f /home/local/hive/st.hql --database default > out.txt
... View more
Labels:
- Labels:
-
Apache Hive
01-19-2017
07:11 AM
Hi , though yarn.nodemanager.log-dirs is set to /hadop/yarn/log , logs are getting written to /var/log/hadoop-yarn/yarn May i know the reason thanks JJ
... View more
Labels:
- Labels:
-
Apache YARN
- « Previous
- Next »