@Govins28 We are creating cluster from scratch   ... View more

@ Govins28   Its a test cluster ... View more

@Govins28   PFA NN logs.   Ambari error is as under Connection failed to http://xxxxx.com:50070(<urlopen error [Errno 111] Connection refused>)     ... View more

@ Govins28 We are using HDP 3.0.1   ... View more

@Shelton    While starting namenode manually facing issue PFA enclosed. ... View more

@Shelton   Sure.. will follow the steps as advised & update you. Thanks ! ... View more

Hi @Shelton,   We have disabled ranger authorization in Ambari & allow to run hive as end user instead of hive user. Still hiveserver2 is not coming up.   2020-03-26 05:36:05,838 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server <master node>:2181,<datanode1>:2181,<data node3>:2181 ls /hiveserver2 | grep 'serverUri=''] {} 2020-03-26 05:36:06,497 - call returned (1, 'Node does not exist: /hiveserver2') 2020-03-26 05:36:06,498 - Will retry 1 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)     Any clue on this ?       ... View more

Hi @Shelton Getting same message    [zk: localhost:2181(CONNECTED) 8] getAcl /hiveserver2 Node does not exist: /hiveserver2​ [zk: localhost:2181(CONNECTED) 9]​     ... View more

  getting below error for HivServer2    2020-03-26 05:36:05,838 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server <master node server>:2181,us01qc1hdpdn01.dev.cds:2181,<date node server>:2181 ls /hiveserver2 | grep 'serverUri=''] {} 2020-03-26 05:36:06,497 - call returned (1, 'Node does not exist: /hiveserver2') 2020-03-26 05:36:06,498 - Will retry 1 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)   ... View more

Hi @dayphache   Could you pls elaborate your response ? I can see only 33 in the reply , didin't get what it means. ... View more

Hi @Shelton,   HDP version is 3.0.1 We managed to change the hive user name in service accounts. However , hive is unable to come up with below error   Error: org.apache.hive.jdbc.ZooKeeperHiveClientException : Unable to read HiveServer2 configs from ZooKeeper (state=,code=0)   Pls advise.   Thanks ! ... View more

Hi @Shelton  Is it possible to change the service user name through linux? If so pls share the details.     Thanks ! ... View more

Right @ Shelton   Is there any way to modify service user account in Ambari   we need to change value for hive to hive_xxxx which has been configured at Isilon side. Due to this we are getting hive error & it needs to be fixed as above as per Linux admin team.   e.g.  within Ambari we need to change following hive user to hive_xxx   Service Users and Groups Hive User hive   ... View more

@ Shelton We thought of re-installing hive service where earlier it was running on data node & we thought of re-enabling it on master node. The idea was to check if there are permission issues still   ... View more

Thanks @Shelton   However, observed below error in Hive within Ambari when tried to re-install hive service   File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call     raise ExecutionFailed(err_msg, code, out, err) resource_management.core.exceptions.ExecutionFailed: Execution of 'hadoop --config /usr/hdp/3.0.1.0-187/hadoop/conf jar /var/lib/ambari-agent/lib/fast-hdfs-resource.jar /var/lib/ambari-agent/tmp/hdfs_resources_1585061727.13.json' returned 1. Initializing filesystem uri: hdfs://<masternode>:8020 Creating: Resource [source=null, target=/hdp/apps/3.0.1.0-187/mapreduce, type=directory, action=create, owner=hdfs-<user>, group=null, mode=555, recursiveChown=false, recursiveChmod=false, changePermissionforParents=false, manageIfExists=true] in default filesystem Creating: Resource [source=/usr/hdp/3.0.1.0-187/hadoop/mapreduce.tar.gz, target=/hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz, type=file, action=create, owner=hdfs-US01QC1, group=hadoop-US01QC1, mode=444, recursiveChown=false, recursiveChmod=false, changePermissionforParents=false, manageIfExists=true] in default filesystem Creating: Resource [source=null, target=/hdp/apps/3.0.1.0-187/tez, type=directory, action=create, owner=hdfs-US01QC1, group=null, mode=555, recursiveChown=false, recursiveChmod=false, changePermissionforParents=false, manageIfExists=true] in default filesystem Creating: Resource [source=/var/lib/ambari-agent/tmp/tez-native-tarball-staging/tez-native.tar.gz, target=/hdp/apps/3.0.1.0-187/tez/tez.tar.gz, type=file, action=create, owner=hdfs-<user>, group=hadoop-US01QC1, mode=444, recursiveChown=false, recursiveChmod=false, changePermissionforParents=false, manageIfExists=true] in default filesystem Creating: Resource [source=null, target=/hdp/apps/3.0.1.0-187/hive, type=directory, action=create, owner=hdfs-<user>, group=null, mode=555, recursiveChown=false, recursiveChmod=false, changePermissionforParents=false, manageIfExists=true] in default filesystem Creating: Resource [source=/usr/hdp/3.0.1.0-187/hive/hive.tar.gz, target=/hdp/apps/3.0.1.0-187/hive/hive.tar.gz, type=file, action=create, owner=hdfs-US01QC1, group=hadoop-<user>, mode=444, recursiveChown=false, recursiveChmod=false, changePermissionforParents=false, manageIfExists=true] in default filesystem Creating: Resource [source=null, target=/hdp/apps/3.0.1.0-187/mapreduce, type=directory, action=create, owner=hdfs-<user>, group=null, mode=555, recursiveChown=false, recursiveChmod=false, changePermissionforParents=false, manageIfExists=true] in default filesystem Creating: Resource [source=/usr/hdp/3.0.1.0-187/hadoop-mapreduce/hadoop-streaming.jar, target=/hdp/apps/3.0.1.0-187/mapreduce/hadoop-streaming.jar, type=file, action=create, owner=hdfs-<user>, group=hadoop-<user>, mode=444, recursiveChown=false, recursiveChmod=false, changePermissionforParents=false, manageIfExists=true] in default filesystem Creating: Resource [source=null, target=/warehouse/tablespace/external/hive/sys.db/, type=directory, action=create, owner=hive, group=null, mode=1755, recursiveChown=false, recursiveChmod=false, changePermissionforParents=false, manageIfExists=true] in default filesystem Exception occurred, Reason: Error mapping uname 'hive' to uid org.apache.hadoop.ipc.RemoteException(java.lang.SecurityException): Error mapping uname 'hive' to uid at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1497) at org.apache.hadoop.ipc.Client.call(Client.java:1443) at org.apache.hadoop.ipc.Client.call(Client.java:1353) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) at com.sun.proxy.$Proxy9.setOwner(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.setOwner(ClientNamenodeProtocolTranslatorPB.java:470) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy10.setOwner(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.setOwner(DFSClient.java:1908) at org.apache.hadoop.hdfs.DistributedFileSystem$36.doCall(DistributedFileSystem.java:1770) at org.apache.hadoop.hdfs.DistributedFileSystem$36.doCall(DistributedFileSystem.java:1767) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.setOwner(DistributedFileSystem.java:1780) at org.apache.ambari.fast_hdfs_resource.Resource.setOwner(Resource.java:276) at org.apache.ambari.fast_hdfs_resource.Runner.main(Runner.java:133) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:318) at org.apache.hadoop.util.RunJar.main(RunJar.java:232) ... View more

Hi @Shelton,   Yes I did that & I can see the user in /etc/passwd file   I do not have any documentation on user mapping as currently it's controlled by EMC team (DELL Support)  ... View more

Hi @Shelton  I changed the permissions as advised. However, getting the same error. The new folder inside _tez_session_dir gets created with root ownership instead of <hive_user>    Pls note that <hive_user> has been mapped to root at Isilon side.   I tried creating another user hdpuser1 using below commands in mysql after login as root   create user 'hdpuser1'@'%'  identified by 'xxxxx'; grant all on hive.* TO 'hdpuser1'@'%' IDENTIFIED BY 'xxxxx'; FLUSH PRIVILEGES;   However, this doesn't work. 20/03/24 02:24:55 [main]: WARN jdbc.HiveConnection: Failed to connect to <FQDN of Data Node>:10000 20/03/24 02:24:55 [main]: ERROR jdbc.Utils: Unable to read HiveServer2 configs from ZooKeeper​ Error: Could not open client transport for any of the Server URI's in ZooKeeper: Failed to open new session: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): Username: 'hdpuser1' not found. Make sure your client's username exists on the cluster (state=08S01,code=0)​ Beeline version 3.1.0.3.0.1.0-187 by Apache Hive​ beeline>​ ... View more

Hi @stevenmatison,   We have another cluster with similar user where we are not facing any issue. Also, we have not set up any Ranger policies yet.   Could you pls advise ?   ... View more

Hi    venkatsambath, Thanks for the response.   Will check permissions for hive metastore.   2020-03-06T09:14:36,393 INFO [hiveServer2.async.multi_dest.batch_hiveServer2.async.multi_des HDFS audit. Event Size:1 2020-03-06T09:14:36,393 ERROR [hiveServer2.async.multi_dest.batch_hiveServer2.async.multi_deso consumer. provider=hiveServer2.async.multi_dest.batch, consumer=hiveServer2.async.multi_des 2020-03-06T09:14:36,393 INFO [hiveServer2.async.multi_dest.batch_hiveServer2.async.multi_des sleeping for 30000 milli seconds. indexQueue=0, queueName=hiveServer2.async.multi_dest.batch 2020-03-06T09:15:36,394 INFO [hiveServer2.async.multi_dest.batch_hiveServer2.async.multi_desg: name=hiveServer2.async.multi_dest.batch.hdfs, interval=01:00.015 minutes, events=1, deferr 2020-03-06T09:15:36,394 INFO [hiveServer2.async.multi_dest.batch_hiveServer2.async.multi_desg HDFS Filesystem Config: Configuration: core-default.xml, core-site.xml, mapred-default.xml,ite.xml 2020-03-06T09:15:36,412 INFO [hiveServer2.async.multi_dest.batch_hiveServer2.async.multi_des whether log file exists. hdfPath=hdfs://localhost:xxxx/hiveServer2/xxxxx/hiveServer2_rang 2020-03-06T09:15:36,413 ERROR [hiveServer2.async.multi_dest.batch_hiveServer2.async.multi_deso log file. java.net.ConnectException: Call From XXXXXX.XXXXX.XXX/10.21.16.60 to localhost:8020 ed; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.GeneratedConstructorAccessor61.newInstance(Unknown Source) ~[?:?] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAcc at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_181] at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831) ~[hadoop-common- at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:755) ~[hadoop-common-3. at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1501) ~[hadoop-common-3.1. at org.apache.hadoop.ipc.Client.call(Client.java:1443) ~[hadoop-common-3.1.1.3.0.1.0- at org.apache.hadoop.ipc.Client.call(Client.java:1353) ~[hadoop-common-3.1.1.3.0.1.0- at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) at com.sun.proxy.$Proxy32.getFileInfo(Unknown Source) ~[?:?] at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(C1.0-187.jar:?] at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source) ~[?:?] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java: at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181] at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHand at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocatio at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandl at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationH at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.ja at com.sun.proxy.$Proxy33.getFileInfo(Unknown Source) ~[?:?] at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1654) ~[hadoop-hdfs-cl at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java: at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java: at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81 at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.j at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734) ~[hadoop-common-3.1.1 at org.apache.ranger.audit.destination.HDFSAuditDestination.getLogFileStream(HDFSAudi at org.apache.ranger.audit.destination.HDFSAuditDestination.access$000(HDFSAuditDesti at org.apache.ranger.audit.destination.HDFSAuditDestination$1.run(HDFSAuditDestinatio at org.apache.ranger.audit.destination.HDFSAuditDestination$1.run(HDFSAuditDestinatio at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_181] at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_181] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:173 at org.apache.ranger.audit.provider.MiscUtil.executePrivilegedAction(MiscUtil.java:52 at org.apache.ranger.audit.destination.HDFSAuditDestination.logJSON(HDFSAuditDestinat at org.apache.ranger.audit.queue.AuditFileSpool.sendEvent(AuditFileSpool.java:879) ~[ at org.apache.ranger.audit.queue.AuditFileSpool.runLogAudit(AuditFileSpool.java:827) at org.apache.ranger.audit.queue.AuditFileSpool.run(AuditFileSpool.java:757) ~[?:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181] Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.8.0_181] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:1.8.0_1 at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) ~[ at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) ~[hadoop-common-3.1.1.3. at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:687) ~[hadoop- at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:790) ~[hadoop-c at org.apache.hadoop.ipc.Client$Connection.access$3600(Client.java:410) ~[hadoop-comm at org.apache.hadoop.ipc.Client.getConnection(Client.java:1558) ~[hadoop-common-3.1.1 at org.apache.hadoop.ipc.Client.call(Client.java:1389) ~[hadoop-common-3.1.1.3.0.1.0- ... View more