Hi   I am trying to configure Viewfs with HDFS Federation.   I tried this with HDP 3 and I am trying to find the procedure it implement on CDP cluster.   May someone please share the procedure to do the same.   The objective it to share the data between tow CDP clusters. Thanks       ... View more

Ranger Installation fails as ranger fileset is missing in the repo. I dont see any ranger-admin package in the repository https:/archive.cloudera.com/p/HDP/3.x/3.1.4.0/centos7-ppc/ranger/   raise RuntimeError(message) RuntimeError: Failed to execute command '/usr/bin/yum -y install ranger_3_1_4_0_315-admin', exited with code '1', message: 'https://archive.cloudera.com/p/HDP/3.x/3.1.4.0/centos7-ppc/ranger/ranger_3_1_4_0_315-admin-1.2.0.3.1.4.0-315.ppc64le.rpm: [Errno 14] Error 404 - The requested URL returned error: 404 Not Found Trying other mirror. To address this issue please refer to the below knowledge base article ... View more

Getting below error post the integration with kerberos   Error starting ResourceManager org.apache.hadoop.service.ServiceStateException: java.io.IOException: DestHost:destPort ces1pub.pbm.ihost.com:8020 , LocalHost:localPort cdp2pub.pbm.ihost.com/129.40.6.167:0. Failed on local exception: java.io.IOException: Server asks us to fall back to SIMPLE auth, but this client is configured to only allow secure connections. at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:203) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:866) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:1269) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1310) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1306) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1306) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1357) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1547) Caused by: java.io.IOException: DestHost:destPort ces1pub.pbm.ihost.com:8020 , LocalHost:localPort cdp2pub.pbm.ihost.com/129.40.6.167:0. Failed on local exception: java.io.IOException: Server asks us to fall back to SIMPLE auth, but this client is configured to only allow secure connections. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:806) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1566) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy89.mkdirs(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:666) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:431) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:166) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:158) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:96) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:362) at com.sun.proxy.$Proxy90.mkdirs(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2463) at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2439) at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1476) at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1473) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1490) at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1465) at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2374) at org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore$3.run(FileSystemRMStateStore.java:679) at org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore$3.run(FileSystemRMStateStore.java:676) at org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore$FSAction.runWithRetries(FileSystemRMStateStore.java:792) at org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore.mkdirsWithRetries(FileSystemRMStateStore.java:682) at org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore.startInternal(FileSystemRMStateStore.java:160) at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore.serviceStart(RMStateStore.java:824) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) ... 12 more Caused by: java.io.IOException: Server asks us to fall back to SIMPLE auth, but this client is configured to only allow secure connections. at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:851) at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:413) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1636) at org.apache.hadoop.ipc.Client.call ... View more

File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call raise ExecutionFailed(err_msg, code, out, err) resource_management.core.exceptions.ExecutionFailed: Execution of 'ambari-sudo.sh -H -E /usr/hdp/3.1.4.0-315/hadoop/bin/hdfs --config /usr/hdp/3.1.4.0-315/hadoop/conf --daemon start datanode' returned 1. ERROR: Cannot set priority of datanode process 45359 stdout: 2021-04-22 03:25:38,875 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.4.0-315 -> 3.1.4.0-315 2021-04-22 03:25:38,931 - Using hadoop conf dir: /usr/hdp/3.1.4.0-315/hadoop/conf 2021-04-22 03:25:39,273 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.4.0-315 -> 3.1.4.0-315 2021-04-22 03:25:39,289 - Using hadoop conf dir: /usr/hdp/3.1.4.0-315/hadoop/conf 2021-04-22 03:25:39,292 - Group['hdfs'] {} 2021-04-22 03:25:39,294 - Group['hadoop'] {} 2021-04-22 03:25:39,295 - Group['users'] {} 2021-04-22 03:25:39,296 - User['yarn-ats'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None} 2021-04-22 03:25:39,297 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}   ... View more

+ BASIC_GC_TUNING_ARGS= + case $JAVA_MAJOR in + BASIC_GC_TUNING_ARGS=' ' + CSD_GC_ARGS=' ' + CSD_JAVA_OPTS+=' ' ++ replace_pid -XX:+HeapDumpOnOutOfMemoryError '-XX:HeapDumpPath=/tmp/spark_on_yarn_spar40365358-SPARK_YARN_HISTORY_SERVER-42af0f75a56c8c9b8b467a684_pid{{PID}}.hprof' -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh ++ echo -XX:+HeapDumpOnOutOfMemoryError '-XX:HeapDumpPath=/tmp/spark_on_yarn_spar40365358-SPARK_YARN_HISTORY_SERVER-42af0f75a56c8c9b8b467a684_pid{{PID}}.hprof' -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh ++ sed 's#{{PID}}#29430#g' + export 'CSD_JAVA_OPTS=-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/spark_on_yarn_spar40365358-SPARK_YARN_HISTORY_SERVER-42af0f75a56c8c9b8b467a684_pid29430.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh' + CSD_JAVA_OPTS='-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/spark_on_yarn_spar40365358-SPARK_YARN_HISTORY_SERVER-42af0f75a56c8c9b8b467a684_pid29430.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh' + echo 'Using -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/spark_on_yarn_spar40365358-SPARK_YARN_HISTORY_SERVER-42af0f75a56c8c9b8b467a684_pid29430.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh as CSD_JAVA_OPTS' + source_parcel_environment + '[' '!' -z /opt/cloudera/parcels/CDH-7.1.4-1.cdh7.1.4.p0.6300266/meta/cdh_env.sh ']' + OLD_IFS=' ' + IFS=: + SCRIPT_ARRAY=($SCM_DEFINES_SCRIPTS) + DIRNAME_ARRAY=($PARCEL_DIRNAMES) ... View more