Hi Team, Do we have a way to enable auto kinit for all the Active Directory Users when they login to edge node. sssd service is enabled and integrated with AD. Needed to know if we can add any parameter to enable auto kinit for the users. ... View more

Hi Team, Whenever AD user is trying to access fileview , the namenode goes down. We investigated and found issue is with sssd as we need to add user search filter from the OU and not from the mail domain. But in the OU we have just groups and users are mapped in the OU. Can anyone help us with ldap filter to search users from the OU and not from the main domain (only groups are present in the OU and users are mapped). Thanks, ... View more

2018-11-14T20:36:33,075 INFO [main] org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool - Creating metastore client for PreUpgradeTool 2018-11-14T20:36:33,106 INFO [main] hive.metastore - Trying to connect to metastore with URI thrift://sjdcdlake02.np1.ril.com:9083 2018-11-14T20:36:33,328 INFO [main] hive.metastore - Opened a connection to metastore, current connections: 1 2018-11-14T20:36:33,329 INFO [main] hive.metastore - Connected to metastore. 2018-11-14T20:36:34,533 INFO [main] hive.metastore - Trying to connect to metastore with URI thrift://sjdcdlake02.np1.ril.com:9083 2018-11-14T20:36:34,548 INFO [main] hive.metastore - Opened a connection to metastore, current connections: 2 2018-11-14T20:36:34,549 INFO [main] hive.metastore - Connected to metastore. 2018-11-14T20:36:36,572 ERROR [main] org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool - PreUpgradeTool failed org.apache.hadoop.hive.metastore.api.MetaException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87) at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:2015) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1404) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:4137) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1137) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:866) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_result$get_table_resultStandardScheme.read(ThriftHiveMetastore.java:53086) ~[hive-metastore-2.1.0.2.6.4.0-91.jar:2.1.0.2.6.4.0-91] at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_result$get_table_resultStandardScheme.read(ThriftHiveMetastore.java:53063) ~[hive-metastore-2.1.0.2.6.4.0-91.jar:2.1.0.2.6.4.0-91] at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_result.read(ThriftHiveMetastore.java:52994) ~[hive-metastore-2.1.0.2.6.4.0-91.jar:2.1.0.2.6.4.0-91] at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1507) ~[hive-metastore-2.1.0.2.6.4.0-91.jar:2.1.0.2.6.4.0-91] ... View more

Atlas Webui is working fine and i am able to login in atlas on port 21000, but i get the below alert and unable to understand how to get rid of this alert. Kindly help me with the below alert : Connection failed to http://localhost:21000/api/atlas/admin/status (Execution of 'curl --location-trusted -k --negotiate -u : -b /var/lib/ambari-agent/tmp/cookies/629e7881-187d-4548-8fa3-7425d62dce9e -c /var/lib/ambari-agent/tmp/cookies/629e7881-187d-4548-8fa3-7425d62dce9e -w '%{http_code}' http://localhost:21000/api/atlas/admin/status --connect-timeout 5 --max-time 7 -o /dev/null 1>/tmp/tmpNteFoC 2>/tmp/tmpeS6CDs' returned 28. % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:56 --:--:-- 0 curl: (28) Resolving timed out after 5517 milliseconds 000) ... View more

Please find the below logs while namenode reboot. sterr: File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/jmx.py", line 42, in get_value_from_jmx return data_dict["beans"][0][property] IndexError: list index out of range 2018-11-05 13:27:52,753 - Getting jmx metrics from NN failed. URL: http://sjdcdlake02.np1.ril.com:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/jmx.py", line 42, in get_value_from_jmx return data_dict["beans"][0][property] IndexError: list index out of range Python script has been killed due to timeout after waiting 1800 secs stdout: 2018-11-05 13:27:55,682 - call returned (255, '18/11/05 13:27:55 INFO ipc.Client: Retrying connect to server: sjdcdlake02.np1.ril.com/10.21.51.76:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From sjdcdlake02.np1.ril.com/10.21.51.76 to sjdcdlake02.np1.ril.com:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused') 2018-11-05 13:27:55,683 - NameNode HA states: active_namenodes = [(u'nn1', 'sjdcdlake01.np1.ril.com:50070')], standby_namenodes = [], unknown_namenodes = [(u'nn2', 'sjdcdlake02.np1.ril.com:50070')] 2018-11-05 13:27:55,684 - Will retry 8 time(s), caught exception: The NameNode nn2 is not listed as Active or Standby, waiting.... Sleeping for 25 sec(s) ... View more

Using Hive configuration directory [/usr/hdp/2.6.2.0-205/hive2/conf] /usr/hdp/2.6.4.0-91/hadoop/conf:/usr/hdp/2.6.4.0-91/hadoop/lib/*:/usr/hdp/2.6.4.0-91/hadoop/.//*:/usr/hdp/2.6.4.0-91/hadoop-hdfs/./:/usr/hdp/2.6.4.0-91/hadoop-hdfs/lib/*:/usr/hdp/2.6.4.0-91/hadoop-hdfs/.//*:/usr/hdp/2.6.4.0-91/hadoop-yarn/lib/*:/usr/hdp/2.6.4.0-91/hadoop-yarn/.//*:/usr/hdp/2.6.4.0-91/hadoop-mapreduce/lib/*:/usr/hdp/2.6.4.0-91/hadoop-mapreduce/.//*:/usr/hdp/2.6.4.0-91/tez/*:/usr/hdp/2.6.4.0-91/tez/lib/*:/usr/hdp/2.6.4.0-91/tez/conf Log file for import is /usr/hdp/current/atlas-server/logs/import-hive.log SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/2.6.2.0-205/hive2/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.6.4.0-91/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] 2018-11-01T14:42:55,450 INFO [main] org.apache.atlas.ApplicationProperties - Looking for atlas-application.properties in classpath 2018-11-01T14:42:55,452 INFO [main] org.apache.atlas.ApplicationProperties - Looking for /atlas-application.properties in classpath 2018-11-01T14:42:55,453 INFO [main] org.apache.atlas.ApplicationProperties - Loading atlas-application.properties from null Exception in thread "main" org.apache.atlas.hook.AtlasHookException: HiveMetaStoreBridge.main() failed. at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:650) Caused by: org.apache.atlas.AtlasException: Failed to load application properties at org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:97) at org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:64) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:622) Caused by: org.apache.commons.configuration.ConfigurationException: Cannot locate configuration source null at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:217) at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:197) at org.apache.commons.configuration.AbstractFileConfiguration.<init>(AbstractFileConfiguration.java:181) at org.apache.commons.configuration.PropertiesConfiguration.<init>(PropertiesConfiguration.java:269) at org.apache.atlas.ApplicationProperties.<init>(ApplicationProperties.java:47) at org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:93) ... 2 more Failed to import Hive Data Model!!! ... View more

zeppeline-error.png Hi Team, I am new to Zeppelin and we are getting permission denied error while trying to run R code to read a csv file from zeppelin. Please check the attached image. We have changed the permission file to 777 also changed the ownership to user level but still getting same permission denied. Kindly guide us with the same. ... View more

We have deployed hdp 2.6.4 production cluster successfully and kerberized it with Local AD , Now we are asked to enabled delegation domain between local AD and Corporate AD , so users can by synced automatically in local AD. Any help here will be appreciated. ... View more

upgrade264.png Hi, I have tried upgrading the hdp 2.6.2 cluster to 2.6.4 using the vdf file, but now i am unable to delet the 2.6.4 version from my list, 2.6.4 has been installed on some of my hosts but when i go to ambari version it is showing 2.6.2 as my current version. Regards Asif ... View more

Hi , I have successfully enabled ssl and ldap with nifi in hdp cluster through (integrating hdf components to hdp using mpack) The Challenge i am facing is the authentication is success for nifiadmin and all users in ldap but receive this error once we are logged in "Unable to view the user interface. Contact the system administrator.". Ranger has been enabled for authorization and i have created policies for complete access for nifiadmin user but still the issue remains. ... View more

INFO: Processing artifact hdp-addon-services of type stack-addon-service-definitions in /var/lib/ambari-server/resources/mpacks/hdf-ambari-mpack-3.1.1.0-35/hdp-addon-services/HDF/3.0 INFO: Loading properties from /etc/ambari-server/conf/ambari.properties Traceback (most recent call last): File "/usr/sbin/ambari-server.py", line 952, in <module> mainBody() File "/usr/sbin/ambari-server.py", line 922, in mainBody main(options, args, parser) File "/usr/sbin/ambari-server.py", line 874, in main action_obj.execute() File "/usr/sbin/ambari-server.py", line 78, in execute self.fn(*self.args, **self.kwargs) File "/usr/lib/python2.6/site-packages/ambari_server/setupMpacks.py", line 896, in install_mpack (mpack_metadata, mpack_name, mpack_version, mpack_staging_dir, mpack_archive_path) = _install_mpack(options, replay_mode) File "/usr/lib/python2.6/site-packages/ambari_server/setupMpacks.py", line 794, in _install_mpack process_stack_addon_service_definitions_artifact(artifact, artifact_source_dir, options) File "/usr/lib/python2.6/site-packages/ambari_server/setupMpacks.py", line 554, in process_stack_addon_service_definitions_artifact sudo.symlink(source_service_version_path, dest_link) File "/usr/lib/python2.6/site-packages/resource_management/core/sudo.py", line 123, in symlink os.symlink(source, link_name) OSError: [Errno 17] File exists I also tried one of the solutions as below , but it didnt work. Cause: This issue occurs when the Symlink for NiFi and other HDF related services are present in the Resource Directories. Solution: To resolve this issue, do the following: Remove (preferred to backup) the directories which was created while installing HDF-mpacks using the following command: rm -rf /var/lib/ambari-server/resources/common-services/NIFI rm -rf /var/lib/ambari-server/resources/common-services/REGISTRY rm -rf /var/lib/ambari-server/resources/common-services/STREAMLINE rm -rf /var/lib/ambari-server/resources/mpacks /var/lib/ambari-server/resources/stacks/HDF Run the the command to install-mpacks for HDF. ... View more

Need to know about ReplAdmin, Service Admin, Lock level access in Ranger for hive, can someone give me details on this also please provide some example for the same. ... View more

Sqoop job fails when try to run using oozie, it works well using command line. Error as below : 2018-07-16 11:19:07,280 WARN ShellActionExecutor:523 - SERVER[Server_hostname] USER[asif] GROUP[-] TOKEN[] APP[auto_ingest] JOB[0000007-180713182341268-oozie-oozi-W] ACTION[0000007-180713182341268-oozie-oozi-W@shell_2] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1] ... View more

I am new to NIFI and need help with the steps through which i can successfully import data from different RDBMS sources to HIVE using NIFI. ... View more

2018-04-19 19:50:02,088 INFO [Thread-58] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Setting job diagnostics to Job init failed : org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.FileNotFoundException: File does not exist: hdfs://bdceocluster/tmp/hadoop-yarn/staging/admin/.staging/job_1524146355137_0004/job.splitmetainfo at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1583) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1447) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1405) at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385) at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302) at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:999) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:139) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1385) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1160) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$5.run(MRAppMaster.java:1599) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1595) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1526) Caused by: java.io.FileNotFoundException: File does not exist: hdfs://bdceocluster/tmp/hadoop-yarn/staging/admin/.staging/job_1524146355137_0004/job.splitmetainfo at org.apache.hadoop.hdfs.DistributedFileSystem$26.doCall(DistributedFileSystem.java:1446) at org.apache.hadoop.hdfs.DistributedFileSystem$26.doCall(DistributedFileSystem.java:1438) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1438) at org.apache.hadoop.mapreduce.split.SplitMetaInfoReader.readSplitMetaInfo(SplitMetaInfoReader.java:51) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1578) ... 17 more 2018-04-19 19:50:02,089 INFO [Thread-58] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: History url is http://slave1.bd-coe.com:19888/jobhistory/job/job_1524146355137_0004 2018-04-19 19:50:02,099 INFO [Thread-58] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Waiting for application to be successfully unregistered. 2018-04-19 19:50:03,103 INFO [Thread-58] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Final Stats: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0 2018-04-19 19:50:03,105 INFO [Thread-58] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Deleting staging directory hdfs://bdceocluster /tmp/hadoop-yarn/staging/yarn/.staging/job_1524146355137_0004 2018-04-19 19:50:03,194 INFO [Thread-58] org.apache.hadoop.ipc.Server: Stopping server on 46285 2018-04-19 19:50:03,196 INFO [IPC Server listener on 46285] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 46285 2018-04-19 19:50:03,196 INFO [TaskHeartbeatHandler PingChecker] org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler thread interrupted 2018-04-19 19:50:03,197 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder ... View more

I have configured Kerberos and Spnego on my Centos Machine , Now Client want to access ambari views , namenode ui , RM UI from Windows Machine via browser and we are unable to get success in that. Can we enable spnego authentication for windows machine ?? ... View more

Every time i try to use the Hadoop Resource Manager web interface (http://resource-manger.host:8088/cluster/) i show up logged in as dr.who. My question, how can I login as another user? In this case i want to login as myself and have a higher lever of privileges than dr.who. My Cluster is non kerberised. ... View more

Hello, I am unable to use the Kafka import/export zookeeper offsets functionality. Below are the details: [root@dhm-rd-AWS-2 kafka_2.10-0.9.0.1]# bin/kafka-run-class.sh kafka.tools.ExportZkOffsets --zkconnect dev-kafka1:2181/kafka --group test-consumer-group --output-file /tmp/out.txt SCALA_BINARY_VERSION 2.10.5 Exception in thread "main" org.I0Itec.zkclient.exception.ZkNoNodeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /consumers/test-consumer-group/offsets at org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47) at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:995) at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:675) at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:671) at kafka.utils.ZkUtils.getChildren(ZkUtils.scala:537) at kafka.tools.ExportZkOffsets$.kafka$tools$ExportZkOffsets$getTopicsList(ExportZkOffsets.scala:125) at kafka.tools.ExportZkOffsets$anonfun$main$1.apply(ExportZkOffsets.scala:96) at kafka.tools.ExportZkOffsets$anonfun$main$1.apply(ExportZkOffsets.scala:95) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at kafka.tools.ExportZkOffsets$.main(ExportZkOffsets.scala:95) at kafka.tools.ExportZkOffsets.main(ExportZkOffsets.scala) Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /consumers/test-consumer-group/offsets at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1500) at org.I0Itec.zkclient.ZkConnection.getChildren(ZkConnection.java:114) at org.I0Itec.zkclient.ZkClient$4.call(ZkClient.java:678) at org.I0Itec.zkclient.ZkClient$4.call(ZkClient.java:675) at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:985) ... 12 more ... View more