Hi , We need to delete some zeppelin notebooks using the notebookid via REST API or curl command. The default storage of zeppelin notebooks are on hdfs. hdp 3.1.5. Zeppelin 0.8.0   Kindly help us with the same. @Akhil S Naik   @Felix Albani  @Scharan  ... View more

Team, Many users are facing intermittent issues using jdbc interpreter  Issue : Unable to instantiate org.apache.hadoop.hive.q1.metadata.SessionHiveMetaStoreClient   Workaround is to restart jdbc interpreter , so as to automate this process need to add this in a shell script.  Can someone help us on this.     Thanks ASIF. ... View more

Team,   Could you provide me the schema details / create table statement for atlas_janus (hbase table) , as in a usecase we need to access this table via phoenix wherein the schema information for the table is required.     Thanks. ASIF. ... View more

Hi Team, Whenever AD user is trying to access fileview , the namenode goes down. We investigated and found issue is with sssd as we need to add user search filter from the OU and not from the mail domain. But in the OU we have just groups and users are mapped in the OU. Can anyone help us with ldap filter to search users from the OU and not from the main domain (only groups are present in the OU and users are mapped). Thanks, ... View more

2018-11-14T20:36:33,075 INFO [main] org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool - Creating metastore client for PreUpgradeTool 2018-11-14T20:36:33,106 INFO [main] hive.metastore - Trying to connect to metastore with URI thrift://sjdcdlake02.np1.ril.com:9083 2018-11-14T20:36:33,328 INFO [main] hive.metastore - Opened a connection to metastore, current connections: 1 2018-11-14T20:36:33,329 INFO [main] hive.metastore - Connected to metastore. 2018-11-14T20:36:34,533 INFO [main] hive.metastore - Trying to connect to metastore with URI thrift://sjdcdlake02.np1.ril.com:9083 2018-11-14T20:36:34,548 INFO [main] hive.metastore - Opened a connection to metastore, current connections: 2 2018-11-14T20:36:34,549 INFO [main] hive.metastore - Connected to metastore. 2018-11-14T20:36:36,572 ERROR [main] org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool - PreUpgradeTool failed org.apache.hadoop.hive.metastore.api.MetaException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87) at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:2015) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1404) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:4137) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1137) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:866) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_result$get_table_resultStandardScheme.read(ThriftHiveMetastore.java:53086) ~[hive-metastore-2.1.0.2.6.4.0-91.jar:2.1.0.2.6.4.0-91] at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_result$get_table_resultStandardScheme.read(ThriftHiveMetastore.java:53063) ~[hive-metastore-2.1.0.2.6.4.0-91.jar:2.1.0.2.6.4.0-91] at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_result.read(ThriftHiveMetastore.java:52994) ~[hive-metastore-2.1.0.2.6.4.0-91.jar:2.1.0.2.6.4.0-91] at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1507) ~[hive-metastore-2.1.0.2.6.4.0-91.jar:2.1.0.2.6.4.0-91] ... View more

Atlas Webui is working fine and i am able to login in atlas on port 21000, but i get the below alert and unable to understand how to get rid of this alert. Kindly help me with the below alert : Connection failed to http://localhost:21000/api/atlas/admin/status (Execution of 'curl --location-trusted -k --negotiate -u : -b /var/lib/ambari-agent/tmp/cookies/629e7881-187d-4548-8fa3-7425d62dce9e -c /var/lib/ambari-agent/tmp/cookies/629e7881-187d-4548-8fa3-7425d62dce9e -w '%{http_code}' http://localhost:21000/api/atlas/admin/status --connect-timeout 5 --max-time 7 -o /dev/null 1>/tmp/tmpNteFoC 2>/tmp/tmpeS6CDs' returned 28. % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:56 --:--:-- 0 curl: (28) Resolving timed out after 5517 milliseconds 000) ... View more

Using Hive configuration directory [/usr/hdp/2.6.2.0-205/hive2/conf] /usr/hdp/2.6.4.0-91/hadoop/conf:/usr/hdp/2.6.4.0-91/hadoop/lib/*:/usr/hdp/2.6.4.0-91/hadoop/.//*:/usr/hdp/2.6.4.0-91/hadoop-hdfs/./:/usr/hdp/2.6.4.0-91/hadoop-hdfs/lib/*:/usr/hdp/2.6.4.0-91/hadoop-hdfs/.//*:/usr/hdp/2.6.4.0-91/hadoop-yarn/lib/*:/usr/hdp/2.6.4.0-91/hadoop-yarn/.//*:/usr/hdp/2.6.4.0-91/hadoop-mapreduce/lib/*:/usr/hdp/2.6.4.0-91/hadoop-mapreduce/.//*:/usr/hdp/2.6.4.0-91/tez/*:/usr/hdp/2.6.4.0-91/tez/lib/*:/usr/hdp/2.6.4.0-91/tez/conf Log file for import is /usr/hdp/current/atlas-server/logs/import-hive.log SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/2.6.2.0-205/hive2/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.6.4.0-91/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] 2018-11-01T14:42:55,450 INFO [main] org.apache.atlas.ApplicationProperties - Looking for atlas-application.properties in classpath 2018-11-01T14:42:55,452 INFO [main] org.apache.atlas.ApplicationProperties - Looking for /atlas-application.properties in classpath 2018-11-01T14:42:55,453 INFO [main] org.apache.atlas.ApplicationProperties - Loading atlas-application.properties from null Exception in thread "main" org.apache.atlas.hook.AtlasHookException: HiveMetaStoreBridge.main() failed. at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:650) Caused by: org.apache.atlas.AtlasException: Failed to load application properties at org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:97) at org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:64) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:622) Caused by: org.apache.commons.configuration.ConfigurationException: Cannot locate configuration source null at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:217) at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:197) at org.apache.commons.configuration.AbstractFileConfiguration.<init>(AbstractFileConfiguration.java:181) at org.apache.commons.configuration.PropertiesConfiguration.<init>(PropertiesConfiguration.java:269) at org.apache.atlas.ApplicationProperties.<init>(ApplicationProperties.java:47) at org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:93) ... 2 more Failed to import Hive Data Model!!! ... View more

upgrade264.png Hi, I have tried upgrading the hdp 2.6.2 cluster to 2.6.4 using the vdf file, but now i am unable to delet the 2.6.4 version from my list, 2.6.4 has been installed on some of my hosts but when i go to ambari version it is showing 2.6.2 as my current version. Regards Asif ... View more

Hi , I have successfully enabled ssl and ldap with nifi in hdp cluster through (integrating hdf components to hdp using mpack) The Challenge i am facing is the authentication is success for nifiadmin and all users in ldap but receive this error once we are logged in "Unable to view the user interface. Contact the system administrator.". Ranger has been enabled for authorization and i have created policies for complete access for nifiadmin user but still the issue remains. ... View more

INFO: Processing artifact hdp-addon-services of type stack-addon-service-definitions in /var/lib/ambari-server/resources/mpacks/hdf-ambari-mpack-3.1.1.0-35/hdp-addon-services/HDF/3.0 INFO: Loading properties from /etc/ambari-server/conf/ambari.properties Traceback (most recent call last): File "/usr/sbin/ambari-server.py", line 952, in <module> mainBody() File "/usr/sbin/ambari-server.py", line 922, in mainBody main(options, args, parser) File "/usr/sbin/ambari-server.py", line 874, in main action_obj.execute() File "/usr/sbin/ambari-server.py", line 78, in execute self.fn(*self.args, **self.kwargs) File "/usr/lib/python2.6/site-packages/ambari_server/setupMpacks.py", line 896, in install_mpack (mpack_metadata, mpack_name, mpack_version, mpack_staging_dir, mpack_archive_path) = _install_mpack(options, replay_mode) File "/usr/lib/python2.6/site-packages/ambari_server/setupMpacks.py", line 794, in _install_mpack process_stack_addon_service_definitions_artifact(artifact, artifact_source_dir, options) File "/usr/lib/python2.6/site-packages/ambari_server/setupMpacks.py", line 554, in process_stack_addon_service_definitions_artifact sudo.symlink(source_service_version_path, dest_link) File "/usr/lib/python2.6/site-packages/resource_management/core/sudo.py", line 123, in symlink os.symlink(source, link_name) OSError: [Errno 17] File exists I also tried one of the solutions as below , but it didnt work. Cause: This issue occurs when the Symlink for NiFi and other HDF related services are present in the Resource Directories. Solution: To resolve this issue, do the following: Remove (preferred to backup) the directories which was created while installing HDF-mpacks using the following command: rm -rf /var/lib/ambari-server/resources/common-services/NIFI rm -rf /var/lib/ambari-server/resources/common-services/REGISTRY rm -rf /var/lib/ambari-server/resources/common-services/STREAMLINE rm -rf /var/lib/ambari-server/resources/mpacks /var/lib/ambari-server/resources/stacks/HDF Run the the command to install-mpacks for HDF. ... View more

Need to know about ReplAdmin, Service Admin, Lock level access in Ranger for hive, can someone give me details on this also please provide some example for the same. ... View more

Sqoop job fails when try to run using oozie, it works well using command line. Error as below : 2018-07-16 11:19:07,280 WARN ShellActionExecutor:523 - SERVER[Server_hostname] USER[asif] GROUP[-] TOKEN[] APP[auto_ingest] JOB[0000007-180713182341268-oozie-oozi-W] ACTION[0000007-180713182341268-oozie-oozi-W@shell_2] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1] ... View more

I am new to NIFI and need help with the steps through which i can successfully import data from different RDBMS sources to HIVE using NIFI. ... View more

2018-04-19 19:50:02,088 INFO [Thread-58] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Setting job diagnostics to Job init failed : org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.FileNotFoundException: File does not exist: hdfs://bdceocluster/tmp/hadoop-yarn/staging/admin/.staging/job_1524146355137_0004/job.splitmetainfo at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1583) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1447) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1405) at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385) at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302) at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:999) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:139) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1385) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1160) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$5.run(MRAppMaster.java:1599) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1595) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1526) Caused by: java.io.FileNotFoundException: File does not exist: hdfs://bdceocluster/tmp/hadoop-yarn/staging/admin/.staging/job_1524146355137_0004/job.splitmetainfo at org.apache.hadoop.hdfs.DistributedFileSystem$26.doCall(DistributedFileSystem.java:1446) at org.apache.hadoop.hdfs.DistributedFileSystem$26.doCall(DistributedFileSystem.java:1438) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1438) at org.apache.hadoop.mapreduce.split.SplitMetaInfoReader.readSplitMetaInfo(SplitMetaInfoReader.java:51) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1578) ... 17 more 2018-04-19 19:50:02,089 INFO [Thread-58] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: History url is http://slave1.bd-coe.com:19888/jobhistory/job/job_1524146355137_0004 2018-04-19 19:50:02,099 INFO [Thread-58] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Waiting for application to be successfully unregistered. 2018-04-19 19:50:03,103 INFO [Thread-58] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Final Stats: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0 2018-04-19 19:50:03,105 INFO [Thread-58] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Deleting staging directory hdfs://bdceocluster /tmp/hadoop-yarn/staging/yarn/.staging/job_1524146355137_0004 2018-04-19 19:50:03,194 INFO [Thread-58] org.apache.hadoop.ipc.Server: Stopping server on 46285 2018-04-19 19:50:03,196 INFO [IPC Server listener on 46285] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 46285 2018-04-19 19:50:03,196 INFO [TaskHeartbeatHandler PingChecker] org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler thread interrupted 2018-04-19 19:50:03,197 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder ... View more

I have configured Kerberos and Spnego on my Centos Machine , Now Client want to access ambari views , namenode ui , RM UI from Windows Machine via browser and we are unable to get success in that. Can we enable spnego authentication for windows machine ?? ... View more

Every time i try to use the Hadoop Resource Manager web interface (http://resource-manger.host:8088/cluster/) i show up logged in as dr.who. My question, how can I login as another user? In this case i want to login as myself and have a higher lever of privileges than dr.who. My Cluster is non kerberised. ... View more