Re: Unable to start HBase master after enabling ke...

baig_mujeeb · ‎11-15-2018

Solution for the desparate souls like me. copied atlas-application.properties from /etc/atlas/conf to /etc/hbase/conf changed permission to 744 and ownership to atlas:hadoop Restart Hbase and Atlas. Don't know if its the right thing but worked!!! Start to laugh again !!!.

baig_mujeeb · ‎11-15-2018

Solution for the desparate souls like me. copied atlas-application.properties from /etc/atlas/conf to /etc/hbase/conf changed permission to 744 and ownership to atlas:hadoop Restart Hbase and Atlas. Don't know if its the right thing but worked!!! Start to laugh again !!!.

baig_mujeeb · ‎11-15-2018

Hi, After upgrading to 3.0.1 and enabling kerberos, Hbase master is failing to start due to missing 'master' znode. It is leading to Atlas start failure. atlas TABLE Took 8.2656 secondsjava exception ERROR Java::OrgApacheZookeeper::KeeperException::NoNodeException: KeeperErrorCode = NoNode for /hbase-secure/master 2018-11-14 17:27:34,307 - Retrying after 10 seconds. Reason: Execution of 'kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-003@IIM.LOCAL; cat /var/lib/ambari-agent/tmp/atlas_hbase_setup.rb | hbase shell -n' returned 1. atlas_janus ATLAS_ENTITY_AUDIT_EVENTS atlas TABLE Took 8.2337 secondsjava exception In zookeeper 'master' znode is present is hbase-unsecure but not in secure-hbase [zk: hdata2.local:2181(CONNECTED) 0] ls /hbase-unsecure [replication, meta-region-server, rs, splitWAL, backup-masters, table-lock, flush-table-proc, master-maintenance, online-snapshot, master, switch, running, draining, namespace, hbaseid, table] [zk: hdata2.local:2181(CONNECTED) 0] ls /hbase-secure [replication, rs, splitWAL, backup-masters, table-lock, flush-table-proc, master-maintenance, online-snapshot, switch, running, tokenauth, draining, hbaseid, table] How do I fix this? Is this a known issue in 3.0.1? Thanks in advance.

baig_mujeeb · ‎11-14-2018

Some more logs atlas TABLE Took 8.2656 secondsjava exception ERROR Java::OrgApacheZookeeper::KeeperException::NoNodeException: KeeperErrorCode = NoNode for /hbase-secure/master 2018-11-14 17:27:34,307 - Retrying after 10 seconds. Reason: Execution of 'kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-003@IIM.LOCAL; cat /var/lib/ambari-agent/tmp/atlas_hbase_setup.rb | hbase shell -n' returned 1. atlas_janus ATLAS_ENTITY_AUDIT_EVENTS atlas TABLE Took 8.2337 secondsjava exception ERROR Java::OrgApacheZookeeper::KeeperException::NoNodeException: KeeperErrorCode = NoNode for /hbase-secure/master

baig_mujeeb · ‎11-14-2018

2018-11-14 16:15:03,009 WARN [master/hdata4:16000] master.ActiveMasterManager: Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2018-11-14 16:15:03,009 INFO [master/hdata4:16000] assignment.AssignmentManager: Stopping assignment manager 2018-11-14 16:15:03,032 WARN [master/hdata4:16000] assignment.AssignmentManager: No servers available; cannot place 1 unassigned regions. 2018-11-14 16:15:03,033 INFO [master/hdata4:16000] procedure2.RemoteProcedureDispatcher: Stopping procedure remote dispatcher 2018-11-14 16:15:03,033 INFO [master/hdata4:16000] procedure2.ProcedureExecutor: Stopping 2018-11-14 16:15:03,036 INFO [master/hdata4:16000] wal.WALProcedureStore: Stopping the WAL Procedure Store, isAbort=false 2018-11-14 16:15:03,070 ERROR [master/hdata4:16000] wal.WALProcedureStore: Unable to close the stream org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): Client (=DFSClient_NONMAPREDUCE_-826562846_1) is not the lease owner (=DFSClient_NONMAPREDUCE_-1195801889_1: /apps/hbase/data/MasterProcWALs/pv2-00000000000000000011.log (inode 10368947) Holder DFSClient_NONMAPREDUCE_-826562846_1 does not have any open files. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2837) at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.completeFileInternal(FSDirWriteFileOp.java:685) at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.completeFile(FSDirWriteFileOp.java:671) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:2858) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:928) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:607) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1497) at org.apache.hadoop.ipc.Client.call(Client.java:1443) at org.apache.hadoop.ipc.Client.call(Client.java:1353) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) at com.sun.proxy.$Proxy18.complete(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:550) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.complete(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy20.complete(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy20.complete(Unknown Source) 2018-11-14 16:15:03,071 INFO [master/hdata4:16000] hbase.ChoreService: Chore service for: master/hdata4:16000.splitLogManager. had [] on shutdown 2018-11-14 16:15:03,071 INFO [master/hdata4:16000] flush.MasterFlushTableProcedureManager: stop: server shutting down. 2018-11-14 16:15:03,071 ERROR [master/hdata4:16000] access.TableAuthManager: Something wrong with the TableAuthManager reference counting: org.apache.hadoop.hbase.security.access.TableAuthManager@7e83992 whose count is null Zookeeper znode listing for unsecure and secure. master znode is missing from hbase-secure [zk: hdata2.local:2181(CONNECTED) 0] ls /hbase-unsecure [replication, meta-region-server, rs, splitWAL, backup-masters, table-lock, flush-table-proc, master-maintenance, online-snapshot, master, switch, running, draining, namespace, hbaseid, table] [zk: hdata2.local:2181(CONNECTED) 0] ls /hbase-secure [replication, rs, splitWAL, backup-masters, table-lock, flush-table-proc, master-maintenance, online-snapshot, switch, running, tokenauth, draining, hbaseid, table]

baig_mujeeb · ‎11-12-2018

After enablling kerberos HBase master failing to restart. On investigation found that in zookeeper /hbase-secure/master node is missing. Zookeeper shows following Acl [zk: localhost:2181(CONNECTED) 1] getAcl /hbase-secure 'world,'anyone : r 'sasl,'hbase : cdrwa 'sasl,'hbase : cdrwa Tried all the threads eg. https://community.hortonworks.com/content/supportkb/151088/how-to-force-remove-znode-with-stale-acl.html https://community.hortonworks.com/articles/82405/how-to-remove-acl-protected-zk-node.html Any ideas?

baig_mujeeb · ‎11-08-2018

Many Thanks Sandeep. Yes it is working now. I would like to add more details for fix: Update hive-interactive-env template and hive-env template Hive config properties from Ambari UI.

baig_mujeeb · ‎11-07-2018

Hi Our Hive Server2 Interactive is stopping after around 10 min after restart. I couldn't find hiveserver2Interactive.log in /var/log/hive, hive-server2-interactive.err is showing following: /usr/hdp/current/hive-server2/conf_llap//hive-env.sh: line 43: [: !=: unary operator expected 2018-11-07 14:40:33: Starting HiveServer2 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/3.0.1.0-187/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/3.0.1.0-187/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Hive Session ID = c4e8edc5-ed5e-4f96-ba70-00779ac20dc9 Hive Session ID = b5ad883d-4a65-45f7-a6dd-1ea3797724ea Any help will be appreciated. Thanks

baig_mujeeb · ‎11-06-2018

I read about it (Data Analytics Studio) in some threads and want to know more about it (trial/download etc.), is it open source? Thanks

baig_mujeeb · ‎11-05-2018

Since we did not have ACID tables, it worked without pre-upgrade tool.

baig_mujeeb · ‎11-05-2018

After some investigation foend that KDC Administrator Credentials to the Ambari Credential Store was not properly set. Then set it from Ambari and it started to work

baig_mujeeb · ‎11-05-2018

When upgrading to 3.0.1 from 2.6.5 from Ambari I am getting following error message An internal system exception occurred: Unable to build keytab regeneration stage Any idea? Where should I look for exception error? Tried restarting KDC but of no help.

baig_mujeeb · ‎11-04-2018

The pre upgrade tool is reporting error requires Hive 2.x. Actual: 1.2.1000 I have Hive Interactive enabled and confirms to this thread https://community.hortonworks.com/questions/55051/no-hive-21-in-hdp-25.html I don't have ACID tables in Hive. Is it OK if I go ahead with version upgrade from Ambari without running pre-upgrade tool? Will it upgrade Hive? Kindly let me know

baig_mujeeb · ‎10-24-2018

Hi, I am in the process of upgrading hdp from 2.6 to 3.0.1, but got stuck with hive version 1.2. Getting following error from PreUpgradetool 18/10/23 17:55:19 ERROR acid.PreUpgradeTool: PreUpgradeTool failed java.lang.IllegalStateException: preUpgrade requires Hive 2.x. Actual: 1.2.1000 In ambari both YARN pre-emption and Hive Interactive are greyed so unable to change them. My beeline command is as follows and it works fine and loads hive 1.2.1000 beeline -u "jdbc:hive2://data1.dev.local:10000/default;" How can I upgrade to hive 2.0+? At the moment I don't have ACID tables so do I still need to go through preUpgrade tool? will version upgrade not automatically upgrade from 1.2 to 3.0? Regards MB

baig_mujeeb · ‎08-16-2018

Thanks for quick reply. I tried but still spark2 service principle is missing. I even checked it with kadmin. Wondering that Spark2 clients are running fine on the respective nodes, but issue is with Spark2 History Server and Spark2 Thrift Server. I can see in Spark2 config that spark.history.kerberos.keytab = /etc/security/keytabs/spark2.headless.keytab Looks like a bug somewhere!

baig_mujeeb · ‎08-15-2018

Hi, Recently upgraded to HDP 2.7 and added Spark2 via Ambari on kerberised cluster. It is failing to start Spark2 history server because of missing Spark2 headless keytab file. What is the right way to resolve it.

baig_mujeeb · ‎07-26-2018

@Sandeep Nemuri Thanks for swift reply. Is it OK if I also upgrade the underlying OS of Ambari host first followed by Ambari upgrade OR do the OS upgrade separately after Ambari is upgraded to 2.6.2.2?

baig_mujeeb · ‎07-26-2018

Thanks Felix, almost there, although we have a small cluster but In-Place & Restore options suits us (set up with best practice in mind -:). Also a silly question: Where in Ambari UI is Ambari "Recover Host" functionality? We are using Ambari 2.5.0.3 Thanks MB

baig_mujeeb · ‎07-26-2018

Hi, I have a 4 node dev cluster which was recently upgraded to 2.6.0.3. It is running on Ubuntu 14. I want to upgrade the underlying OS to 16.04.4. What are the right steps to achieve this? I am thinking on the following lines: 1. Pick one server at a time 2. Switch on maintenance mode and stop all services 3. run 'do-release-upgrade' command to do release upgrade 4. start all the services on the server Once all the servers are done, perform the version upgrade from Ambari UI for Ubuntu 16. Is this the right approach? Kindly let me know. Thanks in advance as usual. Regards

baig_mujeeb · ‎07-26-2018

Following jar files should be present gmetric4j-1.0.3.jar jmxetric-1.0.4.jar oncrpc-1.0.7.jar Since I did the version upgrade from 2.5 to 2.6 I copied them from /usr/hdp/2.5.3.0-37/storm/contrib/storm-jmxetric/lib folder Hope this helps

baig_mujeeb · ‎06-07-2018

Thanks Gaurav, it was helpful.

baig_mujeeb · ‎06-03-2018

Hi, We have a development/test cluster with 4 nodes each having 12 cores vcpu, 2 TB SDD and 32GB RAM previously used for cassandra. All running on Ubuntu 14. It is configured as follows 1 NN 1 SNN 4 DN Jobs are taking longer due to limited resources, so we want to upgrade the HDP version from 2.5 to 2.6 and OS to Ubuntu 16.04 and add more resources. We are planning to upgrade as follows, 1. Increase RAM to 128 GB on all nodes 2. Double the storage by adding additional SAS disks 3. Add 2 more additional Data Nodes with 10 cores, 128 GB Questions I got is 1. Is it a good practice to use all of them as DN, or we should separate the NN 2. Since it was borrowed off Cassandra, is it ok if we keep all nodes of same specs including 2 new nodes 3. There is no harm in mixed storage options ie. adding extra storage in SAS wrt existing SSD 4. With the proposed new cluster, upgrading to HDP 3.0 shouldn't be a issue, I guess Thanks in advance MB

baig_mujeeb · ‎02-13-2018

Thanks Eugene for swift reply. We are running on HDP 2.5 and MERGE is available in 2.6. What options do I have? Thanks MB

baig_mujeeb · ‎02-12-2018

Hi, I have a clustered bucketted table, with ACID enabled Does hive supports updates from another table? This is something I am trying to achieve, updating a column values from another table UPDATE transaction_bak SET fl_t_order = tt.t_order FROM trans_temp tt WHERE transaction_bak.id = tt.id; I tried with left join, but no success, getting ParseException line 2:0 missing SET at 'left' UPDATE transaction_bak LEFT OUTER JOIN trans_temp tt SET fl_t_order = tt.t_order WHERE transaction_bak.id = tt.id; Any idea

baig_mujeeb · ‎02-07-2018

Hi, I am getting this error when trying to insert into a hive table, the host in question (hdata2) is accessible and other services are running fine from it. klist shows valid kerberos ticket. I want to merge many tables into one, so was inserting executing external hql script. Any idea where to look for, logs looks fine. INFO : Tez session hasn't been created yet. Opening session ERROR : Failed to execute tez graph. java.lang.IllegalArgumentException: java.net.UnknownHostException: hdata2.xxxxxxxxxxxx.local at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:438) at org.apache.hadoop.yarn.util.timeline.TimelineUtils.buildTimelineTokenService(TimelineUtils.java:120) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:165) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.tez.client.TezYarnClient.init(TezYarnClient.java:46) at org.apache.tez.client.TezClient.start(TezClient.java:345) at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:196) at org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:272) at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:152) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1745) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1491) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1289) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1156) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1151) at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:197) at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:76) at org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:253) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:264) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.UnknownHostException: hdata2.xxxxxxxxxxxx.local ... 30 more Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask (state=08S01,code=1) Thanks MB

baig_mujeeb · ‎12-22-2017

It is using keytab, is it possible that kdestroy on the host can interfere

baig_mujeeb · ‎12-22-2017

Hi, My Spark Thrift Server goes down after the kerberos ticket time of 10 hrs. How to set up auto renewal of ticket. This is happening with Thrift server only. Regards MB

baig_mujeeb · ‎11-29-2017

ok. found it here https://community.hortonworks.com/questions/100335/kerberos-install-on-hdp-cluster-after-rangerranger.html

baig_mujeeb · ‎11-29-2017

Hi, I am about to kerberise the cluster from Ambari after disabling it. Ranger is also running, should I stop Ranger services before kerberising or decommission it and re-install Rangers after kerberos is installed? Thanks MB

baig_mujeeb · ‎11-28-2017

Also, I just noticed the PolicyRefresher exception in hadoop-hdfs-namenode-hmaster log Ranger services and plugins are disabled at the moment. Should I switch them on? 2017-11-28 13:27:51,752 ERROR util.PolicyRefresher (PolicyRefresher.java:loadPolicyfromPolicyAdmin(255)) - PolicyRefresher(serviceName=003_hadoop): failed t o refresh policies. Will continue to use last known version of policies (2) com.sun.jersey.api.client.ClientHandlerException: java.net.ConnectException: Connection refused at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149) at com.sun.jersey.api.client.Client.handle(Client.java:648)

Online	Offline
Last Visited	‎01-28-2021 08:08 AM

Date Registered	‎11-13-2017 04:35 PM
Last Visited	‎01-28-2021 08:08 AM
Total Messages Posted	50

Re: Unable to start HBase master after enabling ke...

Re: Unable to start HBase master after enabling ke...

Re: Zookeeper - HBase master node is missing after...

Zookeeper - HBase master node is missing after ker...

Re: Unable to start HBase master after enabling ke...

Re: Unable to start HBase master after enabling ke...

Unable to start HBase master after enabling kerber...

Re: hive server2 interactive logs

hive server2 interactive logs

Data Analytics Studio

Re: hdp 3.0.1 upgrade from 2.6.3 pre-requisite

Re: keytab regeneration stage failing - upgrading ...

keytab regeneration stage failing - upgrading to H...

hdp 3.0.1 upgrade from 2.6.3 pre-requisite

Hive 2.1 in hdp 2.5

Re: Spark2 missing keytab file for HDP 2.7

Spark2 missing keytab file for HDP 2.7

Re: HDP 2.6 cluster OS upgrade - Ubuntu

Re: HDP 2.6 cluster OS upgrade - Ubuntu

HDP 2.6 cluster OS upgrade - Ubuntu

Re: Storm - Supervisor and Nimbus dropping immedia...

Re: Upgrading the cluster - OS, HDP, resources

Upgrading the cluster - OS, HDP, resources

Re: Hive update a column from another table

Hive update a column from another table

UnknownHostException Exception while inserting int...

Re: kerberized spark thrift does not renew ticket

kerberized spark thrift does not renew ticket

Re: Kerberos on Ranger

Kerberos on Ranger

Re: Kerberos not fully disabled